Skip to content

v0.7.0 — take entries as they parse, instead of holding the catalogue

Latest

Choose a tag to compare

@ralyodio ralyodio released this 07 Sep 15:11
6798506

parseM3uStream never held the file, but it did hold every entry it kept. On the list that prompted this, 583MB and roughly 2.6 million entries, that array alone is more heap than a container has. The byte ceiling the two sites enforce upstream was really guarding this array, so it was a heap ceiling wearing a disguise.

Now: pass onEntries and the parse holds nothing across chunks. It is handed what each chunk produced and then forgets it, and it is awaited rather than called per entry, because push is synchronous and a consumer writing to a database has to be able to make the parse wait. That is what bounds an import to one batch of memory whatever the size of the list. drain() is the same thing on the line-at-a-time parser, and kept counts across drains so a drained parser does not forget it was full and start the ceiling again from zero.

max of 0 or NaN now means unlimited rather than keep-nothing. An unset PLAYLIST_MAX_CHANNELS parses to 0, and reading that literally would turn a missing config line into an import that succeeds and stores no channels, which is the one failure mode nothing reports.

A caller that passes neither onEntries nor max sees no change: MAX_CHANNELS still defaults to 300,000.

Full detail in #8. pnpm test: 125 pass, 0 fail. tsc --noEmit clean.