use stream instead of fs.readdir()

[ORESoftware]

I have this question on SO:
https://stackoverflow.com/questions/52131509/read-large-directory-with-nodejs

say we have a directory of 10,000,000 items or whatever. Instead of reading all paths into memory, it would be nice to be able to read the directory about 15,000 items concurrently.

Does Node.js offer a way to do that? Even with Unix child processes, still not sure how to do this.

[gireeshpunathil]

@ORESoftware -

the underlying capability that provides us the data is readdir / getdents, and as long as those provide us the data in one shot, whether we stream it or not is immaterial, as the data is already in memory.

In general terms, what this would mean is, where we don't have control over the volume of data that we receive from a source, making a streamified version do not make meaning?

[gireeshpunathil]

on a different note, if child processes use shell commands and stream data to parent, it would be sub-optimal IMO, as we are just forcing the data to be streamed, but:

• the entire data is held by child at some point in time
• latency in getting the data to the parent

[richardlau]

I think you're after nodejs/node#583 which AFAIK hasn't yet been implemented on the Node.js side. Possibly https://www.npmjs.com/package/fs-readdir-stream (from one of the comments in the issue)?

[gireeshpunathil]

ping @ORESoftware

[gireeshpunathil]

inactive, closing