Recursively list file paths in directory

[Ethan-Arrowood]

Is your feature request related to a problem? Please describe.
Please describe the problem you are trying to solve.

Recursively list file paths in a directory

Describe the solution you'd like
Please describe the desired behavior.

Dir structure like:

| a /
| |- b /
| |   |- c /
| |   |    |- d.js
| |   |    |- e.js
| |   |- f.js
| |- g.js

Should return ['a/b/c/d.js', 'a/b/c/e.js', 'a/b/f.js', 'a/g.js']

Describe alternatives you've considered
Please describe alternative solutions or features you have considered.

Some public solutions that are synchronous and may be able to made faster:

• https://attacomsian.com/blog/nodejs-list-directory-files
• https://coderrocketfuel.com/article/recursively-list-all-the-files-in-a-directory-using-node-js
• https://stackoverflow.com/questions/26827920/nodejs-recursively-list-files-in-directory

---

I wrote a version of this using async/await and typescript tweet. Not tested much and needs to be extrapolated for general purpose use

If this something the community is interested in I'd be happy to contribute it!

[ahtee]

Seems like an interesting proposal I can see added to a module like fs or path.

function could take a single parameter string, which would be the starting directory to begin recursively printing. possibly include a second parameter for searching for a specific file or folder match in regex?

[aduh95]

I don't think it's worth adding a new API for that, it's already possible to do it in user land. I would be more relevant as an npm package rather than a core utility.

[richardlau]

FYI @nodejs/tooling who have previously expressed an interest in adding recursive options to Node.js APIs.

[Ethan-Arrowood]

I'd be keen on developing this first as a standalone module. Then we can consider adopting it into core

[bnb]

Related, I stumbled across this issue a while ago. There's currently a module with 16m downloads/month called recursive-readdir that seems to be widely used. At the time I encountered an issue with it and reached out to the author - he said he doesn't really have time to maintain it and asked if I'd wanted to be a maintainer. I am now a maintainer 😅

The implementation in the module is pretty good from what I can tell - the best that currently exists, I think. I'd be happy to help migrate it into Node.js if it makes sense. Also happy to have you as a maintainer if you're interested in improving upon it as a base, @Ethan-Arrowood.

[Ethan-Arrowood]

This looks great and yes I'd be happy to help maintain it with you @bnb

[bnb]

Extending what @richardlau said, I do think it makes sense to move it into Node.js at some point. I've just not done the work to start that discussion 😅

[Ethan-Arrowood]

Yeah I'm also wondering if this is one of those 'add to core' or 'add to org' like undici

[cjihrig]

I don't think we should vendor a module that reads directories via fs.readdir() now that we have a streaming alternative.

[Ethan-Arrowood]

I was thinking about contributing the stream version to recursive-readdir

[Ethan-Arrowood]

i.e. maybe something based around this one: https://nodejs.org/api/fs.html#fs_fspromises_opendir_path_options
but open to other ideas recommendations too

[Ethan-Arrowood]

async function listDir (path, acc = []) {
	const dir = await opendir(path)

	for await (const dirent of dir) {
		if (dirent.isDirectory()) {
			listDir(join(path,dirent.name), acc)
		} else {
			acc.push(join(path, dirent.name))
		}
	}

	return acc
}

[devsnek]

async function* listDir(path) {
  const dir = await opendir(path);
  for await (const dirent of dir) {
    const name = join(path, dirent.name);
    if (dirent.isDirectory()) {
      yield* listDir(name);
    } else {
      yield name;
    }
  }
}

[Ethan-Arrowood]

Fantastic use of async generators! I think this is really solid solution

[Ethan-Arrowood]

Added support for ignoring the root path

async function* listDir(path, opts = { ignoreRoot: false }) {
  if (this.meta == null) {
    this.meta = { root: path }
  }

  const dir = await opendir(path)

  for await (const dirent of dir) {
    const name = join(path, dirent.name)
    if (dirent.isDirectory()) {
      yield* listDir(name, opts)
    } else {
      yield opts.ignoreRoot ? relative(this.meta.root, name) : name
    }
  }
}

Alternate:

async function* listDir(path, opts = { ignoreRoot: false }, meta = { root: path }) {
  const dir = await opendir(path)

  for await (const dirent of dir) {
    const name = join(path, dirent.name)
    if (dirent.isDirectory()) {
      yield* listDir(name, opts, meta)
    } else {
      yield opts.ignoreRoot ? relative(meta.root, name) : name
    }
  }
}

TypeScript version:

async function* listDir(
  path: string,
  opts: options = { ignoreRoot: false },
  meta: meta = { root: path }
): AsyncIterable<string> {
  const dir = await opendir(path)

  for await (const dirent of dir) {
    const name = join(path, dirent.name)
    if (dirent.isDirectory()) {
      yield* listDir(name, opts, meta)
    } else {
      yield opts.ignoreRoot ? relative(meta.root, name) : name
    }
  }
}

[jimmywarting]

I wonder how any of the suggested iterators compare up to find ~ in performances.
if we would have { recursive: true } added to fs.readDir(Sync) like any of the other stuff we added a recursive option to mkdir, rm and so on... seems to be trending.

[bcoe]

@Ethan-Arrowood @jimmywarting I missed this issue when I was flagged on it last year, this sounds like a great addition to the fs API.

[Ethan-Arrowood]

I'll put this back on top of my stack of things to contribute. I'll try to open a PR soon with something and we can all discuss from there!

[bcoe]

@Ethan-Arrowood 👍 I'd make sure to loop in @bnb on the PR, they've been contributing to recursive-readdir I believe, and I'm sure have opinions.