Skip to content

The walker descends into node_modules and reads every file before deciding it is vendored #3

Description

@royalpinto007

src/scan.js skips only .git while walking:

const ALWAYS_SKIP = new Set([".git"]);

VENDOR_DIRS and BUILD_DIRS in classify.js are consulted after the file has already been stat'd and read, by which point the cost has been paid. So on any real JavaScript repo, ctxtrim recurses through the whole of node_modules, reads up to 5 MB of each file, builds a 4000 character sample, and then classifies the result as "vendored dependency directory" and moves on.

On a mid-sized project that is tens of thousands of files read to reach a conclusion the directory name gave away for free. It is by far the slowest part of the tool, and it is pure waste.

Suggested fix

Have the walker consult the same directory lists before recursing, so a vendored or build directory is recorded once and not descended into. The reported counts should not change: the directory still needs to appear in the ignore patterns and the category totals.

Acceptance

  • scanRepo on a fixture containing node_modules/ does not read any file inside it
  • The emitted patterns and byCategory totals for the fixture are unchanged from today
  • A note in the README that directory-level categories are decided by name, not contents

This is the single biggest speedup available in the codebase and is well scoped for a first contribution.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workinghelp wantedExtra attention is needed

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions