src/scan.js skips only .git while walking:
const ALWAYS_SKIP = new Set([".git"]);
VENDOR_DIRS and BUILD_DIRS in classify.js are consulted after the file has already been stat'd and read, by which point the cost has been paid. So on any real JavaScript repo, ctxtrim recurses through the whole of node_modules, reads up to 5 MB of each file, builds a 4000 character sample, and then classifies the result as "vendored dependency directory" and moves on.
On a mid-sized project that is tens of thousands of files read to reach a conclusion the directory name gave away for free. It is by far the slowest part of the tool, and it is pure waste.
Suggested fix
Have the walker consult the same directory lists before recursing, so a vendored or build directory is recorded once and not descended into. The reported counts should not change: the directory still needs to appear in the ignore patterns and the category totals.
Acceptance
scanRepo on a fixture containing node_modules/ does not read any file inside it
- The emitted patterns and
byCategory totals for the fixture are unchanged from today
- A note in the README that directory-level categories are decided by name, not contents
This is the single biggest speedup available in the codebase and is well scoped for a first contribution.
src/scan.jsskips only.gitwhile walking:VENDOR_DIRSandBUILD_DIRSinclassify.jsare consulted after the file has already been stat'd and read, by which point the cost has been paid. So on any real JavaScript repo, ctxtrim recurses through the whole ofnode_modules, reads up to 5 MB of each file, builds a 4000 character sample, and then classifies the result as "vendored dependency directory" and moves on.On a mid-sized project that is tens of thousands of files read to reach a conclusion the directory name gave away for free. It is by far the slowest part of the tool, and it is pure waste.
Suggested fix
Have the walker consult the same directory lists before recursing, so a vendored or build directory is recorded once and not descended into. The reported counts should not change: the directory still needs to appear in the ignore patterns and the category totals.
Acceptance
scanRepoon a fixture containingnode_modules/does not read any file inside itbyCategorytotals for the fixture are unchanged from todayThis is the single biggest speedup available in the codebase and is well scoped for a first contribution.