Skip to content

perf(mapper): lazily evaluate Node.js project discovery in MapperContext#156

Merged
steipete merged 4 commits into
openclaw:mainfrom
Tanmay-008:perf/lazy-node-discovery
Jul 12, 2026
Merged

perf(mapper): lazily evaluate Node.js project discovery in MapperContext#156
steipete merged 4 commits into
openclaw:mainfrom
Tanmay-008:perf/lazy-node-discovery

Conversation

@Tanmay-008

@Tanmay-008 Tanmay-008 commented Jul 12, 2026

Copy link
Copy Markdown
Contributor

Summary

Fix eager Node project and Turbo task-graph work for repositories that do not use Node, while preserving the contributor's lazy shared-context design.

The repaired implementation:

  • Gates the four Node-family mappers behind existing detection plus bounded fallback signals.
  • Starts non-Node mappers without waiting for the fallback Node signal.
  • Memoizes Node project discovery and Turbo graph construction once per mapping run, including concurrent first access and failure propagation.
  • Preserves package-less conventional projects, nested React packages, and package-less Nx projects outside conventional roots.
  • Retries discovery on a fresh mapping run after a prior run fails.

Validation

Exact-head proof, instrumentation, real-repository CLI runs, timings, dependency/security audit, and model gate: #156 (comment)

Hosted exact-head gates:

Release-note context and credit

Release-note context is preserved here for the release process and credits @Tanmay-008. The contributor's original commit remains the first commit in this PR, followed by maintainer repair commits with co-author credit.

Fixes #155

Converts eager  and  on  to lazy,
memoized async getters. This avoids executing costly disk I/O for Node.js
project discovery on non-Node repositories (like pure Go/Python/Rust),
removing startup blockage for other mappers while maintaining shared
concurrency for Node-ecosystem mappers.
@Tanmay-008 Tanmay-008 requested a review from a team as a code owner July 12, 2026 06:53
@clawsweeper clawsweeper Bot added rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. P2 Normal priority bug or improvement with limited blast radius. labels Jul 12, 2026
@clawsweeper

clawsweeper Bot commented Jul 12, 2026

Copy link
Copy Markdown
Contributor

Codex review: needs maintainer review before merge. Reviewed July 12, 2026, 9:56 AM ET / 13:56 UTC.

Summary
The PR adds a lazy memoized MapperContext, gates the four Node-family mappers behind detected or bounded fallback signals, updates their consumers, and adds concurrency, failure, timing, and project-shape regressions.

Reproducibility: yes. Current main directly shows both Node loaders completing before the mapper Promise.all begins, and the PR’s instrumentation demonstrates the resulting unnecessary calls on non-Node repositories.

Review metrics: 3 noteworthy metrics.

  • Mapper eligibility: 4 Node-family mappers gated. The change consistently covers Node, Next, React, and Node-route consumers through one eligibility decision.
  • Regression coverage: 182 test lines added. Focused tests cover concurrent memoization, failure propagation, fresh-run retry, non-Node startup, and package-less project discovery.
  • Hosted gates: 5 successful checks. CI, verified-secret scanning, two CodeQL analysis jobs, and the aggregate CodeQL check pass on the exact head.

Root-cause cluster
Relationship: fixed_by_candidate
Canonical: #155
Summary: This PR is the direct candidate fix for the canonical linked eager Node discovery issue.

Members:

Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything.

Merge readiness
Overall: 🦞 diamond lobster
Proof: 🦞 diamond lobster
Patch quality: 🦞 diamond lobster
Result: ready for maintainer review.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Next step before merge

  • [P2] No repair work remains; the exact head is ready for ordinary maintainer merge handling.

Security
Cleared: The diff is confined to local mapper orchestration and tests and introduces no dependency, workflow, secret, publishing, or downloaded-code risk.

Review details

Best possible solution:

Merge the current head so non-Node mapping starts immediately while Node-family consumers retain one shared, retryable discovery and task-graph context per mapping run.

Do we have a high-confidence way to reproduce the issue?

Yes. Current main directly shows both Node loaders completing before the mapper Promise.all begins, and the PR’s instrumentation demonstrates the resulting unnecessary calls on non-Node repositories.

Is this the best way to solve the issue?

Yes. A per-run memoized context plus a bounded Node-family eligibility gate is narrower and safer than duplicating loaders or changing individual non-Node mappers, and it preserves shared concurrency and retry semantics.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against 61ddfe0df86f.

Label changes

Label changes:

  • add proof: sufficient: Contributor real behavior proof is sufficient. Exact-head proof includes packaged CLI execution, instrumented zero-call and single-call behavior, public Go and Python repository runs, failure/retry validation, and alternating timing measurements.
  • add rating: 🦞 diamond lobster: Overall readiness is 🦞 diamond lobster; proof is 🦞 diamond lobster and patch quality is 🦞 diamond lobster.
  • add status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Sufficient (live_output): Exact-head proof includes packaged CLI execution, instrumented zero-call and single-call behavior, public Go and Python repository runs, failure/retry validation, and alternating timing measurements.

Label justifications:

  • P2: This is a normal-priority mapper performance bug with limited blast radius and strong regression and runtime validation.
  • rating: 🦞 diamond lobster: Overall readiness is 🦞 diamond lobster; proof is 🦞 diamond lobster and patch quality is 🦞 diamond lobster.
  • status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Sufficient (live_output): Exact-head proof includes packaged CLI execution, instrumented zero-call and single-call behavior, public Go and Python repository runs, failure/retry validation, and alternating timing measurements.
  • proof: sufficient: Contributor real behavior proof is sufficient. Exact-head proof includes packaged CLI execution, instrumented zero-call and single-call behavior, public Go and Python repository runs, failure/retry validation, and alternating timing measurements.
Evidence reviewed

What I checked:

  • Current-main reproduction: Current main unconditionally awaits discoverNodeProjects and turboTaskGraph before starting any feature mapper, matching the reported performance problem. (src/mapper.ts:288, 61ddfe0df86f)
  • Non-Node startup independence: On the PR head, only mappers marked usesNodeContext await the shared eligibility promise; other language mappers enter their map functions without waiting for fallback Node scanning. (src/mapper.ts:298, bda80c546507)
  • Lazy concurrency and failure behavior: The new per-run context memoizes project discovery and task-graph construction, sharing concurrent access and failures while allowing a fresh mapping run to retry. (src/mappers/context.ts:10, bda80c546507)
  • Fallback preservation: The eligibility helper checks detected JavaScript/TypeScript or Node package managers, nested package manifests, conventional generic roots, and bounded nested Nx project files; focused tests preserve package-less conventional and non-conventional Nx projects. (src/mappers/projects.ts:129, bda80c546507)
  • Re-review continuity: The only change after the previously reviewed 7cecde2 head removes the release-owned changelog entry, resolving the remaining P3 finding without altering implementation code. (CHANGELOG.md:1, bda80c546507)
  • Exact-head validation: The PR discussion supplies packaged CLI and public-repository runs, loader instrumentation, failure/retry proof, and timings; GitHub reports the exact head mergeable with CI, verified-secret scanning, and CodeQL successful. (bda80c546507)

Likely related people:

  • steipete: Git and GitHub history connect steipete/Peter Steinberger to the original mapper orchestration, subsequent mapper refactors, and all three repair commits on the current PR head. (role: introduced behavior and recent area contributor; confidence: high; commits: 0cd24d07a262, 57aa23b6c5b8, f9a3b30117f5; files: src/mapper.ts, src/mappers/projects.ts, src/mapper.test.ts)
What the crustacean ranks mean
  • 🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
  • 🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
  • 🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
  • 🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
  • 🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
  • 🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
  • 🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works
  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.
Review history (3 earlier review cycles)
  • reviewed 2026-07-12T07:03:46.260Z sha e088649 :: needs real behavior proof before merge. :: [P1] Prevent Node mappers from requesting discovery on non-Node repos | [P2] Make the pure-Go test require zero discovery calls
  • reviewed 2026-07-12T13:42:54.663Z sha 7cecde2 :: needs changes before merge. :: [P3] Remove the release-owned changelog entry
  • reviewed 2026-07-12T13:46:54.702Z sha 7cecde2 :: needs changes before merge. :: [P3] Remove the release-owned changelog entry

steipete added 2 commits July 12, 2026 06:25
Keep contributor's lazy-context direction while gating Node-family mappers behind bounded project signals and preserving package-less workspace discovery.\n\nCo-authored-by: Tanmay-008 <tanmayshirbhayye60@gmail.com>
Match the existing bounded project.json discovery scope when deciding whether Node-family mappers are needed.
@steipete

Copy link
Copy Markdown
Contributor

Maintainer repair and exact-head proof for 7cecde2a11bb3e06832f4407ba6308c541ff1a47.

The contributor's lazy-context direction is preserved. The repair prevents all four Node-family mappers from requesting shared Node/Turbo context for pure non-Node repositories, while retaining bounded package-less conventional, nested React, and Nx discovery. Concurrent Node consumers share one project-discovery promise and one task-graph promise; failures are isolated to one mapping run.

Proof on the exact head:

  • pnpm typecheck: pass
  • pnpm lint: pass
  • pnpm format:check: pass
  • pnpm test -- --silent: 887 passed, 1 intentional skip
  • pnpm build: pass
  • pnpm pack:smoke: packaged CLI mapped 13 features
  • Focused instrumentation: pure Go and Python fixtures invoked Node project discovery 0 times and Turbo graph construction 0 times; concurrent Node consumers invoked each loader exactly once
  • Built CLI against public representative repositories: openclaw/gitcrawl mapped 25 Go features with 0 Node titles; yt-dlp/yt-dlp mapped 121 Python features with 0 Node titles
  • Built fixture proof: package-less conventional Node and package-less Nx projects mapped; Turbo validation command preserved; invalid Turbo JSON failed nonzero and a corrected fresh run succeeded
  • Alternating 10-run current-main comparison: median mapping time improved 203.34 ms to 196.04 ms on the Go repository (3.6%) and 801.80 ms to 792.11 ms on the Python repository (1.2%)
  • pnpm audit --json: 0 advisories and 0 vulnerabilities
  • pnpm outdated --format json: 0 outdated packages
  • AutoReview: accepted one package-less Nx scope finding, fixed in 7cecde2, then clean with no actionable findings
  • Public Model Identifier Gate: PASS; no model identifiers in added lines, changed built artifacts, or this proof text

Risk: medium because mapper eligibility changes across language families; mitigated by concurrency, failure, retry, multi-root, package-less Node/Nx, Turbo, full-suite, packaged-CLI, and real-repository proof.

This head resolves the two prior ClawSweeper findings and supplies the requested runtime/timing proof.

@clawsweeper re-review

@clawsweeper

clawsweeper Bot commented Jul 12, 2026

Copy link
Copy Markdown
Contributor

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

Re-review progress:

@clawsweeper clawsweeper Bot added proof: sufficient Contributor real behavior proof is sufficient. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. and removed rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. labels Jul 12, 2026
@steipete

Copy link
Copy Markdown
Contributor

Maintainer repair and exact-head proof for bda80c546507ff0e20a05758d3171b45d7e925d9.

The contributor's lazy-context direction is preserved. The repair prevents all four Node-family mappers from requesting shared Node/Turbo context for pure non-Node repositories, while retaining bounded package-less conventional, nested React, and Nx discovery. Concurrent Node consumers share one project-discovery promise and one task-graph promise; failures are isolated to one mapping run.

Proof on the exact head:

  • pnpm typecheck: pass
  • pnpm lint: pass
  • pnpm format:check: pass
  • pnpm test -- --silent: 887 passed, 1 intentional skip
  • pnpm build: pass
  • pnpm pack:smoke: packaged CLI mapped 13 features
  • Focused instrumentation: pure Go and Python fixtures invoked Node project discovery 0 times and Turbo graph construction 0 times; concurrent Node consumers invoked each loader exactly once
  • Built CLI against public representative repositories: openclaw/gitcrawl mapped 25 Go features with 0 Node titles; yt-dlp/yt-dlp mapped 121 Python features with 0 Node titles
  • Built fixture proof: package-less conventional Node and package-less Nx projects mapped; Turbo validation command preserved; invalid Turbo JSON failed nonzero and a corrected fresh run succeeded
  • Alternating 10-run current-main comparison: median mapping time improved 203.34 ms to 196.04 ms on the Go repository (3.6%) and 801.80 ms to 792.11 ms on the Python repository (1.2%)
  • pnpm audit --json: 0 advisories and 0 vulnerabilities
  • pnpm outdated --format json: 0 outdated packages
  • AutoReview: accepted one package-less Nx scope finding, fixed in 7cecde2, then clean with no actionable findings
  • Exact-head hosted CI, build, tests, and package smoke: https://github.com/openclaw/clawpatch/actions/runs/29195119721
  • Exact-head verified-secret scan: https://github.com/openclaw/clawpatch/actions/runs/29195119672
  • Exact-head CodeQL analysis: https://github.com/openclaw/clawpatch/actions/runs/29195119726
  • Public Model Identifier Gate: PASS; no model identifiers in added lines, changed built artifacts, or this proof text

Risk: medium because mapper eligibility changes across language families; mitigated by concurrency, failure, retry, multi-root, package-less Node/Nx, Turbo, full-suite, packaged-CLI, and real-repository proof.

This head resolves the prior code/proof findings and the release-owned changelog cleanup. Release-note context and @Tanmay-008 credit remain in the PR body for the release process; the contributor's original commit remains first in the branch.

@clawsweeper re-review

@clawsweeper

clawsweeper Bot commented Jul 12, 2026

Copy link
Copy Markdown
Contributor

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

Re-review progress:

@clawsweeper clawsweeper Bot added proof: sufficient Contributor real behavior proof is sufficient. rating: 🦞 diamond lobster Very strong PR readiness with only minor maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. and removed rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. proof: sufficient Contributor real behavior proof is sufficient. labels Jul 12, 2026
@steipete steipete merged commit 69014be into openclaw:main Jul 12, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

P2 Normal priority bug or improvement with limited blast radius. proof: sufficient Contributor real behavior proof is sufficient. rating: 🦞 diamond lobster Very strong PR readiness with only minor maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug/Perf] Node-centric coupling in collectSeeds causes unnecessary I/O overhead on non-Node projects

2 participants