perf(es): lazy-load ES schemas per endpoint to cut startup heap (#171)

[MattDevy]

Summary

Fixes #171 by switching elastic stack es ... to per-endpoint lazy schema loading, using the per-endpoint Zod layout from elastic/elastic-client-generator-js#164 (now merged).

Scenario	Before (#171)	After
apis.ts import heap delta	373 MB	0.27 MB
elastic stack es --help RSS	599 MB	102 MB
elastic stack es bulk --help RSS	~599 MB	165 MB
elastic stack es search --help RSS	~599 MB	166 MB
elastic stack es indices create RSS	~599 MB	216 MB

Per-leaf deltas match the spot-check table in elastic/elastic-client-generator-js#164 (bulk 34 MB, search 36 MB, ml.get-job-stats 0.9 MB, cat.indices 10 MB).

Review-ability

Split into two commits:

1. perf(es): lazy-load ES schemas per endpoint to cut startup heap (#171) - the architecture change. 7 files, +462/-148. This is what reviewers should read.
2. chore(es): regenerate API surface with per-endpoint Zod layout - the regenerated output (842 new files, 64 deleted). Review it by running the pipeline documented in the commit body.

How it works

• src/es/apis.ts is no longer an eager barrel. It re-exports the metadata-only apiManifest (294 entries, no Zod refs) and adds loadEsApi() / loadEsApisInFile() for dynamic-import on demand.
• src/es/register.ts splits into a sync registerEsCommands(defs) (explicit definitions, used by tests) and an async registerEsCommandsLazy() (production). The lazy path sniffs process.argv to identify the invoked leaf, pre-loads only that endpoint's schema so Commander can register its flags before parsing, and registers every other leaf as a stub.
• Stubs lazy-load on demand if the sniff misses (swap-and-reparse), as defence in depth for any argv shape I haven't explicitly covered.
• Three scripts/*.mjs post-process generator output: prepend @ts-nocheck to 588 schema files (fixes a tsc OOM at 8 GB), split namespace-grouped api files into per-endpoint, build the manifest. They live in this repo for now; elastic/elastic-client-generator-js#166 tracks folding them into the generator.

Behavioural notes for reviewers

• registerEsCommands() no longer has a default argument. Callers must pass an explicit EsApiDefinition[] or switch to registerEsCommandsLazy(). Only one test was affected; it now uses loadAllEsApis().
• All 294 ES commands still appear under elastic stack es --help with the same names, namespaces, and descriptions. The only user-visible change is that top-level help is now fast and light.
• Response format, flag shape, validation errors, and JSON-Schema --help --json output are unchanged for every leaf.

Test plan

• tsc --noEmit clean (8 GB heap; would OOM without @ts-nocheck on schemas)
• npm test passes 867/867
• elastic stack es --help, elastic stack es bulk --help, elastic stack es indices --help, elastic stack es indices create --help, elastic stack es search --help all render correct flags and descriptions
• RSS measured with /usr/bin/time -lp for every case in the table above
• Reviewer: confirm functional tests still pass (npm run test:functional:es) against a live ES. I did not run them.
• Reviewer: spot-check a few leaf invocations against a real cluster (not just --help).

Follow-ups

• elastic/elastic-client-generator-js#166 moves the three post-processing scripts upstream so every consumer of --layout per-endpoint gets ts-nocheck, per-endpoint api files, and a manifest natively.
• Option A (ajv-standalone, median 0.08 MB per endpoint) is deliberately NOT done here. The per-endpoint Zod numbers are sufficient for perf: loading 292 ES API Zod schemas allocates 373 MB of heap #171, and ajv would need handler-path changes. Revisit if docs chat or a future long-running mode is still heap-constrained.

Closes #171.

[github-actions]

✅MegaLinter analysis: Success

Descriptor	Linter	Files	Fixed	Errors	Warnings	Elapsed time
✅ ACTION	actionlint	1		0	0	0.04s
✅ BASH	shellcheck	2		0	0	0.07s
✅ COPYPASTE	jscpd	yes		no	no	10.96s
✅ REPOSITORY	gitleaks	yes		no	no	81.43s
✅ REPOSITORY	git_diff	yes		no	no	0.36s
✅ REPOSITORY	secretlint	yes		no	no	13.43s
✅ REPOSITORY	trivy	yes		no	no	19.36s
✅ TYPESCRIPT	eslint	301		0	0	5.89s
✅ YAML	yamllint	1		0	0	0.76s

See detailed reports in MegaLinter artifacts
Set VALIDATE_ALL_CODEBASE: true in mega-linter.yml to validate all sources, not only the diff

MegaLinter is graciously provided by OX Security
Show us your support by starring ⭐ the repository

[MattDevy]

Binary size impact

Flagging this up-front: the memory win comes with a real package-size regression that deserves review attention.

Measurements (clean builds, main vs this branch)

Metric	Main	PR	Multiplier
dist/ total	23 MB	152 MB	6.6×
npm tarball (compressed)	2.4 MB	21.9 MB	9.1×
npm package unpacked	25.7 MB	196.1 MB	7.6×

Where the bytes live in dist/ (PR branch):

File type	Main	PR
.js	3 MB	35 MB
.d.ts	15 MB	72 MB
.js.map	2 MB	26 MB
.d.ts.map	1 MB	17 MB

Nearly all of it (146 MB of the 152 MB dist/) is src/es/apis/schemas/. 588 per-endpoint files, each with its transitive type closure inlined and compiled once per file. Source maps and declaration maps balloon disproportionately because Zod's generic types expand into a lot of detail.

Mitigation options, sized

Strategy	Unpacked	Complexity
Current (ships src/ + full dist/)	196 MB	baseline
Drop src/ from files[] in package.json	152 MB	one line
Also drop *.map files from the package	108 MB	npmignore / files glob
Also drop *.d.ts* under dist/es/apis/schemas/	40 MB	postbuild rm. No public-API loss - schemas carry @ts-nocheck so their external types are already any
Drop all .d.ts* from dist/	36 MB	loses public-API types

The 40 MB target is the natural stopping point: public-API types preserved, internal schema artefacts stripped. It extends the "schemas are codegen, consumers don't introspect their types" stance that @ts-nocheck already assumes.

Proposed handling

Not on this PR, unless reviewers disagree. #171 is framed as a memory issue; the size regression is a side effect that deserves its own review cycle and its own commit history. Plan:

1. Merge this PR as-is. The memory fix stands on its own.
2. Open a follow-up with:
   • Strip src/ from files[].
   • Add a postbuild step (rm dist/es/apis/schemas/*.d.ts dist/es/apis/schemas/*.d.ts.map dist/es/apis/schemas/*.js.map).
   • Result: 196 MB → 40 MB unpacked, 21.9 MB → ~5 MB tarball. Still ~2× over today, but the rest is genuinely new content (294 inlined closures) that only Option A (ajv-standalone from elastic/elastic-client-generator-js#164) would compress further.

Happy to fold the mitigation into this PR as a third commit if reviewers want the size win in the same merge. It would delay merge slightly because changing files[] needs checking against supply-chain scanners, npm view tooling, and GitHub's Dependency Graph, which I haven't audited.

[JoshMock]

Happy to fold the mitigation into this PR as a third commit if reviewers want the size win in the same merge.

Let's just open an issue for package size mitigation. It's a real concern, but you've already mapped a clear path to solving it and reviewing separately will be much easier.

[MattDevy]

Happy to fold the mitigation into this PR as a third commit if reviewers want the size win in the same merge.

Let's just open an issue for package size mitigation. It's a real concern, but you've already mapped a clear path to solving it and reviewing separately will be much easier.

Thanks for the input Josh! I'll get this tidied up and ready to merge first thing tomorrow

[MattDevy]

Opened #250 to track the package size mitigation.