optimize(generate): speed up make generate flow - #23519
Conversation
|
👋 kalverra, thanks for creating this pull request! To help reviewers, please consider creating future PRs as drafts first. This allows you to self-review and make any final changes before notifying the team. Once you're ready, you can mark it as "Ready for review" to request feedback. Thanks! |
|
✅ No conflicts with other open PRs targeting |
There was a problem hiding this comment.
Pull request overview
Risk Rating: MEDIUM — Changes affect developer/CI tooling (make generate, rm-mocked) and githook-driven codegen triggers. Failures here can block local workflows and CI generation steps.
This PR focuses on speeding up the make rm-mocked + make generate workflow by reducing unnecessary scanning/work and adding smarter trigger conditions for generation (including githooks parity).
Changes:
- Update githooks generation triggers to also react to
//go:generatedirectives in changed.gofiles and tooperator_ui/TAGchanges (triggering./core/webgeneration). - Optimize
make generateandrm-mockedby usingrgwhen available (with prunedfindfallback) and by caching/install-guarding tool prerequisites (plus pinned mockery version checks). - Run mockery configs concurrently via
xargsparallelism.
Scrupulous human review recommended (tooling correctness/portability):
GNUmakefilegenerate/rm-mockedpipelines and their macOS/BSD portability (notablyxargsbehavior,-P0, and argument forms passed togo generate).
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
| tools/githooks/internal/generate/generate.go | Adds trigger logic for operator_ui/TAG and for changed .go files containing //go:generate. |
| tools/githooks/internal/generate/generate_test.go | Adds test coverage for the new githooks triggers. |
| GNUmakefile | Reworks make generate/rm-mocked to speed up discovery/execution and to reduce redundant tool installs. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
|
When I run |
The stats in the PR description were using a cold cache for "Before", but a warm cache for "After", which made the original approach look much worse than it was in a fair fight. Have updated to comparisons. |
|
Should this speedup translate to CI as well? Latest develop vs. this PR looks like a slowdown from 4m to 5m: |
|
Yes, it translates to CI, but not a whole lot. The bigger advantages were actually in my latest commits when I moved things over from an expensive GitHub Actions runner to use 
|
Intent
Speed up the
make generateflow locally and in CIBig Changes
Pruned
make rm-mockedSearchSwitches mock cleanup to use
rg(with a prunedfindfallback) targeting only.gofiles while excluding.git,node_modules,.yarn, andvendordirectories, piping directly toxargs rm -f.The previous
grep -rlunselectively crawled git object packs, node modules, and caches, taking >30s. Pruning unneeded paths cuts execution time to <0.2s.Exact Tool Version Checks Before Install
Adds binary presence and version verification (
go version -mor--version) forgomods,mockery,codecgen,protoc-gen-go,protoc-gen-go-grpc,protoc-gen-go-wsrpc, andmodgraphbefore invokinggo install.Unconditional
go installcommands query module proxies and run build pipelines even when binaries already exist, creating 5–10s+ of cumulative latency on everymake generaterun. Checking binary metadata locally skips rebuilds in <1ms.Concurrent Mockery Generation
Finds
.mockery.yamlconfiguration files while pruning ignored directories and executes mockery across configurations concurrently usingxargs -n1 -P0.Running mockery in parallel across the root and submodule configurations reduces wall-clock execution time during full repo code generation.
Githook Directive Detection & Optimized Execution
Enhances
tools/githooks/internal/generatewith streaming//go:generatedirective detection, early non-code file filtering, deduplicated module discovery,MOCKERY_BINresolution, and boundederrgroupconcurrency.Ensures git pre-commit hooks detect changes requiring code generation accurately without wasteful full-file memory buffering or unbounded subprocess spawning.
Small Changes
tools/githooks/internal/generate/generate_test.gocovering//go:generatedirective detection, non-code asset skipping, parallel execution, andoperator_ui/TAGtrigger handling.MOCKERY_VERSION = 2.53.6variable inGNUmakefilefor consistent reference across commands and version checks.modgraphfrom the defaultgeneratetarget dependencies. This is redundant.Results
Local - Cold Cache
develop)rgmode)findfallback)rg)find)make rm-mockedmockery,protoc,gomods, etc.)go generate.&deployment/)make rm-mocked+make generateflowLocal - Warm Cache
develop)rgmode)findfallback)rg)find)make rm-mockedmockery,protoc,gomods, etc.)go generate.&deployment/)make rm-mocked+make generateflowCI