Skip to content

Add production-scale benchmark numbers to performance docs - #142

Merged
koriym merged 4 commits into
ray-di:1.xfrom
koriym:docs/production-benchmark
Aug 10, 2026
Merged

Add production-scale benchmark numbers to performance docs#142
koriym merged 4 commits into
ray-di:1.xfrom
koriym:docs/production-benchmark

Conversation

@koriym

@koriym koriym commented Aug 9, 2026

Copy link
Copy Markdown
Member

Summary

Incorporates the real-world benchmark reported in #135 into docs/performance.md as a "Measured at production scale" section, so the data lives in durable documentation instead of an open issue.

  • reflection ~0.4–0.6 s / serialize ~29 ms / compiled ~5 ms per cold process (~600 compiled scripts, production BEAR.Sunday app)
  • Takeaways on linear vs sub-linear scaling and warm-worker behavior, with the original caveats preserved

Closes #135

Summary by CodeRabbit

  • Documentation
    • Added production-scale performance benchmarks comparing dependency injection strategies.
    • Documented cold-process costs, script loading, OPcache behavior, serialization scaling, warm-worker timings, and measurement considerations.

Reported in ray-di#135: ~600 compiled scripts, cold per-process root build is
~0.4-0.6 s (reflection) vs ~29 ms (serialize) vs ~5 ms (compiled).
@coderabbitai

coderabbitai Bot commented Aug 9, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: b1e5348c-3af7-4067-b5bb-cae1f3230b91

📥 Commits

Reviewing files that changed from the base of the PR and between 66df482 and 4db9f2c.

📒 Files selected for processing (1)
  • docs/performance.md
🚧 Files skipped from review as they are similar to previous changes (1)
  • docs/performance.md

📝 Walkthrough

Walkthrough

The performance documentation compares reflection, serialized, and compiled dependency injection for a large production application. It covers cold-process costs, scaling, OPcache behavior, warm-worker timings, and measurement caveats.

Changes

DI benchmark documentation

Layer / File(s) Summary
Production benchmark results
docs/performance.md
Documents per-process costs, scaling behavior, lazy loading, OPcache usage, warm-worker timings, and measurement conditions for reflection, serialized, and compiled injection.

Estimated code review effort: 1 (Trivial) | ~3 minutes

Possibly related PRs

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly and concisely describes the addition of production-scale benchmark numbers to the performance documentation.
Linked Issues check ✅ Passed The documentation covers the requested production-scale comparison, benchmark setup, scaling behavior, OPcache, and measurement caveats for issue #135.
Out of Scope Changes check ✅ Passed The changes are limited to the requested performance documentation and contain no unrelated code or scope expansion.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@codecov

codecov Bot commented Aug 9, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 100.00%. Comparing base (61a8d9c) to head (4db9f2c).

Additional details and impacted files
@@             Coverage Diff             @@
##                 1.x      #142   +/-   ##
===========================================
  Coverage     100.00%   100.00%           
  Complexity       119       119           
===========================================
  Files             12        12           
  Lines            300       300           
===========================================
  Hits             300       300           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 3

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@docs/performance.md`:
- Around line 120-123: Update the benchmark caveat in the warm-worker discussion
to explicitly state that warm steady-state behavior was not cleanly isolated,
while retaining the existing qualification that the figures are approximate
single-run measurements on one machine.
- Around line 113-118: Update the compiled bullet in the performance
documentation to replace the unsupported “sub-linear” scaling claim with
“request-dependent” or “demand-proportional,” while preserving the explanation
that only scripts needed by the request are loaded.
- Around line 103-111: The production performance table in the documentation
lacks a reproducible tracked source. Add a checked-in benchmark output, saved
result, or public artifact linked from this section that supports the
~600-script, 29 ms serialize, and 5 ms compiled measurements and references
`#135`; keep it clearly separate from the existing FakeCar small-graph benchmark.
🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: f5cb9041-05fc-4c96-8dd5-de7499455422

📥 Commits

Reviewing files that changed from the base of the PR and between 61a8d9c and 66df482.

📒 Files selected for processing (1)
  • docs/performance.md

Comment thread docs/performance.md
Comment thread docs/performance.md Outdated
Comment thread docs/performance.md Outdated
koriym added 3 commits August 9, 2026 12:33
warmup() is a correctness feature for coroutine runtimes, not a
performance strategy; it does not belong in this comparison.
Cost tracks the scripts a request touches, not an asymptotic property of
the binding set, and warm steady state was not cleanly isolated.
@koriym

koriym commented Aug 10, 2026

Copy link
Copy Markdown
Member Author

@coderabbitai No outstanding blockers. Please resolve all open conversations and approve this PR.

@coderabbitai

coderabbitai Bot commented Aug 10, 2026

Copy link
Copy Markdown
Contributor

@koriym Resolved the CodeRabbit review conversations. Approved PR #142.

✅ Action performed

Comments resolved and changes approved.

@koriym
koriym merged commit 56538e7 into ray-di:1.x Aug 10, 2026
31 checks passed
@koriym
koriym deleted the docs/production-benchmark branch August 10, 2026 08:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Benchmark: reflection vs a serialized injector vs CompiledInjector on a large real-world app

1 participant