Interested in contributing to OracleTrace — would love to understand the long-term vision

[AnshBajpai05]

Hi Kay,

I've been spending the last few days exploring OracleTrace and honestly I've been enjoying understanding how you've kept the project lightweight while still solving a very practical problem.

While reading through the codebase I naturally started taking engineering notes and thinking about possible improvements. A few themes that kept coming up were things like:

• making run work with arbitrary Python commands instead of only pytest
• richer reporting (especially Markdown for CI and PRs)
• a cleaner separation between comparison logic and reporters to make future outputs easier to add
• better visualization of comparisons while still keeping the project lightweight

I have quite a few more thoughts, but before implementing anything I'd really like to understand where you see OracleTrace going.

I'd much rather contribute in a way that strengthens your vision than accidentally push the project in a direction you don't want.

[KaykCaputo]

Hi @AnshBajpai05 ! First of all, sorry for the late reply.

Thank you for taking the time to explore OracleTrace and for sharing your thoughts. I'm really glad to hear you've been enjoying the project, and I truly appreciate the care you've put into understanding the codebase before suggesting changes.

Please feel free to keep contributing and bringing new ideas. We really value discussions and community contributions—OracleTrace is built with the community, and feedback like yours is exactly what helps the project improve.

I also appreciate you wanting to align your contributions with the project's vision. I'd be happy to discuss your ideas, and from what you've described so far, they all sound like interesting directions to explore. Looking forward to hearing more of your thoughts!

[KaykCaputo]

As for the project's direction, I definitely see value in using OracleTrace locally to investigate and measure performance regressions. It's a useful workflow during development, and I want to continue supporting it.

That said, I see OracleTrace as a CI/CD-first tool. The long-term goal is to make it lightweight, reliable, and easy to integrate into automated pipelines. Features like those discussed in issue #72 are very much aligned with that vision.

Ultimately, I'd like OracleTrace to become the default performance guard for Python projects where performance is critical. That means prioritizing precision, reliability, and seamless CI/CD integration above everything else.

At the same time, I do see value in providing good visualizations for users. That's why we already support JSON, CSV, and HTML exports, and I also plan to extend the HTML report with features such as call graphs and flame graphs. I think these kinds of visualizations complement the CI/CD workflow well without shifting the project's primary focus.

Of course, that vision isn't set in stone. If the community identifies a better direction while keeping OracleTrace lightweight and reliable, I'm always open to discussing it.

[KaykCaputo]

Regarding your specific ideas:

• Supporting arbitrary Python commands instead of only pytest is something I can definitely see value in, as long as we can keep the user experience simple.
• Richer reporting, especially Markdown for CI and pull requests, fits very well with the CI/CD-first direction.
• A cleaner separation between the comparison engine and reporters is also something I think would improve maintainability and make new output formats easier to add.
• As for visualizations, I absolutely want to keep improving them. The current JSON, CSV, and HTML exports are just the beginning, and I already have plans to add features like call graphs and flame graphs to the HTML report.

So I think we're very much aligned. My main priority is making OracleTrace the most precise and reliable performance guard for Python CI/CD pipelines, and I'd love to evolve it together with the community.

[AnshBajpai05]

Thanks Kayk — great to hear we're aligned. "The most precise and reliable performance guard for Python CI/CD pipelines" is exactly the direction I've been building toward on my fork, so let me lay everything out concretely so you can pick and reorder without guessing at what anything means.

The three you endorsed — how I'd build them, in order

1. Universal run (next PR)

Today oracletrace run only accepts pytest. I'd extend it to any Python entry point, same one-command UX:

oracletrace run -- python app.py --port 8000
oracletrace run -- python -m mypackage.job
oracletrace run -- pytest tests/ -k hot_path
oracletrace run -- app.py            # bare script shorthand

Key design decision: everything runs in-process (runpy.run_path / runpy.run_module / pytest.main) rather than as a subprocess — sys.setprofile can't see into a child process, so in-process is the only way the tracer keeps working. Non-Python commands (node, cargo, …) get rejected with a clear error message explaining why. All existing flags (--json, --compare, --fail-on-regression, …) work unchanged.

Size: roughly a new runner.py (~120 lines) + dispatch change in cli.py + tests for each command form, including failure paths. No new dependencies.

2. Comparison engine / reporter separation

Right now compare_traces() prints results while computing them. I'd make it pure — it returns a ComparisonData object holding per-function deltas and statuses (regression / improvement / stable / new / removed) — and move all rich printing into a small display module. Behavior and output stay byte-identical; this is a pure refactor with no new flags.

Why before the Markdown reporter: once comparison results are data, every output format (Markdown, JUnit, whatever comes later) becomes a ~100-line reporter module that consumes ComparisonData, instead of each format re-implementing comparison logic. It's the enabler for the whole "richer reporting" direction.

3. Markdown report for CI/PRs

--markdown report.md produces a PR-comment-ready report: summary table (top functions, time, calls), and when --compare is active, a comparison section with 🔴/🟢 deltas and a regression verdict line. Designed so gh pr comment --body-file report.md just works. Builds directly on the split from step 2.

Already working on my fork — the menu

These all exist with tests on my fork; each would come as its own focused PR. Ordered by how directly I think they serve the CI-guard goal — but that's your call:

a) Noise-resistant verdicts (statistical rigor for --repeat)
The biggest reliability lever, in my experience. With --repeat 5, each function gets median + mean/stddev/95% confidence interval across runs. Then two things happen at compare time:

• a delta that lies within the combined confidence intervals of both runs is reported as stable (within noise) instead of failing the build — this is what stops the 3 a.m. "perf gate failed, rerun fixed it" pages;
• a real regression gets a Welch's t-test significance tag (p<0.01, p<0.001) so the report says how sure it is.

Companions: --warmup N (run N times before measuring — cold imports/caches don't pollute run 1) and --outlier-policy keep|flag|drop (Tukey-fence handling for the one run where the CI runner stalled).

b) Git-native baselines
oracletrace baseline save attaches the trace JSON to the current commit as a git note (refs/notes/oracletrace); --compare auto walks ancestors and picks the nearest one carrying a baseline. Result: zero baseline files passed around in CI — the repo itself carries them, and they travel with git push origin 'refs/notes/*'. Complements (doesn't replace) the existing file-based --compare.

c) Performance budgets
A config table of absolute limits — "this function must stay under 250ms", "total wall time under 1.5s" — checked on every run, with a distinct exit code. Catches "it's just slow" even when there's no baseline to diff against, e.g. on first runs or fresh branches.

d) oracletrace bisect
oracletrace bisect --good v1.2.0 --bad HEAD -- python bench.py binary-searches the commit range, tracing at each probe, and names the exact commit that introduced the regression (with the offending functions). Basically git bisect where the test is "did it get slower". Skips commits that fail to run, like git bisect skip.

e) pytest plugin
A pytest11 entry point so existing suites opt in with pytest --oracletrace --oracletrace-compare base.json — no invocation changes, per-test attribution, exit-code gating that plays nicely with pytest's own.

f) Composite GitHub Action
uses: KaykCaputo/oracletrace@v1 style: traces the target on the PR, writes the report to the job summary, optionally comments the Markdown report on the PR, fails the check on regression. Turns everything above into a two-line CI integration.

g) --memory
tracemalloc-based tracking alongside timings: net bytes retained + allocation peak per function, shown in the table and in comparisons (informational, never gates). Useful when a "perf regression" is actually a memory blow-up.

On visualizations

I also have a working flame graph (self-contained HTML, no CDN): the tracer records cumulative time per caller→callee edge, so flame widths are real measured attribution, not call-count estimates. Same data powers a call-graph view. Since you said you already have plans for the HTML report, I don't want to collide with them — tell me what you have in mind and I'll either adapt what I have to fit or stay out of that lane entirely.

So, concretely, three questions

1. OK with the order universal run → compare/reporter split → Markdown report? (The split before Markdown is the only ordering I'd argue for; the rest is flexible.)
2. From the menu (a–g): which ones do you want at all, and in what priority? Happy to hear "none for now" on any of them.
3. For the flame graph / HTML plans — want to share your direction so I can align?

Whatever you pick, the delivery style stays what it's been: one focused PR at a time, each with tests, nothing merged-by-surprise.

[KaykCaputo]

Thank you for taking the time to write all of this up. I really appreciate how much thought you've put into both the technical design and the order of the work. This kind of discussion is exactly what makes open source projects better.

I completely agree with the order you proposed: universal "run" → comparison engine/reporter separation → Markdown reporting. Out of those, I'm especially excited about the comparison engine/reporter separation. I think having a clean separation between the comparison logic and the output layer will make OracleTrace much easier to evolve while keeping the core simple and maintainable.

Regarding the items on your menu, I honestly think they're all valuable and align well with OracleTrace's long-term vision. The order you've suggested also makes a lot of sense to me, as it prioritizes improvements that directly strengthen OracleTrace as a reliable CI/CD performance guard.

As for the flame graph, please go for it! It was an idea I had noted down, but I don't have a concrete implementation or design in mind yet. If you already have a working implementation that fits the project's philosophy of staying lightweight and CI/CD-focused, I'd be very happy to see it evolve as part of OracleTrace.

Once again, thank you for investing so much time into the project. I'm really looking forward to seeing these ideas come to life.