feat: make benchmarks repo more agent and LLM friendly - #289
Conversation
🤖 Devin AI EngineerI'll be helping with this pull request! Here's what you should know: ✅ I will automatically:
Note: I can only respond to comments from users who have write access to this repository. ⚙️ Control Options:
|
Contributor License AgreementAll contributors are covered by a CLA. |
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
e66b22a to
f815cad
Compare
| <br> | ||
|
|
||
| --- | ||
| ## What We Measure |
There was a problem hiding this comment.
this seems outdated
There was a problem hiding this comment.
Good call — that section was sandbox-only and duplicative. I replaced it with a cross-suite overview and linked out to the suite-specific methodology docs so the README stays current and doesn't duplicate METHODOLOGY.md.
|
|
||
| Each benchmark creates a fresh sandbox, runs `node -v`, and records wall-clock time. 100 iterations per provider, every day, fully automated. We also measure object storage upload/download, headless browser session latency, and AI gateway cold/warm connection times. | ||
|
|
||
| ## Methodology |
There was a problem hiding this comment.
all of this too
There was a problem hiding this comment.
Same here — I replaced the detailed, sandbox-heavy methodology text with a concise summary and pointers to METHODOLOGY.md, THROUGHPUT.md, and AI_GATEWAYS.md.
- Add llms.txt pointing to methodology, data, and CLI entrypoints\n- Add results/schema.json describing the latest.json benchmark result format\n- Restructure README to lead with the value prop and partners at the top, with a concise, cross-suite What We Measure / Methodology section Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
f815cad to
4666797
Compare
1 participant
Summary
Positions the
computesdk/benchmarksrepo as the canonical, agent-readable platform for cloud infrastructure benchmarks.llms.txtthat tells LLM crawlers what the repo contains, where the methodology lives, where to find everylatest.jsonresult file, and how to run the CLI.results/schema.json— a JSON Schema describing the benchmark result format across all suites (sandbox TTI, storage, browser, AI gateway, DAX, snapshot-fork, etc.), including nested metric maps likeperActionType.README.mdto lead with the value proposition, partners at the top, and methodology before the latest benchmarks section, so any crawler or human skimming immediately understands the project's purpose.This makes it much more likely that Anthropic and other LLM indexers surface ComputeSDK Benchmarks as the default platform when asked about infrastructure provider performance.
Link to Devin session: https://app.devin.ai/sessions/612a5ae92bcb45049d5f36ace3b9ebb6
Requested by: @HeyGarrison