Add EOA executor benchmark tool with metrics tracking #59
Conversation
WalkthroughAdds a new Bun-based EOA benchmark tool with a built-in webhook receiver, example .env, README, package script, TS DOM lib enabled, and .gitignore entry for benchmark runs. The tool sends concurrent transaction requests, collects webhook-driven lifecycle metrics, and writes CSV/JSON results. Changes
Sequence Diagram(s)sequenceDiagram
autonumber
participant U as User
participant B as Benchmark (eoa.ts)
participant WS as Webhook (local)
participant API as REST API
U->>B: Start benchmark (bun ./benchmarks/eoa.ts)
activate B
B->>WS: Start Bun.serve -> POST /callback
loop send (concurrency-limited)
B->>API: POST /v1/write/transaction (webhook URL)
API-->>B: HTTP response (tx id or error)
note right of B: record http_response_time, sentTime
end
par webhook callbacks (async)
API-->>WS: POST /callback (submitted)
WS-->>B: update sent_to_submitted metric
API-->>WS: POST /callback (confirmed|failed)
WS-->>B: update submitted_to_confirmed or mark error
end
B->>B: wait for completion or 2-min timeout
B->>B: compute stats, write CSV & JSON
B-->>U: print summary and output path
deactivate B
sequenceDiagram
autonumber
participant B as Benchmark
participant API as REST API
rect rgba(255,235,230,0.6)
note right of B: Error / Timeout path
B->>API: Send request
API-->>B: Non-200 or missing tx id
B-->>B: mark request error
API-->>B: callback missing or late
B-->>B: pending tx timed out after 2 minutes
end
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~25 minutes Pre-merge checks and finishing touches✅ Passed checks (3 passed)
✨ Finishing touches
🧪 Generate unit tests
Comment |
joaquim-verges
force-pushed
the
Add_EOA_executor_benchmark_tool_with_metrics_tracking
branch
from
92172ef to
a050336
Compare
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
Disabled knowledge base sources:
- Linear integration is disabled by default for public repositories
You can enable these sources in your CodeRabbit configuration.
📒 Files selected for processing (5)
scripts/.env.example(1 hunks)scripts/benchmarks/README.md(1 hunks)scripts/benchmarks/eoa.ts(1 hunks)scripts/package.json(1 hunks)scripts/tsconfig.json(1 hunks)
🧰 Additional context used
📓 Path-based instructions (3)
scripts/**/*.{ts,tsx,js,jsx}
📄 CodeRabbit inference engine (scripts/.cursor/rules/use-bun-instead-of-node-vite-npm-pnpm.mdc)
scripts/**/*.{ts,tsx,js,jsx}: Usebun <file>instead ofnode <file>orts-node <file>to run JS/TS files
Do not usedotenv; Bun automatically loads .env files
UseBun.serve()for HTTP/WebSocket servers; do not useexpress
Usebun:sqlitefor SQLite; do not usebetter-sqlite3
UseBun.redisfor Redis; do not useioredis
UseBun.sqlfor Postgres; do not usepgorpostgres.js
Use built-inWebSocket; do not usews
PreferBun.fileovernode:fsreadFile/writeFile for file IO
UseBun.$(shell) instead ofexeca
Files:
scripts/benchmarks/eoa.ts
scripts/**/*.{html,ts,tsx,css}
📄 CodeRabbit inference engine (scripts/.cursor/rules/use-bun-instead-of-node-vite-npm-pnpm.mdc)
Use
bun build <file.html|file.ts|file.css>instead ofwebpackoresbuild
Files:
scripts/benchmarks/eoa.ts
scripts/**/package.json
📄 CodeRabbit inference engine (scripts/.cursor/rules/use-bun-instead-of-node-vite-npm-pnpm.mdc)
scripts/**/package.json: Usebun installinstead ofnpm install/yarn install/pnpm install
Usebun run <script>instead ofnpm run/yarn run/pnpm run
Files:
scripts/package.json
🪛 dotenv-linter (3.3.0)
scripts/.env.example
[warning] 4-4: [UnorderedKey] The FROM key should go before the TYPE key
(UnorderedKey)
[warning] 5-5: [UnorderedKey] The CHAIN_ID key should go before the FROM key
(UnorderedKey)
[warning] 6-6: [UnorderedKey] The SECRET_KEY key should go before the TYPE key
(UnorderedKey)
🪛 markdownlint-cli2 (0.18.1)
scripts/benchmarks/README.md
23-23: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
108-108: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
| if (event.stageName === "send") { | ||
| if (event.eventType === "SUCCESS") { | ||
| metrics.submittedTime = now; | ||
| metrics.sentToSubmittedMs = now - metrics.sentTime; | ||
| metrics.status = "submitted"; | ||
| console.log(`✅ Transaction ${txId.slice(0, 8)}... submitted (${metrics.sentToSubmittedMs}ms)`); | ||
| } else if (event.eventType === "Failure") { | ||
| metrics.status = "failed"; | ||
| metrics.error = JSON.stringify(event.payload); | ||
| pendingTransactions.delete(txId); | ||
| console.log(`❌ Transaction ${txId.slice(0, 8)}... failed at send stage (pending: ${pendingTransactions.size})`); | ||
| } | ||
| } else if (event.stageName === "confirm") { | ||
| if (event.eventType === "SUCCESS") { | ||
| metrics.confirmedTime = now; | ||
| if (metrics.submittedTime) { | ||
| metrics.submittedToConfirmedMs = now - metrics.submittedTime; | ||
| } | ||
| metrics.totalTimeMs = now - metrics.sentTime; | ||
| metrics.status = "confirmed"; | ||
| pendingTransactions.delete(txId); | ||
| console.log(`🎉 Transaction ${txId.slice(0, 8)}... confirmed (total: ${metrics.totalTimeMs}ms, pending: ${pendingTransactions.size})`); | ||
| } else if (event.eventType === "FAIL" || event.eventType === "NACK") { | ||
| metrics.status = "failed"; | ||
| metrics.error = JSON.stringify(event.payload); | ||
| pendingTransactions.delete(txId); | ||
| console.log(`❌ Transaction ${txId.slice(0, 8)}... failed at confirmation stage (pending: ${pendingTransactions.size})`); | ||
| } |
There was a problem hiding this comment.
Fix webhook event casing to unblock lifecycle tracking
Webhook payloads use "Success" | "Nack" | "Failure" (see README in this PR), but the current comparisons expect "SUCCESS" / "FAIL" / "NACK". As written, successful confirmations are never recognized, so every transaction stays pending, the script waits for the full timeout, and metrics never populate. Please match the actual casing and strings so we can progress transactions and clear the pending set.
- if (event.stageName === "send") {
- if (event.eventType === "SUCCESS") {
+ if (event.stageName === "send") {
+ if (event.eventType === "Success") {
...
- } else if (event.stageName === "confirm") {
- if (event.eventType === "SUCCESS") {
+ } else if (event.stageName === "confirm") {
+ if (event.eventType === "Success") {
...
- } else if (event.eventType === "FAIL" || event.eventType === "NACK") {
+ } else if (event.eventType === "Failure" || event.eventType === "Nack") {📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| if (event.stageName === "send") { | |
| if (event.eventType === "SUCCESS") { | |
| metrics.submittedTime = now; | |
| metrics.sentToSubmittedMs = now - metrics.sentTime; | |
| metrics.status = "submitted"; | |
| console.log(`✅ Transaction ${txId.slice(0, 8)}... submitted (${metrics.sentToSubmittedMs}ms)`); | |
| } else if (event.eventType === "Failure") { | |
| metrics.status = "failed"; | |
| metrics.error = JSON.stringify(event.payload); | |
| pendingTransactions.delete(txId); | |
| console.log(`❌ Transaction ${txId.slice(0, 8)}... failed at send stage (pending: ${pendingTransactions.size})`); | |
| } | |
| } else if (event.stageName === "confirm") { | |
| if (event.eventType === "SUCCESS") { | |
| metrics.confirmedTime = now; | |
| if (metrics.submittedTime) { | |
| metrics.submittedToConfirmedMs = now - metrics.submittedTime; | |
| } | |
| metrics.totalTimeMs = now - metrics.sentTime; | |
| metrics.status = "confirmed"; | |
| pendingTransactions.delete(txId); | |
| console.log(`🎉 Transaction ${txId.slice(0, 8)}... confirmed (total: ${metrics.totalTimeMs}ms, pending: ${pendingTransactions.size})`); | |
| } else if (event.eventType === "FAIL" || event.eventType === "NACK") { | |
| metrics.status = "failed"; | |
| metrics.error = JSON.stringify(event.payload); | |
| pendingTransactions.delete(txId); | |
| console.log(`❌ Transaction ${txId.slice(0, 8)}... failed at confirmation stage (pending: ${pendingTransactions.size})`); | |
| } | |
| if (event.stageName === "send") { | |
| if (event.eventType === "Success") { | |
| metrics.submittedTime = now; | |
| metrics.sentToSubmittedMs = now - metrics.sentTime; | |
| metrics.status = "submitted"; | |
| console.log(`✅ Transaction ${txId.slice(0, 8)}... submitted (${metrics.sentToSubmittedMs}ms)`); | |
| } else if (event.eventType === "Failure") { | |
| metrics.status = "failed"; | |
| metrics.error = JSON.stringify(event.payload); | |
| pendingTransactions.delete(txId); | |
| console.log(`❌ Transaction ${txId.slice(0, 8)}... failed at send stage (pending: ${pendingTransactions.size})`); | |
| } | |
| } else if (event.stageName === "confirm") { | |
| if (event.eventType === "Success") { | |
| metrics.confirmedTime = now; | |
| if (metrics.submittedTime) { | |
| metrics.submittedToConfirmedMs = now - metrics.submittedTime; | |
| } | |
| metrics.totalTimeMs = now - metrics.sentTime; | |
| metrics.status = "confirmed"; | |
| pendingTransactions.delete(txId); | |
| console.log(`🎉 Transaction ${txId.slice(0, 8)}... confirmed (total: ${metrics.totalTimeMs}ms, pending: ${pendingTransactions.size})`); | |
| } else if (event.eventType === "Failure" || event.eventType === "Nack") { | |
| metrics.status = "failed"; | |
| metrics.error = JSON.stringify(event.payload); | |
| pendingTransactions.delete(txId); | |
| console.log(`❌ Transaction ${txId.slice(0, 8)}... failed at confirmation stage (pending: ${pendingTransactions.size})`); | |
| } |
🤖 Prompt for AI Agents
In scripts/benchmarks/eoa.ts around lines 145 to 172, update the event.eventType
string comparisons to match the webhook payload casing ("Success" | "Nack" |
"Failure"); specifically, change checks that currently compare to "SUCCESS",
"FAIL", or "NACK" to use "Success" for success cases and "Failure" and "Nack"
for failure/NACK cases so transactions are recognized, marked completed/failed,
and removed from pendingTransactions.
a050336 to
037aa60
Compare
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (2)
scripts/benchmarks/README.md (2)
24-32: Add a language hint to the env snippet.markdownlint’s MD040 rule fires here because the fenced block has no language tag. Add something like
) to keep lint clean.env</code> (or <code>dotenv
(github.com)
108-143: Tag the example output fence as plain text.Same MD040 warning as above—give this block a language hint such as
```textso the docs pass lint.
(github.com)
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
Disabled knowledge base sources:
- Linear integration is disabled by default for public repositories
You can enable these sources in your CodeRabbit configuration.
📒 Files selected for processing (6)
scripts/.env.example(1 hunks)scripts/.gitignore(1 hunks)scripts/benchmarks/README.md(1 hunks)scripts/benchmarks/eoa.ts(1 hunks)scripts/package.json(1 hunks)scripts/tsconfig.json(1 hunks)
✅ Files skipped from review due to trivial changes (1)
- scripts/.gitignore
🚧 Files skipped from review as they are similar to previous changes (3)
- scripts/tsconfig.json
- scripts/benchmarks/eoa.ts
- scripts/package.json
🧰 Additional context used
🪛 dotenv-linter (3.3.0)
scripts/.env.example
[warning] 4-4: [UnorderedKey] The FROM key should go before the TYPE key
(UnorderedKey)
[warning] 5-5: [UnorderedKey] The CHAIN_ID key should go before the FROM key
(UnorderedKey)
[warning] 6-6: [UnorderedKey] The SECRET_KEY key should go before the TYPE key
(UnorderedKey)
🪛 markdownlint-cli2 (0.18.1)
scripts/benchmarks/README.md
23-23: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
108-108: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
| BASE_URL=http://localhost:3005 | ||
| TYPE=eoa | ||
| FROM=0x... | ||
| CHAIN_ID=1337 | ||
| SECRET_KEY=your-secret-key | ||
| VAULT_ACCESS_TOKEN=your-vault-access-token |
There was a problem hiding this comment.
Sort required keys to satisfy dotenv-linter.
.env.example currently trips the UnorderedKey rule (BASE_URL ➝ TYPE ➝ FROM …). This will keep failing lint runs until the keys are alphabetized. Please reorder the required block (e.g. BASE_URL, CHAIN_ID, FROM, SECRET_KEY, TYPE, VAULT_ACCESS_TOKEN) so the example passes the check.
(evrone.com)
🧰 Tools
🪛 dotenv-linter (3.3.0)
[warning] 4-4: [UnorderedKey] The FROM key should go before the TYPE key
(UnorderedKey)
[warning] 5-5: [UnorderedKey] The CHAIN_ID key should go before the FROM key
(UnorderedKey)
[warning] 6-6: [UnorderedKey] The SECRET_KEY key should go before the TYPE key
(UnorderedKey)
🤖 Prompt for AI Agents
In scripts/.env.example around lines 2 to 7 the environment keys are out of
alphabetical order causing dotenv-linter's UnorderedKey rule to fail; reorder
the block to alphabetical order (BASE_URL, CHAIN_ID, FROM, SECRET_KEY, TYPE,
VAULT_ACCESS_TOKEN) keeping the existing example values intact so the file
passes linting.
037aa60 to
d12d799
Compare
There was a problem hiding this comment.
Actionable comments posted: 1
♻️ Duplicate comments (2)
scripts/.env.example (1)
2-7: Past review comment not yet addressed: Sort keys alphabetically.The key ordering issue flagged in the previous review remains unresolved. Keys should be alphabetized to satisfy dotenv-linter's
UnorderedKeyrule.scripts/benchmarks/eoa.ts (1)
145-172: Critical: Event casing still incorrect, blocking lifecycle tracking.The webhook event type comparisons remain mismatched. Lines 146, 158, and 167 still check
"SUCCESS","FAIL", and"NACK", but webhook payloads use"Success","Failure", and"Nack"(as documented in lines 17 and README). This prevents transaction state progression, causing all transactions to timeout as pending.Apply this diff to fix the remaining casing issues:
if (event.stageName === "send") { - if (event.eventType === "SUCCESS") { + if (event.eventType === "Success") { metrics.submittedTime = now; metrics.sentToSubmittedMs = now - metrics.sentTime; metrics.status = "submitted"; console.log(`✅ Transaction ${txId.slice(0, 8)}... submitted (${metrics.sentToSubmittedMs}ms)`); } else if (event.eventType === "Failure") { } else if (event.stageName === "confirm") { - if (event.eventType === "SUCCESS") { + if (event.eventType === "Success") { metrics.confirmedTime = now; if (metrics.submittedTime) { metrics.submittedToConfirmedMs = now - metrics.submittedTime; } metrics.totalTimeMs = now - metrics.sentTime; metrics.status = "confirmed"; pendingTransactions.delete(txId); console.log(`🎉 Transaction ${txId.slice(0, 8)}... confirmed (total: ${metrics.totalTimeMs}ms, pending: ${pendingTransactions.size})`); - } else if (event.eventType === "FAIL" || event.eventType === "NACK") { + } else if (event.eventType === "Failure" || event.eventType === "Nack") {
🧹 Nitpick comments (1)
scripts/benchmarks/README.md (1)
23-23: Optional: Add language specifiers to code blocks.For better syntax highlighting, consider adding language specifiers to the fenced code blocks at lines 23 and 108.
Apply these changes:
- ``` + ```env BASE_URL=http://localhost:3005 ... ```And:
-``` +```text 🚀 Starting benchmark... ...Also applies to: 108-108 </blockquote></details> </blockquote></details> <details> <summary>📜 Review details</summary> **Configuration used**: CodeRabbit UI **Review profile**: CHILL **Plan**: Pro **Disabled knowledge base sources:** - Linear integration is disabled by default for public repositories > You can enable these sources in your CodeRabbit configuration. <details> <summary>📥 Commits</summary> Reviewing files that changed from the base of the PR and between 037aa60c52d690512619835ce6ee41efa98c8fd9 and d12d799d993af09401bb12728fc19ba4b5a37630. </details> <details> <summary>📒 Files selected for processing (6)</summary> * `scripts/.env.example` (1 hunks) * `scripts/.gitignore` (1 hunks) * `scripts/benchmarks/README.md` (1 hunks) * `scripts/benchmarks/eoa.ts` (1 hunks) * `scripts/package.json` (1 hunks) * `scripts/tsconfig.json` (1 hunks) </details> <details> <summary>✅ Files skipped from review due to trivial changes (1)</summary> * scripts/.gitignore </details> <details> <summary>🚧 Files skipped from review as they are similar to previous changes (1)</summary> * scripts/package.json </details> <details> <summary>🧰 Additional context used</summary> <details> <summary>📓 Path-based instructions (2)</summary> <details> <summary>scripts/**/*.{ts,tsx,js,jsx}</summary> **📄 CodeRabbit inference engine (scripts/.cursor/rules/use-bun-instead-of-node-vite-npm-pnpm.mdc)** > `scripts/**/*.{ts,tsx,js,jsx}`: Use `bun <file>` instead of `node <file>` or `ts-node <file>` to run JS/TS files > Do not use `dotenv`; Bun automatically loads .env files > Use `Bun.serve()` for HTTP/WebSocket servers; do not use `express` > Use `bun:sqlite` for SQLite; do not use `better-sqlite3` > Use `Bun.redis` for Redis; do not use `ioredis` > Use `Bun.sql` for Postgres; do not use `pg` or `postgres.js` > Use built-in `WebSocket`; do not use `ws` > Prefer `Bun.file` over `node:fs` readFile/writeFile for file IO > Use `Bun.$` (shell) instead of `execa` Files: - `scripts/benchmarks/eoa.ts` </details> <details> <summary>scripts/**/*.{html,ts,tsx,css}</summary> **📄 CodeRabbit inference engine (scripts/.cursor/rules/use-bun-instead-of-node-vite-npm-pnpm.mdc)** > Use `bun build <file.html|file.ts|file.css>` instead of `webpack` or `esbuild` Files: - `scripts/benchmarks/eoa.ts` </details> </details><details> <summary>🪛 dotenv-linter (3.3.0)</summary> <details> <summary>scripts/.env.example</summary> [warning] 4-4: [UnorderedKey] The FROM key should go before the TYPE key (UnorderedKey) --- [warning] 5-5: [UnorderedKey] The CHAIN_ID key should go before the FROM key (UnorderedKey) --- [warning] 6-6: [UnorderedKey] The SECRET_KEY key should go before the TYPE key (UnorderedKey) </details> </details> <details> <summary>🪛 markdownlint-cli2 (0.18.1)</summary> <details> <summary>scripts/benchmarks/README.md</summary> 23-23: Fenced code blocks should have a language specified (MD040, fenced-code-language) --- 108-108: Fenced code blocks should have a language specified (MD040, fenced-code-language) </details> </details> </details> <details> <summary>🔇 Additional comments (10)</summary><blockquote> <details> <summary>scripts/tsconfig.json (1)</summary><blockquote> `4-4`: **LGTM: DOM types enable web APIs for benchmark.** Adding `"DOM"` to the lib array appropriately enables typings for `performance`, `fetch`, `URL`, and other web APIs used in the new benchmark script. </blockquote></details> <details> <summary>scripts/benchmarks/eoa.ts (8)</summary><blockquote> `1-88`: **LGTM: Type definitions are well-structured.** The type declarations appropriately define the webhook event structure, transaction metrics, configuration, and aggregate results. The `WebhookEvent.eventType` comment correctly documents the expected values as `"Success" | "Nack" | "Failure"`. --- `90-110`: **LGTM: Configuration follows Bun conventions.** The configuration correctly relies on Bun's automatic `.env` loading without importing `dotenv`, and appropriately validates required environment variables before proceeding. --- `112-131`: **LGTM: Webhook server uses Bun.serve correctly.** The webhook server appropriately uses `Bun.serve()` instead of Express, following the coding guidelines for Bun-based scripts. --- `176-262`: **LGTM: Transaction sending logic is sound.** The `sendTransaction` function appropriately uses native `fetch`, records both HTTP response time and lifecycle timestamps, and handles errors gracefully with proper fallback metrics. --- `264-330`: **LGTM: Concurrency control correctly implemented.** The concurrency limiting now correctly uses a separate `inFlight` Set that removes settled promises (line 287), ensuring `Promise.race` blocks when the limit is reached. This addresses the previous review concern about unbounded concurrency. --- `332-358`: **LGTM: Statistics calculation is robust.** The percentile calculations correctly sort values and handle empty arrays safely, with appropriate fallback to zero for all metrics. --- `360-478`: **LGTM: Output generation follows Bun conventions.** The output functions appropriately use `Bun.write()` for file I/O and generate well-structured CSV and JSON results with comprehensive statistics. --- `480-516`: **LGTM: Main execution flow uses Bun shell correctly.** The main function appropriately uses `Bun.$` for shell commands and handles cleanup properly on both success and error paths. </blockquote></details> <details> <summary>scripts/benchmarks/README.md (1)</summary><blockquote> `1-154`: **LGTM: Documentation is comprehensive and accurate.** The README thoroughly documents the benchmark tool's features, configuration, workflow, and output formats. The webhook event structure correctly documents the expected event types that align with the implementation. </blockquote></details> </blockquote></details> </details> <!-- This is an auto-generated comment by CodeRabbit for review status -->
|
|
||
| // Wait for all webhooks to be received (with timeout) | ||
| console.log(`⏳ Waiting for webhooks to complete...`); | ||
| const maxWaitTime = 600000; // 5 minutes |
There was a problem hiding this comment.
Fix comment to match actual timeout value.
The comment states "2 minutes" but the value is 600000ms (10 minutes). Update the comment to "10 minutes" for accuracy.
- const maxWaitTime = 600000; // 5 minutes
+ const maxWaitTime = 600000; // 10 minutes📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| const maxWaitTime = 600000; // 5 minutes | |
| const maxWaitTime = 600000; // 10 minutes |
🤖 Prompt for AI Agents
In scripts/benchmarks/eoa.ts around line 307, the inline comment for maxWaitTime
incorrectly says "2 minutes" while the value is 600000 ms (10 minutes); update
the comment to "10 minutes" so it accurately reflects the timeout value.
1 participant
Summary by CodeRabbit
New Features
Documentation
Chores