Skip to content

chore: split dax into separate workflow, rename TTI workflow#193

Merged
HeyGarrison merged 2 commits into
masterfrom
chore/split-dax-workflow
Jul 17, 2026
Merged

chore: split dax into separate workflow, rename TTI workflow#193
HeyGarrison merged 2 commits into
masterfrom
chore/split-dax-workflow

Conversation

@HeyGarrison

@HeyGarrison HeyGarrison commented Jul 17, 2026

Copy link
Copy Markdown
Collaborator

Summary

Splits the dax benchmark into its own workflow file and renames the existing TTI workflow for clarity.

Changes

New: sandbox-dax-benchmarks.yml

  • Dax job with provider matrix (weekly Friday schedule + PR trigger + manual dispatch)
  • Dedicated collect job that merges dax results and posts a PR comment with per-phase table
  • Defaults to 1 iteration
  • Own concurrency group (dax-benchmarks)

Renamed: sandbox-benchmarks.yml → sandbox-tti-benchmarks.yml

  • Removed dax job, sandbox-dax mode option, and dax PR comment section
  • Restored iterations default to 100 (TTI-appropriate)
  • Collect job depends only on bench
  • Workflow name changed to "Sandbox TTI Benchmark"

This matches the existing pattern where each benchmark category has its own workflow file (browser, storage, snapshot-fork).

@open-cla

open-cla Bot commented Jul 17, 2026

Copy link
Copy Markdown

Contributor License Agreement

All contributors are covered by a CLA.

superagent-security[bot]
superagent-security Bot reviewed Jul 17, 2026
View reviewed changes

@superagent-security superagent-security Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Superagent found 2 security concern(s).

Comment thread .github/workflows/sandbox-dax-benchmarks.yml
Comment thread .github/workflows/sandbox-dax-benchmarks.yml
@superagent-security superagent-security Bot added pr:flagged PR flagged for review by security analysis. and removed pr:flagged PR flagged for review by security analysis. labels Jul 17, 2026
@HeyGarrison
HeyGarrison force-pushed the chore/split-dax-workflow branch from a1be76d to 1bf4a37 Compare July 17, 2026 17:46
@github-actions

github-actions Bot commented Jul 17, 2026

Copy link
Copy Markdown
Contributor

Storage Benchmark Results

1MB Files

# Provider Score Download Throughput Upload Status
1 Azure Blob Storage 96.2 0.03s 281.9 Mbps 0.08s 10/10
2 Tigris 95.0 0.07s 113.6 Mbps 0.21s 10/10
3 Vercel Blob 94.7 0.10s 81.7 Mbps 0.28s 10/10
4 Cloudflare R2 94.6 0.11s 77.5 Mbps 0.23s 10/10
5 AWS S3 94.6 0.09s 91.6 Mbps 0.21s 10/10

View full run · SVGs available as build artifacts

@github-actions

github-actions Bot commented Jul 17, 2026

Copy link
Copy Markdown
Contributor

Snapshot/Fork Benchmark Results

small dataset

# Provider Score Snapshot create Fork (snapshot) Fork (live) First read Status
1 Tigris 99.7 0.08s 0.33s 0.17s 0.08s 1/1
2 Azure-blob 98.8 0.76s 0.82s 0.78s 0.02s 1/1
3 Cloudflare R2 95.8 1.68s 3.88s 3.21s 0.21s 1/1
4 AWS S3 0.0 0.00s 0.00s 0.00s 0.00s 0/1

View full run

@github-actions

github-actions Bot commented Jul 17, 2026

Copy link
Copy Markdown
Contributor

Browser Throughput Benchmark Results

# Provider Score APS (med) Task (med) Task (p95) Screenshot Status
1 Kernel 48.7 5.45/s 1.83s 10.71s 294ms 2/3
2 Browseruse 43.9 3.85/s 2.60s 10.94s 260ms 2/3
3 Browserbase 38.7 2.35/s 4.25s 11.44s 398ms 2/3
4 Steel 37.4 2.07/s 4.82s 11.82s 631ms 2/3
5 Notte 34.2 1.51/s 6.60s 13.19s 954ms 2/3
6 Hyperbrowser 30.9 0.99/s 10.14s 12.94s 1094ms 2/3

View full run · SVG available as build artifact

@github-actions

github-actions Bot commented Jul 17, 2026

Copy link
Copy Markdown
Contributor

Browser Benchmark Results

# Provider Score Create Connect Navigate Release Total Status
1 Kernel 97.6 0.01s 0.08s 0.10s 0.05s 0.25s 10/10
2 Browserbase 93.9 0.22s 0.12s 0.13s 0.13s 0.66s 10/10
3 Notte 91.8 0.16s 0.50s 0.20s 0.09s 1.01s 10/10
4 Browseruse 90.9 0.56s 0.10s 0.06s 0.05s 0.79s 10/10
5 Hyperbrowser 89.3 0.32s 0.54s 0.26s 0.10s 1.27s 10/10
6 Steel 63.0 1.65s 0.77s 0.12s 0.19s 3.18s 10/10

View full run · SVG available as build artifact

@github-actions

github-actions Bot commented Jul 17, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Sandbox Dax Benchmark Results

Provider Phases Total Prepare Clone Install Typecheck Status
archil 0/7 0.00s 0.00s 0.00s 0.00s 0.00s 0/1 OK
beam 0/7 0.00s 0.00s 0.00s 0.00s 0.00s 0/1 OK
blaxel 0/7 0.00s 0.00s 0.00s 0.00s 0.00s 0/1 OK
cloud-run 0/7 0.00s 0.00s 0.00s 0.00s 0.00s 0/1 OK
cloudflare 0/7 0.00s 0.00s 0.00s 0.00s 0.00s 0/1 OK
codesandbox 0/7 0.00s 0.00s 0.00s 0.00s 0.00s 0/1 OK
createos 0/7 0.00s 0.00s 0.00s 0.00s 0.00s 0/1 OK
daytona 0/7 0.00s 0.00s 0.00s 0.00s 0.00s 0/1 OK
declaw -- 0.00s 0.00s 0.00s 0.00s 0.00s 0/1 OK
e2b 6/7 18.18s 11.96s 3.17s 1.07s 0.07s 0/1 OK
hopx -- 0.00s 0.00s 0.00s 0.00s 0.00s 0/1 OK
isorun 0/7 0.00s 0.00s 0.00s 0.00s 0.00s 0/1 OK
lightning -- 0.00s 0.00s 0.00s 0.00s 0.00s 0/1 OK
modal 7/7 59.56s 7.09s 2.62s 20.64s 20.55s 1/1 OK
northflank 0/7 0.00s 0.00s 0.00s 0.00s 0.00s 0/1 OK
runloop 6/7 84.66s 23.21s 3.75s 30.49s 11.89s 0/1 OK
superserve -- 0.00s 0.00s 0.00s 0.00s 0.00s 0/1 OK
tensorlake 6/7 19.36s 13.43s 1.97s 2.63s 0.02s 0/1 OK
upstash 0/7 0.00s 0.00s 0.00s 0.00s 0.00s 0/1 OK
vercel 0/7 0.00s 0.00s 0.00s 0.00s 0.00s 0/1 OK

View full run

@github-actions

github-actions Bot commented Jul 17, 2026

Copy link
Copy Markdown
Contributor

Sandbox Benchmark Results

Sequential

# Provider Score Median TTI P95 P99 Status
1 isorun 99.7 0.01s 0.05s 0.05s 10/10
2 declaw 98.8 0.05s 0.21s 0.21s 10/10
3 archil 98.4 0.14s 0.19s 0.19s 10/10
4 northflank 98.0 0.13s 0.30s 0.30s 10/10
5 daytona 97.8 0.17s 0.29s 0.29s 10/10
6 blaxel 97.6 0.21s 0.29s 0.29s 10/10
7 lightning 97.5 0.19s 0.33s 0.33s 10/10
8 createos 97.1 0.24s 0.37s 0.37s 10/10
9 upstash 96.8 0.29s 0.37s 0.37s 10/10
10 vercel 96.2 0.35s 0.44s 0.44s 10/10
11 e2b 95.0 0.47s 0.56s 0.56s 10/10
12 modal 94.4 0.49s 0.67s 0.67s 10/10
13 superserve 94.3 0.46s 0.73s 0.73s 10/10
14 runloop 94.1 0.49s 0.74s 0.74s 10/10
15 tensorlake 91.7 0.80s 0.88s 0.88s 10/10
16 cloud-run 91.1 0.61s 1.31s 1.31s 10/10
17 beam 89.7 0.39s 2.00s 2.00s 10/10
18 cloudflare 76.0 1.85s 3.22s 3.22s 10/10
19 codesandbox 72.6 2.36s 3.30s 3.30s 10/10
20 hopx 22.4 5.33s 5.82s 5.82s 5/10

Staggered

# Provider Score Median TTI P95 P99 Status
1 isorun 99.8 0.01s 0.02s 0.02s 10/10
2 declaw 99.0 0.10s 0.11s 0.11s 10/10
3 archil 98.6 0.12s 0.17s 0.17s 10/10
4 northflank 98.3 0.15s 0.19s 0.19s 10/10
5 daytona 98.2 0.11s 0.29s 0.29s 10/10
6 blaxel 97.9 0.20s 0.24s 0.24s 10/10
7 lightning 97.8 0.21s 0.24s 0.24s 10/10
8 createos 97.3 0.25s 0.28s 0.28s 10/10
9 upstash 95.9 0.30s 0.58s 0.58s 10/10
10 beam 95.7 0.38s 0.52s 0.52s 10/10
11 superserve 95.6 0.39s 0.53s 0.53s 10/10
12 e2b 95.2 0.37s 0.65s 0.65s 10/10
13 vercel 95.0 0.32s 0.76s 0.76s 10/10
14 runloop 94.3 0.55s 0.61s 0.61s 10/10
15 modal 93.9 0.47s 0.81s 0.81s 10/10
16 cloud-run 92.4 0.62s 0.97s 0.97s 10/10
17 tensorlake 91.2 0.84s 0.93s 0.93s 10/10
18 cloudflare 76.6 1.89s 3.02s 3.02s 10/10
19 codesandbox 73.6 2.45s 2.93s 2.93s 10/10
20 hopx 0.0 11.91s 13.04s 13.04s 8/10

Burst

# Provider Score Median TTI P95 P99 Status
1 isorun 99.8 0.02s 0.02s 0.02s 10/10
2 declaw 98.2 0.17s 0.19s 0.19s 10/10
3 northflank 98.2 0.16s 0.20s 0.20s 10/10
4 archil 98.1 0.16s 0.23s 0.23s 10/10
5 daytona 98.0 0.14s 0.29s 0.29s 10/10
6 blaxel 97.7 0.22s 0.25s 0.25s 10/10
7 createos 97.0 0.29s 0.32s 0.32s 10/10
8 lightning 96.7 0.26s 0.43s 0.43s 10/10
9 superserve 95.8 0.38s 0.49s 0.49s 10/10
10 vercel 94.6 0.37s 0.79s 0.79s 10/10
11 upstash 94.3 0.56s 0.60s 0.60s 10/10
12 e2b 94.0 0.44s 0.84s 0.84s 10/10
13 modal 92.8 0.59s 0.90s 0.90s 10/10
14 cloud-run 92.4 0.53s 1.10s 1.10s 10/10
15 tensorlake 90.9 0.89s 0.94s 0.94s 10/10
16 beam 89.4 0.86s 1.35s 1.35s 10/10
17 runloop 77.8 0.60s 4.66s 4.66s 10/10
18 codesandbox 68.6 2.98s 3.38s 3.38s 10/10
19 cloudflare 59.7 3.59s 4.68s 4.68s 10/10
20 hopx 0.0 13.19s 15.81s 15.81s 7/10

View full run · SVGs available as build artifacts

@HeyGarrison
chore: split dax into separate workflow, rename TTI workflow
7442dbc 
- Create sandbox-dax-benchmarks.yml with dax job, own collect job,
  and dedicated PR comment with per-phase table
- Rename sandbox-benchmarks.yml to sandbox-tti-benchmarks.yml
- Remove dax job, mode option, and workload PR comment from TTI workflow
- Restore TTI iterations default to 100
@HeyGarrison
fix(browser): handle absolute https Wikipedia URLs in throughput sele…
a60ec8b 
…ctor

The CSS selector :not([href*=':']) was rejecting article body links
rendered as absolute https:// URLs (e.g. https://en.wikipedia.org/wiki/Foo)
because 'https:' contains a colon. Wikipedia now renders some body links
in this format, causing ~54% of iterations to fail across all providers.

Replace the CSS :not() filter with JS-based filtering that only checks
for namespace colons (Help:, File:, Category:) after /wiki/, correctly
handling relative, protocol-relative, and absolute URL formats.
@HeyGarrison
HeyGarrison force-pushed the chore/split-dax-workflow branch from 1bf4a37 to a60ec8b Compare July 17, 2026 18:44
@HeyGarrison
HeyGarrison merged commit 86a8666 into master Jul 17, 2026
24 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

1 more reviewer
@superagent-security superagent-security[bot] superagent-security[bot] left review comments
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@HeyGarrison