fix: standardize dax benchmark resources to 8 vCPU / 16 GiB#196
Open
HeyGarrison wants to merge 10 commits into
Open
fix: standardize dax benchmark resources to 8 vCPU / 16 GiB#196HeyGarrison wants to merge 10 commits into
HeyGarrison wants to merge 10 commits into
Conversation
Each provider gets 8 vCPUs and 16 GiB RAM for fair comparison: - E2B: cpuCount=8, memoryMB=16384 - Modal: cpu=4 (4 physical = 8 vCPUs), memoryMiB=16384 - Tensorlake: cpus=8, memoryMb=16384 - isorun: vcpus=8, memMiB=16384 - runloop: customCpuCores=8, customMemoryGb=16 - daytona: resources.cpu=8, resources.memory=16 (GiB) - upstash: size=large (8 cores, 16 GB) - vercel: resources.vcpus=8 (no memory control) - blaxel: memory=16384 (CPU derived: 16384/2048=8 cores) Providers without CPU/memory control (archil, cloudflare, northflank, beam, declaw, hopx, codesandbox) use their defaults.
Contributor License AgreementAll contributors are covered by a CLA. |
Contributor
Sandbox Benchmark ResultsSequential
Staggered
Burst
View full run · SVGs available as build artifacts |
Contributor
Sandbox Dax Benchmark Results
|
… time E2B's Sandbox.create() accepts SandboxOpts which does not include cpuCount/memoryMB. Those are template build options only.
|
For declaw, the cpu, memory and disk are property of the template, you can create a custom template with the required size or we can have a node-large template with above spec |
- Fix runloop: nest resource params in launch_parameters with snake_case - Add beam: cpu=8, memory=16384 MiB - Add codesandbox: vmTier=VMTier.Small (8 CPU / 16 GiB) - Add northflank: deploymentPlan resolved via API at runtime - Add E2B template build prereq (base-8cpu-16gb with 8 vCPU / 16 GiB) - Add Northflank plan discovery prereq script
- Add scripts/provider-benchmark.sh: vendored from upstream with dnf support (RHEL/Fedora) alongside apt-get (Debian/Ubuntu) and aarch64 architecture detection for Node.js and Bun downloads - Update dax.ts to load script from local filesystem instead of curl, eliminating curl dependency for providers that don't ship it
Adds declaw to DAX_RESOURCE_OPTIONS so it targets the same 8 vCPU / 16 GiB profile as the other providers in the map. The node-large template provisions 8 vCPU / 16 GiB RAM with an 8 GiB disk overlay. Co-authored-by: shivam-declaw <272314190+shivam-declaw@users.noreply.github.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
![superagent-security[bot]](https://avatars.githubusercontent.com/in/3287076?s=60&v=4){kind=small}
superagent-security
Bot
added
the
pr:flagged
…ce node image - Fix BUN_INTERNAL_PATH: remove .zip extension from zip-internal path (was bun-linux-x64-baseline.zip/bun, should be bun-linux-x64-baseline/bun) - Northflank: robust API response parsing, graceful fallback to nf-compute-50, sanitize plan ID before writing to GITHUB_ENV - Daytona: remove from DAX_RESOURCE_OPTIONS (resources not supported with snapshot-based sandbox creation) - Namespace: add sandboxOptions image node:22 (dax benchmark requires node)
superagent-security
Bot
removed
the
pr:flagged
- Northflank: pick smallest suitable plan instead of closest to reduce chance of exceeding project resource allowance - E2B: workaround for 60s command timeout by setting defaultProcessConnectionTimeout=0 via getInstance() while waiting for upstream wrapper fix to forward timeoutMs
Add image: 'node:22' to daytona sandboxOptions to switch from snapshot-based to image-based creation, which allows specifying resources. Re-add daytona to DAX_RESOURCE_OPTIONS with 8 vCPU / 16 GiB.
| if [[ "$(node --version 2>/dev/null || true)" != "v${NODE_VERSION}" ]]; then | ||
| local archive="node-v${NODE_VERSION}-${NODE_ARCH}.tar.xz" | ||
| local prefix="/opt/node-v${NODE_VERSION}-${NODE_ARCH}" | ||
| curl -fsSL "https://nodejs.org/download/release/v${NODE_VERSION}/${archive}" -o "/tmp/${archive}" |
There was a problem hiding this comment.
P2: Benchmark script downloads and executes binaries without checksum verification
Downloads Node.js and Bun archives without verifying checksums or signatures before execution.
Verify SHA256 checksums of downloaded archives against official manifests before extraction.
AI prompt
Check if this security scanner issue is valid. If so, understand the root cause and fix it. If appropriate, update or add tests. Keep the change focused and preserve intended behavior.
<file name="scripts/dax-benchmark.sh">
<violation number="1" location="scripts/dax-benchmark.sh:133">
<priority>P2</priority>
<title>Benchmark script downloads and executes binaries without checksum verification</title>
<evidence>The script uses curl to download Node.js and Bun release archives and then directly extracts and executes them without verifying SHA256 checksums, signatures, or release manifests.</evidence>
<recommendation>Add checksum verification for downloaded Node.js and Bun archives before extraction. For Node.js, verify against the official SHASUMS256.txt file; for Bun, use the published checksum files or hardcode known-good checksums for each supported version.</recommendation>
</violation>
</file>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Standardizes sandbox sizing across providers for fair comparison. Previously Modal was bursting to 17 vCPUs while Tensorlake had 1 vCPU, making results incomparable.
Sized providers (8 vCPU / 16 GiB)
Providers without CPU/memory control (use defaults)
archil, cloudflare, northflank, beam, declaw, hopx, codesandbox