Feat execute bash#119
Open
simongdavies wants to merge 15 commits intohyperlight-dev:mainfrom
Open
Conversation
Adds a new execute_bash tool that runs bash commands inside the
Hyperlight micro-VM using just-bash (pure TypeScript bash interpreter).
Commands run in hardware-isolated QuickJS with no host process access.
40+ Unix commands available: ls, cat, grep, jq, sed, awk, sort, find,
tree, diff, head, tail, wc, cut, tr, xargs, tee, cp, mkdir, touch,
base64, md5sum, date, env vars, pipes, redirects, and more.
Architecture:
- just-bash bundled via esbuild with node: API stubs (1.4 MB)
- Polyfills for URL, Buffer, process, AbortController, crypto, setTimeout
- Auto-registered internal handler (sys-bash-runner)
- Stateless: fresh Bash instance per call
- Heap auto-expanded to 64 MB on first use
- Rebuilds automatically as part of npm run build:modules
Key design decisions:
- Tool, not module: LLM calls execute_bash({command}) directly
- No in-memory FS: all file ops go through host:fs-read/host:fs-write
- curl requires fetch plugin enabled (actionable error if not)
- Unsupported commands (rm, mv, python3, sqlite3) return clear errors
- Command list in tool description for strong LLM guidance
Signed-off-by: Simon Davies <simongdavies@users.noreply.github.com>
The LLM didn't know about execute_bash because the system prompt only mentioned register_handler + execute_javascript. Added bash tool guidance between WORKFLOW and DIRECT FILE I/O sections. Signed-off-by: Simon Davies <simongdavies@users.noreply.github.com>
Shows '$ command' inline when show-code is enabled and writes bash commands to the code log file alongside JS handler code. Signed-off-by: Simon Davies <simongdavies@users.noreply.github.com>
Without this, the tool exists but the SDK blocks the LLM from calling it. Absolute rookie mistake. Signed-off-by: Simon Davies <simongdavies@users.noreply.github.com>
When a skill is active, the SDK only exposes tools listed in the skill's allowed-tools. execute_bash was missing from all 9 skills, so the LLM couldn't see or use it when any skill was active. Also adds a pattern-integrity test that verifies every defineTool in index.ts has a corresponding entry in ALLOWED_TOOLS — catches this class of bug automatically. Signed-off-by: Simon Davies <simongdavies@users.noreply.github.com>
Signed-off-by: Simon Davies <simongdavies@users.noreply.github.com>
The SDK has THREE places that control tool visibility: 1. tools[] — implementation registration 2. ALLOWED_TOOLS — pre-execution gate (tool-gating.ts) 3. availableTools[] — tells SDK what to show the model execute_bash was in 1 and 2 but not 3, so the model literally couldn't see it. Added test that verifies every defineTool has entries in BOTH ALLOWED_TOOLS AND availableTools. Signed-off-by: Simon Davies <simongdavies@users.noreply.github.com>
Adds hardware random number generation to the Hyperlight QuickJS runtime using x86_64 RDRAND instruction: - crypto.getRandomValues(Uint8Array) — fills array with HW random bytes - crypto.randomUUID() — generates RFC 4122 v4 UUIDs with proper version/variant bits, backed by RDRAND (not a fake hex generator) - Math.random() — overrides QuickJS built-in with RDRAND-backed implementation, enabling awk rand() in just-bash Also fixes execute_bash: - Auto-configures heap (64MB) and buffers (2048KB) at startup when ha:bash module is present (not lazily on first call) - Stubs sql.js to reduce bundle size - Removes JS crypto polyfill (now native) - Fixes curl listed as 'NOT AVAILABLE' in tool description (curl is supported when fetch plugin is enabled) Signed-off-by: Simon Davies <simongdavies@users.noreply.github.com>
…JS APIs
- Bash runs in its own dedicated sandbox (getBashSandbox) with only
ha:bash loaded — no pptx/pdf/xlsx memory competition
- Excluded ha:bash from main JS sandbox module loading
- Filtered bash from list_modules output (LLM won't see it)
- module_info('bash') returns clear error: 'use execute_bash tool'
- Fixed contradictory system message ('No bash' vs bash tool section)
- JS handlers cannot import ha:bash — it only exists in the bash sandbox
Signed-off-by: Simon Davies <simongdavies@users.noreply.github.com>
… execute_bash - onPostToolUse now handles execute_bash memory errors (was JS only) - Clearly tells LLM to increase SCRATCH for 'Out of physical memory' and HEAP for 'malloc failed' — with specific configure_sandbox call - configure_sandbox description now lists common memory errors and which setting to change for each Signed-off-by: Simon Davies <simongdavies@users.noreply.github.com>
- Add sha1/sha2 RustCrypto crates (no_std, audited) to native-globals
- Implement crypto.subtle.digest('SHA-1'|'SHA-256', data) → Promise<ArrayBuffer>
matching the Web Crypto API — fixes sha256sum/sha1sum in just-bash
- Fix Buffer polyfill: HaBuffer subclass with .toString(encoding) supporting
base64, latin1/binary, hex, utf-8 — fixes silent data corruption in
curl (Basic auth), yq, and unguarded Buffer.from().toString() paths
- Restore md5sum/sha256sum to BASH_SUPPORTED_COMMANDS (were removed
when crypto.subtle was missing — now backed by RustCrypto)
- Add POSIX-only flag caveat to execute_bash description
- Update large-output guidance to mention execute_bash as alternative
to handler-based file processing
Signed-off-by: Simon Davies <simongdavies@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
Pull request overview
Adds an execute_bash tool to HyperAgent (implemented via a bundled just-bash runtime module) and extends the sandbox runtime with missing WebCrypto-style primitives needed by that bash environment (notably crypto.subtle.digest). Also updates tool-gating/skill metadata and build scripts to generate the new builtin module bundle.
Changes:
- Add
execute_bashtool + gating/visibility wiring (ALLOWED_TOOLS, availableTools, skill allowed-tools). - Extend native-globals to provide
crypto.getRandomValues,crypto.randomUUID,crypto.subtle.digest, and overrideMath.random. - Add a bash bundling pipeline (esbuild + stubs) and a test to ensure tool gating completeness.
Reviewed changes
Copilot reviewed 34 out of 37 changed files in this pull request and generated 12 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/pattern-integrity.test.ts | Adds a test to ensure every defineTool() is present in tool gating + availableTools. |
| src/sandbox/runtime/modules/native-globals/src/lib.rs | Adds crypto globals (RNG/UUID/digest) and overrides Math.random. |
| src/sandbox/runtime/modules/native-globals/Cargo.toml | Adds RustCrypto deps (sha1, sha2, digest). |
| src/sandbox/runtime/Cargo.lock | Locks new Rust dependencies. |
| src/agent/tool-gating.ts | Adds execute_bash to ALLOWED_TOOLS. |
| src/agent/system-message.ts | Updates system guidance to mention execute_bash. |
| src/agent/index.ts | Implements execute_bash, adjusts module listing/access rules, and adds memory-error guidance for execute_bash. |
| skills/xlsx-expert/SKILL.md | Allows execute_bash for the skill. |
| skills/web-scraper/SKILL.md | Allows execute_bash for the skill. |
| skills/research-synthesiser/SKILL.md | Allows execute_bash for the skill. |
| skills/report-builder/SKILL.md | Allows execute_bash for the skill. |
| skills/pptx-expert/SKILL.md | Allows execute_bash for the skill. |
| skills/pdf-expert/SKILL.md | Allows execute_bash for the skill. |
| skills/mcp-services/SKILL.md | Allows execute_bash for the skill. |
| skills/data-processor/SKILL.md | Allows execute_bash for the skill. |
| skills/api-explorer/SKILL.md | Allows execute_bash for the skill. |
| scripts/build-modules.js | Adds a step to build the ha:bash bundle during module builds. |
| scripts/bash-bundle/zstd-stub.mjs | Adds a stub for optional zstd dependency during bundling. |
| scripts/bash-bundle/zlib-stub.mjs | Adds a stub for node:zlib APIs (gzip/deflate) in the sandbox bundle. |
| scripts/bash-bundle/worker-stub.mjs | Adds stubs for worker_threads/async_hooks. |
| scripts/bash-bundle/url-stub.mjs | Adds a minimal stub for node:url. |
| scripts/bash-bundle/turndown-stub.mjs | Adds a stub for turndown (HTML-to-markdown). |
| scripts/bash-bundle/sqljs-stub.mjs | Adds a stub for sql.js (sqlite unsupported). |
| scripts/bash-bundle/node-path-stub.mjs | Adds a minimal node:path polyfill for bundling. |
| scripts/bash-bundle/liblzma-stub.mjs | Adds a stub for optional liblzma dependency. |
| scripts/bash-bundle/fs-stub.mjs | Adds a stub for node:fs usage in the bundle. |
| scripts/bash-bundle/entry.mjs | esbuild entrypoint re-exporting just-bash symbols. |
| scripts/bash-bundle/dns-stub.mjs | Adds a stub for node:dns. |
| scripts/bash-bundle/crypto-stub.mjs | Adds a stub for node:crypto (randomness + hash unsupported). |
| scripts/bash-bundle/bzip-stub.mjs | Adds a stub for seek-bzip. |
| scripts/bash-bundle/build.mjs | Adds the esbuild bundling script that emits builtin-modules/bash.js. |
| scripts/bash-bundle/build-ha-bash.mjs | Adds an additional build script (currently not integrated). |
| package.json | Adds esbuild and just-bash dependencies. |
| package-lock.json | Locks new npm dependencies. |
| Justfile | Adds a build-bash task. |
| builtin-modules/bash.json | Adds metadata for the new ha:bash builtin module. |
| .gitignore | Ignores the generated bash bundle and temp bundle output. |
Comment on lines
+9
to
+12
| //! crypto.getRandomValues and crypto.randomUUID use a xorshift128+ | ||
| //! PRNG seeded from a hash of the compilation timestamp. This is NOT | ||
| //! cryptographically secure — it's for awk rand(), UUID generation, | ||
| //! and similar non-security uses. |
Comment on lines
+160
to
+174
| /// Read a hardware random u64 via RDRAND. Retries on transient failure. | ||
| #[inline] | ||
| fn rdrand64() -> u64 { | ||
| let mut val: u64; | ||
| let mut ok: u8; | ||
| // RDRAND can fail if the HW buffer is exhausted — retry up to 10 times | ||
| for _ in 0..10 { | ||
| unsafe { | ||
| core::arch::asm!( | ||
| "rdrand {val}", | ||
| "setc {ok}", | ||
| val = out(reg) val, | ||
| ok = out(reg_byte) ok, | ||
| options(nostack, nomem), | ||
| ); |
Comment on lines
+388
to
+392
| digest: function(algorithm, data) { | ||
| try { | ||
| const bytes = new Uint8Array(data instanceof ArrayBuffer ? data : data.buffer || data); | ||
| const result = subtleDigest(algorithm, Array.from(bytes)); | ||
| return Promise.resolve(result.buffer); |
Comment on lines
2408
to
2410
| const result = await sandbox.resetSandbox(); | ||
| bashRunnerRegistered = false; // Force re-registration on next execute_bash | ||
| if (result.success) { |
Comment on lines
5338
to
+5356
| // For sandbox memory errors, tell the LLM about the current | ||
| // heap size and how to suggest an increase to the user. | ||
| if ( | ||
| toolName === "execute_javascript" && | ||
| (toolName === "execute_javascript" || toolName === "execute_bash") && | ||
| toolResult.resultType === "failure" && | ||
| /out of memory|out of physical memory|heap|stack overflow|guest aborted/i.test( | ||
| toolResult.error ?? "", | ||
| ) | ||
| ) { | ||
| const heapMb = state.heapOverride ?? sandbox.config.heapSizeMb; | ||
| const scratchMb = | ||
| state.scratchOverride ?? sandbox.config.scratchSizeMb; | ||
| return { | ||
| additionalContext: | ||
| `Current heap: ${heapMb}MB, scratch: ${scratchMb}MB. ` + | ||
| `If the error says "Out of physical memory" or "Guest aborted: 13", ` + | ||
| `the scratch setting is too low — suggest /set scratch <MB> (e.g. double it). ` + | ||
| `For general OOM, suggest /set heap <MB>. ` + | ||
| `Or try breaking the computation into smaller pieces.`, | ||
| `IMPORTANT: "Out of physical memory" or "Guest aborted: 13" means SCRATCH is too low, NOT heap. ` + | ||
| `Call configure_sandbox({ scratch: ${scratchMb * 2} }) to double scratch memory. ` + | ||
| `For "malloc failed" or general OOM, call configure_sandbox({ heap: ${heapMb * 2} }). ` + | ||
| `For "buffer" errors, increase inputBuffer or outputBuffer.`, |
Comment on lines
38
to
39
| You have NO direct access to filesystem, network, or shell. No curl, Python. | ||
| EVERYTHING goes through sandbox tools — register_handler, execute_javascript, etc. |
Comment on lines
161
to
+165
| UNAVAILABLE: setTimeout, fetch(), Buffer, fs, process. | ||
| AVAILABLE GLOBALS: TextEncoder, TextDecoder, atob, btoa, queueMicrotask. | ||
| For Latin-1 byte encoding: import { strToBytes } from "ha:str-bytes" | ||
| No SQL, no bash, no web browsing — only sandbox tools and plugins exist. | ||
| No SQL, no web browsing — only sandbox tools and plugins exist. | ||
| For bash commands, use the execute_bash tool (separate from JavaScript handlers). |
Comment on lines
+14
to
+20
| const parts = p.split("/"); | ||
| const out = []; | ||
| for (const part of parts) { | ||
| if (part === "..") { if (out.length > 1) out.pop(); } | ||
| else if (part !== "." && part !== "") out.push(part); | ||
| } | ||
| return (p.startsWith("/") ? "/" : "") + out.join("/") || "."; |
Comment on lines
+1
to
+21
| // Build the ha:bash module for Hyperlight | ||
| // Prepends polyfills to the esbuild bundle | ||
| import { readFileSync, writeFileSync } from "node:fs"; | ||
|
|
||
| const polyfills = ` | ||
| // ── QuickJS Polyfills for just-bash ────────────────────────── | ||
| if(typeof globalThis.URL==='undefined'){globalThis.URL=class URL{constructor(input,base){let full=String(input);if(base&&!full.match(/^[a-z]+:\\/\\//i)){full=String(base).replace(/\\/[^\\/]*$/,'/')+full}const m=full.match(/^(https?:)\\/\\/([^\\/:]+)(:\\d+)?(\\/[^?#]*)?(\\?[^#]*)?(#.*)?$/i);if(m){this.protocol=m[1];this.hostname=m[2];this.port=m[3]?m[3].slice(1):'';this.pathname=m[4]||'/';this.search=m[5]||'';this.hash=m[6]||'';this.host=this.hostname+(this.port?':'+this.port:'');this.origin=this.protocol+'//'+this.host;this.href=this.origin+this.pathname+this.search+this.hash;this.searchParams=new URLSearchParams(this.search);this.username='';this.password=''}else{this.href=full;this.protocol='';this.hostname='';this.port='';this.pathname=full;this.search='';this.hash='';this.host='';this.origin='';this.searchParams=new URLSearchParams();this.username='';this.password=''}}toString(){return this.href}}} | ||
| if(typeof globalThis.URLSearchParams==='undefined'){globalThis.URLSearchParams=class URLSearchParams{constructor(init){this._p=[];if(typeof init==='string'){const s=init.startsWith('?')?init.slice(1):init;for(const pair of s.split('&')){const[k,v]=pair.split('=');if(k)this._p.push([decodeURIComponent(k),decodeURIComponent(v||'')])}}}get(k){const p=this._p.find(([a])=>a===k);return p?p[1]:null}has(k){return this._p.some(([a])=>a===k)}toString(){return this._p.map(([k,v])=>encodeURIComponent(k)+'='+encodeURIComponent(v)).join('&')}entries(){return this._p[Symbol.iterator]()}[Symbol.iterator](){return this._p[Symbol.iterator]()}forEach(fn){this._p.forEach(([k,v])=>fn(v,k))}}} | ||
| if(typeof globalThis.Buffer==='undefined'){const _e=new TextEncoder();globalThis.Buffer={from(d,e){if(typeof d==='string'){if(e==='base64'){const b=atob(d);const a=new Uint8Array(b.length);for(let i=0;i<b.length;i++)a[i]=b.charCodeAt(i);return a}return _e.encode(d)}if(d instanceof Uint8Array)return d;if(Array.isArray(d))return new Uint8Array(d);return new Uint8Array(0)},isBuffer(o){return o instanceof Uint8Array},concat(l){const t=l.reduce((s,b)=>s+b.length,0);const r=new Uint8Array(t);let o=0;for(const b of l){r.set(b,o);o+=b.length}return r},alloc(s){return new Uint8Array(s)},byteLength(s,e){if(typeof s==='string')return _e.encode(s).length;return s.length}}} | ||
| if(typeof globalThis.process==='undefined'){globalThis.process={env:{},nextTick(fn){queueMicrotask(fn)},execPath:'/usr/bin/node',mainModule:null,umask(){return 18},type:'renderer'}} | ||
| if(typeof globalThis.AbortController==='undefined'){globalThis.AbortController=class AbortController{constructor(){this.signal={aborted:false,addEventListener(){}}}abort(){this.signal.aborted=true}}} | ||
| if(typeof globalThis.crypto==='undefined'||!globalThis.crypto.randomUUID){if(!globalThis.crypto)globalThis.crypto={};globalThis.crypto.randomUUID=function(){const h='0123456789abcdef';let u='';for(let i=0;i<36;i++){if(i===8||i===13||i===18||i===23)u+='-';else u+=h[Math.floor(Math.random()*16)]}return u}} | ||
| if(typeof globalThis.setTimeout==='undefined'){globalThis.setTimeout=(fn)=>{fn();return 0};globalThis.clearTimeout=()=>{};globalThis.setInterval=()=>0;globalThis.clearInterval=()=>{}} | ||
| if(typeof globalThis.performance==='undefined'){globalThis.performance={now(){return Date.now()}}} | ||
| // ── End Polyfills ──────────────────────────────────────────── | ||
| `; | ||
|
|
||
| const bundle = readFileSync("just-bash-bundle.js", "utf-8"); | ||
| const output = polyfills + bundle; | ||
| writeFileSync("ha-bash-bundle.js", output); | ||
| console.log("ha-bash-bundle.js: " + (output.length / 1024).toFixed(0) + " KB"); |
Comment on lines
+2146
to
+2152
| // Runs bash commands inside the Hyperlight sandbox using just-bash | ||
| // (a pure-JS bash interpreter). Commands execute in the same sandbox | ||
| // as JavaScript handlers — same plugins, same baseDir, same isolation. | ||
| // | ||
| // Stateless: each call creates a fresh Bash instance. Use && or ; | ||
| // to chain commands that share state within a single call. | ||
| // |
…ght cfg The CI runs clippy without the hyperlight cfg flag, so 'extern crate alloc' is not active and alloc::format! fails with E0433. Plain format! works in both std and no_std+alloc contexts via the prelude. Signed-off-by: Simon Davies <simongdavies@users.noreply.github.com>
- lib.rs: update module docs — RDRAND not xorshift128+ PRNG
- lib.rs: add cfg(target_arch=x86_64) guard on rdrand64 with fallback
- lib.rs: use #[macro_use] extern crate alloc for format! in no_std
- lib.rs: fix crypto.subtle.digest to respect byteOffset/byteLength
on ArrayBufferViews (subarray would hash wrong data)
- node-path-stub.mjs: fix normalize('foo/..') returning 'foo' instead
of '.' — pop when out.length > 0, preserve '..' for relative paths
- index.ts: sync plugins to bashSandbox via syncPluginsToSandbox()
so curl/fs-read/fs-write/MCP work in execute_bash
- index.ts: reset bashSandbox on reset_sandbox (was leaking old sandbox)
- index.ts: separate memory error guidance for bash vs JS sandbox
(bash has fixed limits, configure_sandbox doesn't affect it)
- system-message.ts: fix 'No curl' → 'same plugins', add crypto to
AVAILABLE GLOBALS (getRandomValues, randomUUID, subtle.digest)
- Remove unused scripts/bash-bundle/build-ha-bash.mjs
Signed-off-by: Simon Davies <simongdavies@users.noreply.github.com>
- Add 'curl' to BASH_SUPPORTED_COMMANDS — it was in just-bash but missing from the allowlist, causing 'command not found' - /sessions now shows full session UUIDs (not truncated) so users can copy-paste into /resume - /sessions -all now works as alias for --all - /resume without an ID shows interactive numbered picker (top 20 most recent sessions) — type a number to resume, like az CLI subscription selector Signed-off-by: Simon Davies <simongdavies@users.noreply.github.com>
- curl was in BASH_SUPPORTED_COMMANDS but just-bash's curl needs a fetch function to make HTTP requests. Build a Web Fetch adapter over host:fetch (fetchText/post) and pass it to the Bash constructor. curl now works when the fetch plugin is enabled. - Fix arrow keys printing ^[[A after slash commands: use rl.prompt() instead of raw process.stdout.write() in questionCapturingPaste() so readline properly manages terminal raw mode Signed-off-by: Simon Davies <simongdavies@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds an execute_bash tool based on just_bash