How are reading stdin and writing to stdout implemented for native executable and WASI?

[guest271314]

How are reading stdin and writing to stdout implemented for native executable and WASI?

[mateuscmtropical]

With --host wasi both go through WASI Preview 1 file descriptors.

Writing: console.log, console.info and console.debug call fd_write on fd 1. console.warn and console.error use fd 2. The value is stringified inside the module, so nothing on the host side is needed (module/console.js#L194-L198).

Reading: there is a readStdin() global. It calls fd_read on fd 0 in a loop until a read returns 0 bytes, doubles its buffer when it fills up, and returns the bytes as a string (module/console.js#L127-L156). It only exists with host: 'wasi'. With the default js host the compile fails with 'readStdin' is not in scope.

Tested with jz 0.9.2 and Node 24.14.1:

// echo.js
export let _start = () => {
  let s = readStdin()
  console.log("got", s.length, "bytes:", s)
}

jz echo.js --host wasi -o echo.wasm

The module imports only fd_read and fd_write. With Node's own WASI:

import { WASI } from 'node:wasi'
import { readFileSync } from 'node:fs'
const wasi = new WASI({ version: 'preview1' })
const mod = await WebAssembly.compile(readFileSync('echo.wasm'))
const inst = await WebAssembly.instantiate(mod, wasi.getImportObject())
wasi.start(inst)

$ node run.mjs < input.txt
got 12 bytes: hello stdin

In a browser or any host without WASI, the bundled jz/wasi shim implements just these imports. fd_write goes to process.stdout/stderr, or to console when there is no process. fd_read is served by an opts.read(fd, buf) callback that you supply and that returns the number of bytes written into buf (wasi.js#L48-L59). Without opts.read, stdin reads as empty.

import { instantiate } from 'jz/wasi'
const input = new TextEncoder().encode('hello from opts.read')
let off = 0
const inst = instantiate(wasm, {
  read: (fd, buf) => {
    const n = Math.min(buf.length, input.length - off)
    buf.set(input.subarray(off, off + n))
    off += n
    return n
  },
})
inst.exports._start() // got 20 bytes: hello from opts.read

For the native executable in scripts/native, there is no stdio layer at all. That pipeline (jz, then wasm-opt, wasm2c and clang) is built around the watr benchmark. harness.c reads the input file with fread, copies it into linear memory through the exported _alloc, calls the export and reads the result string back out of memory. env-stubs.c only stubs the __ext_* imports. So in that build the C harness does the I/O and the wasm never touches a file descriptor.

[guest271314]

I'm looking in to using jz to compile to native executable and WASM for a Native Messaging host. Reading and writing strings won't do.

I'll poke around later.

[guest271314]

@mateuscmtropical

Looks like readStdin() is not parsing the "32-bit message length in native byte order" or console.log() is not printing the data

echo 'export let _start = () => {
  let s = readStdin()
  console.log(s);
}' > echo.js

bun x jz echo.js --host wasi
echo.js → echo.wasm (4556 bytes)

printf '\x0d\x00\x00\x00{"ping":true}' | wasmtime echo.wasm
{"ping":true}

[dy]

@guest271314 the header is there; the terminal hides it. 0x0d is a carriage return and 0x00 prints nothing:

$ printf '\x0d\x00\x00\x00{"ping":true}' | wasmtime echo.wasm | xxd
00000000: 0d00 0000 7b22 7069 6e67 223a 7472 7565  ....{"ping":true
00000010: 7d0a                                     }.

It breaks for real once a length byte reaches 0x80: readStdin() decodes UTF-8, so \xc8 becomes U+FFFD (ef bf bd), and console.log writes text plus a newline. readStdin() also reads to EOF, which a browser never sends while the port is open. Strings are the wrong tool here, so:

jz now has byte-level stdio under --host wasi: fs.readSync(fd, buffer, offset, length) and fs.writeSync(fd, buffer | string, offset, length), with Node's signatures and errors (9f3afa7, on main, in the next release). The same file runs under node host.js, wasmtime host.wasm and node:wasi with byte-identical output:

import fs from 'fs'
const readFull = (buf) => {
  let n = 0
  while (n < buf.length) { const r = fs.readSync(0, buf, n, buf.length - n); if (r === 0) return false; n += r }
  return true
}
const head = new Uint8Array(4), h = new Uint8Array(4)
const serve = () => {
  const body = new Uint8Array(head[0] | head[1] << 8 | head[2] << 16 | head[3] << 24)
  if (!readFull(body)) return false
  const out = new TextEncoder().encode(JSON.stringify({ echo: JSON.parse(new TextDecoder().decode(body)) }))
  h[0] = out.length; h[1] = out.length >> 8; h[2] = out.length >> 16; h[3] = out.length >> 24
  fs.writeSync(1, h); fs.writeSync(1, out)
  return true
}
export let _start = () => { while (readFull(head) && serve()); }

jz host.js --host wasi -o host.wasm

Notes:

• Keep per-message work in a function like serve. jz has no GC and reclaims a call's memory when it returns: this loop stays at one 64 KB page over 10,000 messages; written inline in the while, it grows.
• Use import fs from 'fs'; node:fs isn't mapped yet.
• For a native executable, the wasm2c pipeline has no WASI layer yet, so point the manifest at a script that runs wasmtime host.wasm.
• nm_typescript.ts as written still needs node:process streams, async iteration over stdin and a resizable ArrayBuffer, which jz doesn't provide. The generator error is Incorrect variable is declared twice error #117, fixed on main.

[guest271314]

I did this

git clone https://github.com/dy/jz.git

bun install --trust

bun build --target node --outfile=nm_jz.js /home/user/native-messaging-js2wasm/nm_js2wasm_node_fs.ts

Source: https://github.com/loopdive/js2/blob/main/examples/native-messaging/nm_js2wasm_node_fs.ts

Modified the import to

import { readSync, writeSync } from "fs";

bun cli.js nm_jz.js --host wasi -o nm_jz.wasm
fs.readSync(fd, buffer, offset) is an options call in Node; pass (fd, buffer, offset, length)
  at line 208:18
    return readSync(0, buf, { offset: 0, length: buf.length });
                   ^
  current AST: ["()","fs.readSync",[",",[null,0],"buf",["{}",[":","offset",[null,0]],[":","length",[".","buf","length"]]]]]

Modify the signature

//! https://github.com/loopdive/js2/blob/main/examples/native-messaging/nm_js2wasm_node_fs.ts
function nodeFsRead(buf) {
  return readSync(0, buf, 0, buf.length/* { offset: 0, length: buf.length } */);
}

bun cli.js nm_jz.js --host wasi -o nm_jz.wasm
warning[deopt-dyn-write] (21:28): dynamic property write `buff2_1[…] = …` couldn't resolve a static type — it falls back to a runtime hash store (~2× slower than a typed/slot write, far worse in a hot loop). Use a literal key, a numeric typed-array index, or a Map for genuinely dynamic keys.
nm_jz.js → nm_jz.wasm (41100 bytes)

I think the warning is generated here, not sure though

function readFillExact(read, buf, start, n) {
  let got = 0;
  while (got < n) {
    const tmp = new Uint8Array(n - got);
    const r = read(tmp);
    if (r === null || r <= 0)
      return false;
    let i = 0;
    while (i < r) {
      buf[start + got + i] = tmp[i];
      i = i + 1;
    }
    got = got + r;
  }
  return true;
}

Test using this https://github.com/guest271314/NativeMessagingHosts/blob/main/nm_standalone_test.js. It works.

jz$ ~/bin/nm_standalone_test.js wasmtime ./nm_jz.wasm
{ path: "wasmtime", allowed_origin: "./nm_jz.wasm" }
Testing wasmtime Native Messaging host

[host] received 1048580 chars, declared body length 1048576
{ messageLength: 1048576 }
{
  message: [
    null, null, null, null, null, null, null, null, null, null,
    null, null, null, null, null, null, null, null, null, null,
    null, null, null, null, null, null, null, null, null, null,
    null, null, null, null, null, null, null, null, null, null,
    null, null, null, null, null, null, null, null, null, null,
    null, null, null, null, null, null, null, null, null, null,
    null, null, null, null, null, null, null, null, null, null,
    null, null, null, null, null, null, null, null, null, null,
    null, null, null, null, null, null, null, null, null, null,
    null, null, null, null, null, null, null, null, null, null,
    ... 209615 more items
  ],
  jsonLength: 1048576
}
[host] received 10 chars, declared body length 6
{ messageLength: 6 }
{ message: "test", jsonLength: 6 }
[host] received 6 chars, declared body length 2
{ messageLength: 2 }
{ message: "", jsonLength: 2 }
[host] received 5 chars, declared body length 1
{ messageLength: 1 }
{ message: 1, jsonLength: 1 }
[host] received 12 chars, declared body length 8
{ messageLength: 8 }
{ message: { "0": 97 }, jsonLength: 8 }
[host] received 67108805 chars, declared body length 67108801
{ messageLength: 1048571 }
{ messageLength: 1048571 }
# ...
{ outputLength: 13211982, totalMessageLength: 13421760 }
{ messageLength: 1048571 }
{ messageLength: 321 }
{ outputLength: 13421696, totalMessageLength: 13421760 }
{ outputLength: 13421760, totalMessageLength: 13421760 }
{ totalMessageLength: 13421760, outputLength: 13421760 }