WasiWorker for the web

[olanod]

Perhaps this belongs more in the web standards repositories, perhaps in both.
At first it might seem that WASI is all about running WebAssembly outside of the web and staying as far from the browser as possible but I see a lot of potential in browsers being the go to WASI runtime, the discoverability of the web is a great way to distribute and install WASI based applications and both worlds can benefit a lot from each other.

A WasiWorker could have would have a similar API as the other types of workers

let worker = new WasiWorker('app.wasi', { type: 'background', ... otherOpts });
worker.addEventListener('message', e =>  {});
worker.postMessage('foobar');
// or 
navigator.wasiWorker.register('app.wasi', {...opts});

The options object could have several interesting possibilities like specity the kind of worker, maybe is a background type that is installed in a similar way to service workers and can live as a system service? maybe a onetime/simple worker that executes a task and returns? in that case maybe the register function only compiles it and then several instances can be fired later with different context data.
Other options would include the capabilities to request access to the file system, network sockets, etc. another interesting option can be something like a virtual file system.

new WasiWorker({
    vfs: {
        '/dev/camera_feed': new ReadableStream({}),
        '/smiling_face.jpg': new WritableStream({}),
        stdout: consoleWriterStream,
    }
    ...opts,
})

The virtual filesystem I think is already a good way to communicate with the WASI process but it can also communicate in a web worker like fashion if an interface types API for the onmessage/postMessage or broadcast API is formalized. Such interface can also include other callbacks to control the life cycle of the app like service workers do.

Would love to hear opinions :) I personally think it would be the ultimate marriage, UIs still developed with web technologies at the reach of a https://my.app and leaving the business logic and heavy lifting to a headless WASI service or set of services that can access the file system, hardware and network in a more traditional but secure way. For the decentralized web use case for example, no need to do hacks trying to get webrtc in a service worker, a performant p2p node that would run fine in any other WASI runtime could be spawned on the web and work as expected because it was explicitly granted the right set of permissions.

[sunfishcode]

Interesting ideas! As you observe, the WASI Subgroup itself doesn't have the scope to add new APIs to the Web platform. And any proposal to add a way to let Web content access host filesystems, networks, or hardware will of course require careful consideration. But that said, we can certainly talk about the idea and see where it goes.

One of the things we'd need to do here (and which would be good do do in general) is to generalize WASI's support for read and write operations into more general stream mechanisms, possibly compatible with ReadableStream and WriteableStream.

[guest271314]

Currently attempting to create a solution to the issue of Native File System getFile() and File.stream() limitations as to a live file handle where a native application is writing to a local file https://bugs.chromium.org/p/chromium/issues/detail?id=985665, https://bugs.chromium.org/p/chromium/issues/detail?id=1084880 it is currently not possible to use the Native File System API to stream the latest write to the browser. In order to read the most recent bytes of the file the entire file needs to be read from the beginning then sliced from the last offset.

One limitation of Native File System to get files written, for example, by

parec -v --raw -d alsa_output.pci-0000_00_1b.0.analog-stereo.monitor | split -b 512 -d -a 1 - output

is that when getEntries() is executed the order of the, for example, 713 files, is not guaranteed to be in any order https://wicg.github.io/native-file-system/#api-filesystemdirectoryhandle-asynciterable, which would require another program to be run to sort the current entries before converting to audio output, before running the code again and repeating the process until the original process is ended.

Otherwise using Native File System and inotify-tools am able to execcute arbitrary code locally at the get the result in the browser as files.

The current use case are workarounds for Chromium implementation of getUserMedia({audio:true}) not providing a means to capture PulseAudio monitor device "What-U-Hear" https://github.com/guest271314/captureSystemAudio. Have created code to write data captured from to one or more files using native parec and split. Getting the file to the browser would be simpler if STDOUT could be written directly to shared memory that is accessible by WebAssembly.memory.buffer.

Some concrete example configurations include extending transferable streams and MessagePort to be a TransformStream between the browser and WASI in a specific local directory with permissions granted, similar to how Native File System showDirectory() and requestPermissions({writable: true}) allows read and write to local directory, backed by at least one growable https://github.com/tc39-transfer/proposal-resizablearraybuffer WebAssembly.Memory, the concept from perspective here is described at https://bugs.chromium.org/p/chromium/issues/detail?id=910471#c19.

Would add to the suggestion

const {readable, writable} = new WasiNativeStream({configuration})

where there is a direct connection between native application, with permissions granted, and the browser, to execute arbitrary native code in the directory, an updated version of Native Messaging.

Instead of MessageEvent that contains largely unused properties, {value, done} of readable alone can be used. The configuration can be read/write in either direction.

[guest271314]

@sunfishcode A concrete use case #307 for this proposal. Is this within the purview of WASI?

[sunfishcode]

In general, WASI is not a forum for working around browser limitations.

Also, WASI is about WebAssmbly APIs, and const {readable, writable} = new WasiNativeStream({configuration}) is JS syntax. For the WASI subgroup to be able to consider an idea or proposal, it should ideally have a clear path to being expressible in terms of WebAssembly features (standardized or proposed).

[guest271314]

@sunfishcode Alright. Have no experience writing WebAssembly save for using Memory and no experience writing wat by hand. However, when decide to achieve a specific goal, do not place limitations on self as to how to achieve that goal. The goals mentioned audio processing and devices, which is what am currently working on. Thus it is reasonable to try WASI to achieve own goal. If that is not possible, so be it. Must ask the question anyway to get the answer. Cheers.

[linclark]

The original idea in the OP is indeed interesting. There are a lot of more foundational pieces that will need to be built before it makes sense to tackle this, and the Web API space will almost certainly have changed by then, so I'm going to close this one out for now. But I could definitely see something in this direction being a topic to explore in a couple of years.

[JohnStarich]

@olanod For what it's worth, I've made a demo along these lines: demo, repo
Go Wasm is not using workers per-se, but swapping out processes and the file system for use in Go.

Unfortunately it depends on a binding layer in JS, so I'm also interested where these ideas will go – even if it isn't the right time or place for WASI.

[guest271314]

another interesting option can be something like a virtual file system.

new WasiWorker({
    vfs: {
        '/dev/camera_feed': new ReadableStream({}),
        '/smiling_face.jpg': new WritableStream({}),
        stdout: consoleWriterStream,
    }
    ...opts,
})

This is possible using WebTransport. I have no experience with WASI, though if you have WASI code you can execute that code and get STDOUT to the browser, for example, streaming audio in "real-time", growing a dynamic WebAssembly.Memory instance using the obsolete QuicTransport (https://groups.google.com/a/chromium.org/g/web-transport-dev/c/1h6EhmYh2vQ) https://github.com/guest271314/quictransport/blob/main/audioworklet-webassembly-memory-grow/quicTransportAudioWorkletMemoryGrow.js, in that case streaming raw PCM from parec and WAV from espeak-ng; using shell scripts called from Python subprocess. If WASI is capable of the functionality it should be possible to substitute WASI code for the shell script, and the server for quic-transport protocol.

Relevant to the use-case of streaming media, this code

#!/bin/bash
  SOURCE_OUTPUT_INDEX=`pactl list source-outputs | grep -E "(Source\sOutput|application\.name)" | grep -E "Chromium input" -B1 | grep -o -E -m1 "[0-9]+"`
  SOURCE_INDEX=`pactl list sources | grep -E "(Source\s|Description:\s)" | grep "Description: $1" -B1 | grep -o -E -m1 "[0-9]+"`
  pactl move-source-output $SOURCE_OUTPUT_INDEX $SOURCE_INDEX
  echo "{\"source_output\":\"Chromium input\", \"source_output_index\":\"$SOURCE_OUTPUT_INDEX\", \"source\":\"$1\", \"source_index\":\"$SOURCE_INDEX\"}"

var track; 
async function setUserMediaAudioSource(source) {
  const quicTransportServerURL = `quic-transport://localhost:4433/toggle_source`;
  try {
    const transport = new WebTransport(quicTransportServerURL);
    console.log(transport, source);
    transport.onstatechange = async (e) => {
      console.log(e);
    };
    await transport.ready;
    transport.closed
      .then((reason) => {
        console.log('Connection closed normally.', { reason });
      })
      .catch((e) => {
        console.error(e.message);
        console.trace();
      });
    const sender = await transport.createUnidirectionalStream();
    const writer = sender.writable.getWriter();
    const encoder = new TextEncoder('utf-8');
    let data = encoder.encode(source);
    await writer.write(data);
    await writer.close();
    const reader = transport.incomingUnidirectionalStreams.getReader();
    console.log({ transport, sender, reader });
    const result = await reader.read();
    if (result.done) {
      console.log(result);
      return;
    }
    let stream = result.value;
    console.log({ stream });
    const { readable } = stream;
    const sources = await new Response(readable).json();
    console.log({ reader, transport });    
    await reader.cancel();
    transport.close({ closeInfo: { reason: 'Done downloading' } });
    console.log(await reader.closed, await stream.readingAborted);
    return sources;
  } catch (e) {
    console.warn(e.message);
    throw e;
  }
}

navigator.mediaDevices.ondevicechange = e => console.log(e);
navigator.mediaDevices.getUserMedia({audio: true})
.then(stream => {
  track = stream.getTracks()[0];
  console.log(stream, track, track.id);
  return navigator.mediaDevices.enumerateDevices()
})
.then(devices => {
  console.log(devices);
  return setUserMediaAudioSource('Monitor of Built-in Audio Analog Stereo');
})
.then(sources => {
  console.log(sources);
  return navigator.mediaDevices.enumerateDevices() // Monitor device will not be listed at Chromium
})
.then(devices => console.log(devices))
.catch(e => console.error(e));
//...
// setUserMediaAudioSource('<device_description>')

in the Python script at Google Samples

        if isinstance(event, StreamDataReceived):
            if event.data:
                print('event.data', event.data, event.end_stream)
                data = subprocess.run(['./toggle_source.sh', event.data], capture_output=True)
                self.payload = data.stdout
                print(data.stdout)

dynamically changes the audio source of a MediaStreamTrack from getUserMedia() programmatically, outside of the Web API (Chromium refuses to list or capture monitor devices at *nix that uses PulseAudio); next I will compose getUserMediaAudioSources() to reflect accurate current devices (users connect and disconnect various physical and virtual devices during a media capture session at any given time).

Thus if WASI is capable of performing the processing already, you can create a WASI server where the ReadableStreams will be whatever you set in WASI; video; audio; speech; image; data; et al.

In fact, a configurable WASI server for WebTransport, perhaps even the ability to compile JavaScript to WASI, so we can write the server in JavaScript, will be an interesting experiment.

The Charter https://github contains affirmative language at "will consider" as to the goals

Scope
The Subgroup will consider topics related to system interface APIs, including:

• APIs for host filesystems, network stacks, and other resources.
• APIs for graphics, audio, input devices

Re

There are a lot of more foundational pieces that will need to be built before it makes sense to tackle this,

The capability already exists right now, as demonstrated by the above example code using QuicTransport and WebTransport https://github.com/guest271314/samples-1/tree/gh-pages/webtransport.

I see no reason why we cannot experiment with substituting a WASI quic-transport server for a Python quic-transport server, as OP describes, which is consistent with the goals at Charter.