error in importing wasi binary into a Go application

[hassan-shahbazi]

Description

I have a Go application that uses wasmer runtime to work with some wasi binaries. When I use Rust to write and generate wasm binary, I can access and use both imported and exported functions and it works perfectly fine.

import (
    wasm "github.com/wasmerio/go-ext-wasm/wasmer"
)

func main() {
    bytes, error := wasm.ReadBytes("binary.wasm")
    if error != nil {
        panic(err)
    }

    instance, err := wasm.NewInstance(bytes)
    if err != nil {
        panic(err)
    }

    println("instance loaded!", instance)
}

However, when I try to import a binary generated by swiftwasm, it gives the following error:

Failed to instantiate the module:
            28 link errors: (1 of 28) Import not found, namespace: wasi_snapshot_preview1, name: proc_exit (2 of 28) Import not found, namespace: wasi_snapshot_preview1, name: args_sizes_get (3 of 28) Import not found, namespace: wasi_snapshot_preview1, name: args_get (4 of 28) Import not found, namespace: wasi_snapshot_preview1, name: sched_yield (5 of 28) Import not found, namespace: wasi_snapshot_preview1, name: fd_read (6 of 28) Import not found, namespace: wasi_snapshot_preview1, name: fd_write (7 of 28) Import not found, namespace: wasi_snapshot_preview1, name: fd_close (8 of 28) Import not found, namespace: wasi_snapshot_preview1, name: clock_time_get (9 of 28) Import not found, namespace: wasi_snapshot_preview1, name: poll_oneoff (10 of 28) Import not found, namespace: wasi_snapshot_preview1, name: fd_fdstat_get (11 of 28) Import not found, namespace: wasi_snapshot_preview1, name: path_open (12 of 28) Import not found, namespace: wasi_snapshot_preview1, name: path_filestat_get (13 of 28) Import not found, namespace: wasi_snapshot_preview1, name: path_rename (14 of 28) Import not found, namespace: wasi_snapshot_preview1, name: path_filestat_set_times (15 of 28) Import not found, namespace: wasi_snapshot_preview1, name: path_link (16 of 28) Import not found, namespace: wasi_snapshot_preview1, name: path_create_directory (17 of 28) Import not found, namespace: wasi_snapshot_preview1, name: fd_readdir (18 of 28) Import not found, namespace: wasi_snapshot_preview1, name: path_readlink (19 of 28) Import not found, namespace: wasi_snapshot_preview1, name: path_symlink (20 of 28) Import not found, namespace: wasi_snapshot_preview1, name: fd_prestat_get (21 of 28) Import not found, namespace: wasi_snapshot_preview1, name: fd_prestat_dir_name (22 of 28) Import not found, namespace: wasi_snapshot_preview1, name: path_unlink_file (23 of 28) Import not found, namespace: wasi_snapshot_preview1, name: path_remove_directory (24 of 28) Import not found, namespace: wasi_snapshot_preview1, name: random_get (25 of 28) Import not found, namespace: wasi_snapshot_preview1, name: environ_sizes_get (26 of 28) Import not found, namespace: wasi_snapshot_preview1, name: environ_get (27 of 28) Import not found, namespace: wasi_snapshot_preview1, name: fd_fdstat_set_flags (28 of 28) Import not found, namespace: wasi_snapshot_preview1, name: fd_seek

This is my swift file:

print("Hello World!")

and I use the following command to generate the wasi binary, according to the guide here

$ TOOLCHAIN_PATH = $HOME/.swiftenv/versions/wasm-DEVELOPMENT-SNAPSHOT-2020-09-29-a/usr/share
$ swiftc -target wasm32-unknown-wasi -sdk $TOOLCHAIN_PATH/wasi-sysroot binary.swift -o binary.wasm

Also, running the binary directly with wasmer does not have any issues:

$ wasmer binary.wasm
Hello World!

[MaxDesiatov]

Thanks for creating the issue! 5.3 snapshots should be more stable in general, do you get the same results with wasm-5.3-SNAPSHOT-2020-09-28-a?

My current hypothesis is that Rust binaries somehow don't link these symbols, and you need to configure Go Wasmer bindings to link with WASI symbols (those do look like WASI symbols to me). But I may be wrong, I never used Go Wasmer bindings before.

@kateinoigakukun have you seen any issues like this before?

[kateinoigakukun]

@hassan-shahbazi
You need to use InstantiateWithImportObject instead of NewInstance to import wasi functions. Please refer to this example.

package main

import (
	wasm "github.com/wasmerio/go-ext-wasm/wasmer"
)

func main() {
	bytes, _ := wasm.ReadBytes("hello.wasm")
	module, _ := wasm.Compile(bytes)
	wasiVersion := wasm.WasiGetVersion(module)
	importObject := wasm.NewDefaultWasiImportObjectForVersion(wasiVersion)
	instance, _ := module.InstantiateWithImportObject(importObject)
	defer instance.Close()

	instance.Exports["_start"]()
}

[MaxDesiatov]

Hassan, please let us know if that resolves your issue. 🙂

[hassan-shahbazi]

@kateinoigakukun
Thanks for your help. I could import a swift binary wasm, however, using wasm.Compile(bytes) seems to make a significant performance cost. When I use wasm.NewInstanceWithImports(bytes, imports), 10 tests run only in 0.632s for a complex Rust binary, but with the Compile API, it takes 17.50s to run a single test for a simple one-line Swift binary.
I'm wondering now how I can use DefaultWasiImportObject without compiling the binary.

[kateinoigakukun]

@hassan-shahbazi SwiftWasm uses wasi_snapshot1_preview version of wasi module. So if you can assume that the input wasm binary is always Swift binary, you can create wasi import object by wasm.NewDefaultWasiImportObjectForVersion(wasm.Snapshot1).

But I'm not well sure how to use the wasmer-go, so please open an issue on wasmer-go repo and ask them.

[hassan-shahbazi]

@kateinoigakukun and @MaxDesiatov
Thanks for you incredible helps. Despite the performance costs, it seems that it works fine now.
However, I cannot have access to any public functions on the Swift file except for _start() which looks to run the swift file. Do we have any examples about how to export/import functions? Something like the example for Rust here: https://github.com/wasmerio/wasmer-go#imported-function

[kateinoigakukun]

@hassan-shahbazi

You can expose a Swift function for host environment using special attribute and linker option.

@_cdecl("add")
func add(_ lhs: Int, _ rhs: Int) -> Int {
    return lhs + rhs
}

You need to compile the Swift code with linker option --export.

$ TOOLCHAIN_PATH=$(dirname $(swiftenv which swiftc))/../
$ swiftc \
    -target wasm32-unknown-wasi \
    -sdk $TOOLCHAIN_PATH/share/wasi-sysroot \
    lib.swift -o lib.wasm \
    -Xlinker --export=add

Then, you can use the exported function from host environment.

const WASI = require("@wasmer/wasi").WASI;
const WasmFs = require("@wasmer/wasmfs").WasmFs;

const promisify = require("util").promisify;
const fs = require("fs");
const readFile = promisify(fs.readFile);

const main = async () => {
  // Instantiate a new WASI Instance
  const wasmFs = new WasmFs();
  let wasi = new WASI({
    args: [],
    env: {},
    bindings: {
      ...WASI.defaultBindings,
      fs: wasmFs.fs,
    },
  });

  const wasmBinary = await readFile("lib.wasm");

  // Instantiate the WebAssembly file
  let { instance } = await WebAssembly.instantiate(wasmBinary, {
    wasi_snapshot_preview1: wasi.wasiImport,
  });
  // Get the exported function
  const addFn = instance.exports.add;
  console.log("2 + 3 = " + addFn(2, 3))

};

main()

If you use SwiftPM package, you can omit linker flag using clang's __atribute__. Please see swiftwasm/JavaScriptKit#91 for more detail info

[hassan-shahbazi]

@kateinoigakukun
Awesome update, thanks a million. I could successfully call exposed Swift functions in my Go application.
If I'm not mistaken, this is an example of exported functions where the host env calls functions from the binary. But what about imported functions where the binary calls functions from the env? This is done by defining extern functions in Rust as described in the link I had posted: https://github.com/wasmerio/wasmer-go#imported-function

[MaxDesiatov]

You'll have to declare your functions in a header file and provide a module map file. You can find an example in JavaScriptKit, which consumes JavaScript functions from browser hosts that are defined in the runtime part of JavaScriptKit. That module with the header and the module map file then needs to be declared as a separate target in Package.swift.

[MaxDesiatov]

Another option is to use the @_silgen_name attribute, but this attribute is underscored, which means it can go away in future versions of Swift. I feel that a separate target with a header and a module map is safer from that perspective. @kateinoigakukun maybe both approaches should be documented in the SwiftWasm book?

[kateinoigakukun]

@MaxDesiatov Yes, both approaches should be documented. But I think we should provide a safe way to extern a Swift function without C code in the near future.

[MaxDesiatov]

Are you referring to what I mentioned in #1503 previously?

[kateinoigakukun]

Or if we can provide a similar feature like Clang's __attribute__((__import_name__))