setMemory fails silently in transform (regression?)

[mnater]

Bug description

I'm using the following transform to prefill wasm memory with ~100kb of data:

import {Transform} from "assemblyscript/transform";
import fs from "fs";

const data = fs.readFileSync(`./foo.data`);

const dataOffset = 1920;
const dataSize = data.buffer.byteLength;
const heapSizeBytes = Math.max(
    Math.ceil((dataOffset + dataSize) / 65536) * 65536,
    65536
);

const heapSizePages = (heapSizeBytes / 1024 / 64);

export default class MyTransform extends Transform {
    afterCompile(asModule) {
        this.log("  [mytransform.js] add data (" + dataSize + " Bytes)");
        asModule.setMemory(heapSizePages, -1, "mem", [
            {
                data,
                "offset": asModule.i32.const(dataOffset)
            }
        ]);
    }
}

This worked great up to version 0.26.0. But with the update to 0.26.1 it stopped working. I haven't noticed until now, because here's no error message – the compiler just doesn't output the .wasm file.

I'm not sure if this is an issue with asc or with Binaryen. The release notes of 0.26.1 say that binaryen is updated, so I checked the binaryen source and it looks like the code of setMemory has changed. Notably allocate has been replaced by stackAlloc. So maybe there's a stack overflow happening?

Shall I report this in the binaryen repo?
Is there a more elegant way to prefill memory with data in asc (instead of using transforms)?

Steps to reproduce

Use the transform from above.

AssemblyScript version

v0.27.1

[MaxGraey]

Your signature is wrong. setMemory was changed and looks as:
https://github.com/AssemblyScript/assemblyscript/blob/main/src/module.ts#L2394

[mnater]

Thanks for the answer, I still don't get it.

asModule.setMemory calls binaryen.js -> setMemory() which has the signature setMemory(initial: number, maximum: number, exportName?: string | null, segments?: MemorySegment[] | null, shared?: boolean, memory64?: boolean, internalName?: string): void; and not assemblyscript.js -> setMemory().

How can I call assemblyscript.js -> setMemory() from within the afterCompile-hook?

[MaxGraey]

asModule.setMemory calls setMemory of Module which internally uses binaryen's _BinaryenSetMemory. In your example asModule.setMemory has wrong order of arguments. Currently, it slightly different. See prev comment and actual signature for setMemory

[mnater]

Yes, I'm aware of the different order of arguments. There are two functions called setMemory:

1. assemblyscript.js

 setMemory(initial: Index, maximum: Index, segments: MemorySegment[], target: Target, exportName?: string | null, name?: string, shared?: boolean): void;

2. binaryen.js

setMemory(initial: number, maximum: number, exportName?: string | null, segments?: MemorySegment[] | null, shared?: boolean, memory64?: boolean, internalName?: string): void;

asModule.setMemory calls the second (binaryen.js) function. How can I call the first (assemblyscript.js) function from within the afterCompile-hook?

[MaxGraey]

AS doesn't use second version at all. As well as high level js API of binaryen.js. You can only use first version

[mnater]

Thank you for your patience.

I think I found the issue:

assemblyscript/cli/index.js

Line 737 in 7a35ff8

// From here on we are going to use Binaryen.js

It looks like asModule gets converted to a Binaryen-module for the afterCompile-hook.

If I replace binaryenModule with module on line 752 asModule.setMemory uses the first version from the comment above and then everything works as expected. Could you please have a look at it?

[HerrCai0907]

@mnater check again, you are right, in afterCompile is binaryen module. The type of this module is in binaryen.js.

If you want to use assemblyscript module, you can get it in afterInitialize stage.

  afterInitialize(program) {
    let asModule = program.module;
    console.log("asModule", asModule);
  }

[HerrCai0907]

I cannot reproduce it in my mac m1, node v16.19.1.
Which environment are you using?

[mnater]

@HerrCai0907 Thanks, that's indeed a way to access the assemblyscript module. But I fear to run in to other issues, when I setMemory at this early stage.

I'm on an old mac i7, node 19.6.0
But I don't think that's important here?

[mnater]

@MaxGraey: what is the correct way to set up the MemorySegement-argument in setMemory?

https://github.com/AssemblyScript/assemblyscript/blob/main/src/module.ts#L1359 says offset is of type i64, but as of L2408 setMemory expects an ExpressionRef.

If offset is of type ExpressionRef the call to i64_low(offset) on L2418 returns undefined. And i64_new isn't exported so I can't generate an offset of type i64.

[MaxGraey]

instead i64_new, i64_add ant etc you can use long.js module which also uses by AssemblyScript for js

[mnater]

@HerrCai0907 setMemory doesn't work in afterInitialize because the memory gets overwritten by

assemblyscript/src/compiler.ts

Line 731 in 9497c3d

private initDefaultMemory(memoryOffset: i64): void {

So this approach seems a dead end.

So I'm stuck with setMemory in afterCompile which is the setMemory function defined in binaryen.js (see #2670 (comment)).

So my initial question remains: how can I prepopulate memory with (a large amount of) data?
It was so easy in v0.26 and now I'm bothering you guys since days 🤦‍♂️

[mnater]

WebAssembly/binaryen#5595