Spec tests

[tlively]

When it comes time to write up SIMD spec tests, it will be useful to be aware of the tests that exist in other places.

Binaryen: https://github.com/WebAssembly/binaryen/blob/master/test/spec/simd.wast
Wabt: https://github.com/WebAssembly/wabt/tree/master/test/interp simd-*.txt

[AndrewScheidecker]

I also wrote some pretty limited tests for WAVM here.

Trying the binaryen and wabt tests in WAVM I noticed a few incompatibilities:

• wabt implements SIMD compares to return 1 for true comparisons, but the proposed spec is to return -1 for true comparisons. Binaryen (and now WAVM) return -1 for true comparisons.
• Neither the Binaryen or wabt tests use the proposed v128.const syntax.
• wabt has a v8x16.shuffle test that uses a 4xi32 immediate. WAVM and Binaryen use a 16xi5 syntax.

[gnzlbg]

wabt implements SIMD compares to return 1 for true comparisons, but the proposed spec is to return -1 for true comparisons.

That's a bug, I've sent a PR that fixes that: WebAssembly/wabt#1031

[tlively]

• wabt has a v8x16.shuffle test that uses a 4xi32 immediate. WAVM and Binaryen use a 16xi5 syntax.

I submitted #60 to standardize this.

[gnzlbg]

wabt has a v8x16.shuffle test that uses a 4xi32 immediate. WAVM and Binaryen use a 16xi5 syntax.

I've implemented that (hopefully correctly) in WABT WebAssembly/wabt#1032

[tlively]

@gnzlbg Thanks for your work on this! Are you planning to update WABT's v128.const syntax as well? No worries if not.

[gnzlbg]

Are you planning to update WABT's v128.const syntax as well?

I've updated the v128.const i32 syntax to use v128.const i32x4 syntax here: WebAssembly/wabt#1033

But that PR does not implement any of the other constructors (e.g. v128.const i64x2 errors).

[gnzlbg]

So the syntax part is fixed. I think this issue can be closed.

[tlively]

@gnzlbg this is a tracking issue for having spec tests, which we still don't have. We have a bunch of tests for different tools that can be used as a starting point for writing spec tests, though.

[Honry]

@tlively and all,

I've been involved and improved the test coverage of WASM spec tests (My PRs) and would like to contribute spec tests for SIMD as well. I've already made a rough plan with @arunetm internally, and now back to this thread to seek feedback from community.

Test Design:
We will create WAST tests to align with the test design and test harness of the WASM core spec tests to reduce efforts of upstreaming SIMD tests to WASM spec repo in future. Existing tests from various tools can be used as a starting point, including WAVM, WABT, and Binaryen.

Test Framework:
Before we have spec interpreter implementation for SIMD, I think WAVM is a good choice to be used as the test target since which has already supported all core proposal now and one more important point is that its test framework supports almost all the assertions of the WASM interpreter.

Test Allocation:

• test/core

Test Validation:
How to make sure tests pass on WAVM? Local validation or is that possible to integrate test framework from WAVM and enable it on CI for each test PR? The latter one seems not make sense as it's better to contribute tests to WAVM directly. thoughts?

This is really a rough plan and I may miss some other details. Please kindly provide your comments and suggestions, very appreciate!

[rossberg]

Is there a reason not to put the tests into test/core?

[Honry]

Is there a reason not to put the tests into test/core?

@rossberg, yeah, it could be another option, I just missed it. So how about a new folder test/core/simd?

[binji]

For now, we've been putting tests for a proposal directly in the test/core directory in the proposal's repository. It seems reasonable to do the same for SIMD.

[Honry]

@binji, thanks for the info! Let's use the test/core directly.

[tlively]

This sounds like a good plan. Thanks for taking on this work. As far as running the tests, I suggest writing the spec tests in a PR to the WAVM repo first, then copying them into this repo (in test/core, as suggested above). That way we will not need to set up any separate executable CI for this repo.