Skip to content

Commit

Permalink
spectest-interp: Report when assert_trap passes and include error str…
Browse files Browse the repository at this point in the history 
…ing. NFC (WebAssembly#1235)
  • Loading branch information
@sbc100
sbc100 committed Nov 19, 2019
1 parent cdd59d1 commit b51f2cd
Show file tree
Hide file tree
Showing 33 changed files with 1,657 additions and 0 deletions.
2 changes: 2 additions & 0 deletions src/tools/spectest-interp.cc
View file
Expand Up @@ -1404,6 +1404,8 @@ wabt::Result CommandRunner::OnAssertTrapCommand(
return wabt::Result::Error;
}

PrintError(command->line, "assert_trap passed: %s",
ResultToString(exec_result.result));
return wabt::Result::Ok;
}

Expand Down
32 changes: 32 additions & 0 deletions test/spec/address.txt
View file
@@ -1,9 +1,41 @@
;;; TOOL: run-interp-spec
;;; STDIN_FILE: third_party/testsuite/address.wast
(;; STDOUT ;;;
out/test/spec/address.wast:192: assert_trap passed: out of bounds memory access
out/test/spec/address.wast:194: assert_trap passed: out of bounds memory access
out/test/spec/address.wast:195: assert_trap passed: out of bounds memory access
out/test/spec/address.wast:196: assert_trap passed: out of bounds memory access
out/test/spec/address.wast:197: assert_trap passed: out of bounds memory access
out/test/spec/address.wast:198: assert_trap passed: out of bounds memory access
out/test/spec/address.wast:200: assert_trap passed: out of bounds memory access
out/test/spec/address.wast:201: assert_trap passed: out of bounds memory access
out/test/spec/address.wast:202: assert_trap passed: out of bounds memory access
out/test/spec/address.wast:203: assert_trap passed: out of bounds memory access
out/test/spec/address.wast:204: assert_trap passed: out of bounds memory access
out/test/spec/address.wast:207: assert_malformed passed:
out/test/spec/address/address.1.wat:1:33: error: offset must be less than or equal to 0xffffffff
(memory 1)(func (drop (i32.load offset=4294967296 (i32.const 0))))
^^^^^^^^^^^^^^^^^
out/test/spec/address.wast:479: assert_trap passed: out of bounds memory access
out/test/spec/address.wast:481: assert_trap passed: out of bounds memory access
out/test/spec/address.wast:482: assert_trap passed: out of bounds memory access
out/test/spec/address.wast:483: assert_trap passed: out of bounds memory access
out/test/spec/address.wast:484: assert_trap passed: out of bounds memory access
out/test/spec/address.wast:485: assert_trap passed: out of bounds memory access
out/test/spec/address.wast:486: assert_trap passed: out of bounds memory access
out/test/spec/address.wast:487: assert_trap passed: out of bounds memory access
out/test/spec/address.wast:489: assert_trap passed: out of bounds memory access
out/test/spec/address.wast:490: assert_trap passed: out of bounds memory access
out/test/spec/address.wast:491: assert_trap passed: out of bounds memory access
out/test/spec/address.wast:492: assert_trap passed: out of bounds memory access
out/test/spec/address.wast:493: assert_trap passed: out of bounds memory access
out/test/spec/address.wast:494: assert_trap passed: out of bounds memory access
out/test/spec/address.wast:495: assert_trap passed: out of bounds memory access
out/test/spec/address.wast:539: assert_trap passed: out of bounds memory access
out/test/spec/address.wast:541: assert_trap passed: out of bounds memory access
out/test/spec/address.wast:542: assert_trap passed: out of bounds memory access
out/test/spec/address.wast:586: assert_trap passed: out of bounds memory access
out/test/spec/address.wast:588: assert_trap passed: out of bounds memory access
out/test/spec/address.wast:589: assert_trap passed: out of bounds memory access
239/239 tests passed.
;;; STDOUT ;;)
1 change: 1 addition & 0 deletions test/spec/align.txt
View file
Expand Up @@ -296,5 +296,6 @@ out/test/spec/align.wast:448: assert_invalid passed:
out/test/spec/align.wast:452: assert_invalid passed:
error: alignment must not be larger than natural alignment (8)
000002a: error: OnStoreExpr callback failed
out/test/spec/align.wast:864: assert_trap passed: out of bounds memory access
131/131 tests passed.
;;; STDOUT ;;)
24 changes: 24 additions & 0 deletions test/spec/bulk-memory-operations/bulk.txt
View file
Expand Up @@ -5,32 +5,56 @@
fill(i32:1, i32:255, i32:3) =>
fill(i32:0, i32:48042, i32:2) =>
fill(i32:0, i32:0, i32:65536) =>
out/test/spec/bulk-memory-operations/bulk.wast:43: assert_trap passed: out of bounds memory access
fill(i32:65536, i32:0, i32:0) =>
out/test/spec/bulk-memory-operations/bulk.wast:52: assert_trap passed: out of bounds memory access
copy(i32:10, i32:0, i32:4) =>
copy(i32:8, i32:10, i32:4) =>
copy(i32:10, i32:7, i32:6) =>
copy(i32:65280, i32:0, i32:256) =>
copy(i32:65024, i32:65280, i32:256) =>
out/test/spec/bulk-memory-operations/bulk.wast:104: assert_trap passed: out of bounds memory access
copy(i32:65536, i32:0, i32:0) =>
copy(i32:0, i32:65536, i32:0) =>
out/test/spec/bulk-memory-operations/bulk.wast:114: assert_trap passed: out of bounds memory access
out/test/spec/bulk-memory-operations/bulk.wast:116: assert_trap passed: out of bounds memory access
init(i32:0, i32:1, i32:2) =>
init(i32:65532, i32:0, i32:4) =>
out/test/spec/bulk-memory-operations/bulk.wast:144: assert_trap passed: out of bounds memory access
init(i32:65536, i32:0, i32:0) =>
init(i32:0, i32:4, i32:0) =>
out/test/spec/bulk-memory-operations/bulk.wast:154: assert_trap passed: out of bounds memory access
out/test/spec/bulk-memory-operations/bulk.wast:156: assert_trap passed: out of bounds memory access
init_passive() =>
drop_passive() =>
out/test/spec/bulk-memory-operations/bulk.wast:176: assert_trap passed: data segment dropped
out/test/spec/bulk-memory-operations/bulk.wast:177: assert_trap passed: data segment dropped
out/test/spec/bulk-memory-operations/bulk.wast:178: assert_trap passed: data segment dropped
out/test/spec/bulk-memory-operations/bulk.wast:179: assert_trap passed: data segment dropped
init(i32:0, i32:1, i32:2) =>
out/test/spec/bulk-memory-operations/bulk.wast:205: assert_trap passed: uninitialized table element
init(i32:1, i32:2, i32:2) =>
out/test/spec/bulk-memory-operations/bulk.wast:211: assert_trap passed: out of bounds table access
init(i32:3, i32:0, i32:0) =>
init(i32:0, i32:4, i32:0) =>
out/test/spec/bulk-memory-operations/bulk.wast:220: assert_trap passed: out of bounds table access
out/test/spec/bulk-memory-operations/bulk.wast:222: assert_trap passed: out of bounds table access
init_passive() =>
drop_passive() =>
out/test/spec/bulk-memory-operations/bulk.wast:244: assert_trap passed: element segment dropped
out/test/spec/bulk-memory-operations/bulk.wast:245: assert_trap passed: element segment dropped
out/test/spec/bulk-memory-operations/bulk.wast:246: assert_trap passed: element segment dropped
out/test/spec/bulk-memory-operations/bulk.wast:247: assert_trap passed: element segment dropped
copy(i32:3, i32:0, i32:3) =>
copy(i32:0, i32:1, i32:3) =>
copy(i32:2, i32:0, i32:3) =>
copy(i32:6, i32:8, i32:2) =>
copy(i32:8, i32:6, i32:2) =>
out/test/spec/bulk-memory-operations/bulk.wast:295: assert_trap passed: uninitialized table element
out/test/spec/bulk-memory-operations/bulk.wast:296: assert_trap passed: out of bounds table access
copy(i32:10, i32:0, i32:0) =>
copy(i32:0, i32:10, i32:0) =>
out/test/spec/bulk-memory-operations/bulk.wast:305: assert_trap passed: out of bounds table access
out/test/spec/bulk-memory-operations/bulk.wast:307: assert_trap passed: out of bounds table access
102/102 tests passed.
;;; STDOUT ;;)
17 changes: 17 additions & 0 deletions test/spec/bulk-memory-operations/linking.txt
View file
Expand Up @@ -14,11 +14,28 @@ out/test/spec/bulk-memory-operations/linking.wast:87: assert_unlinkable passed:
out/test/spec/bulk-memory-operations/linking.wast:91: assert_unlinkable passed:
error: mutability mismatch in imported global, expected immutable but got mutable.
0000016: error: OnImportGlobal callback failed
out/test/spec/bulk-memory-operations/linking.wast:136: assert_trap passed: uninitialized table element
out/test/spec/bulk-memory-operations/linking.wast:137: assert_trap passed: uninitialized table element
out/test/spec/bulk-memory-operations/linking.wast:139: assert_trap passed: uninitialized table element
out/test/spec/bulk-memory-operations/linking.wast:141: assert_trap passed: uninitialized table element
out/test/spec/bulk-memory-operations/linking.wast:142: assert_trap passed: uninitialized table element
out/test/spec/bulk-memory-operations/linking.wast:144: assert_trap passed: uninitialized table element
out/test/spec/bulk-memory-operations/linking.wast:146: assert_trap passed: undefined table index
out/test/spec/bulk-memory-operations/linking.wast:147: assert_trap passed: undefined table index
out/test/spec/bulk-memory-operations/linking.wast:148: assert_trap passed: undefined table index
out/test/spec/bulk-memory-operations/linking.wast:149: assert_trap passed: undefined table index
out/test/spec/bulk-memory-operations/linking.wast:152: assert_trap passed: indirect call signature mismatch
out/test/spec/bulk-memory-operations/linking.wast:184: assert_trap passed: uninitialized table element
out/test/spec/bulk-memory-operations/linking.wast:185: assert_trap passed: uninitialized table element
out/test/spec/bulk-memory-operations/linking.wast:187: assert_trap passed: uninitialized table element
out/test/spec/bulk-memory-operations/linking.wast:188: assert_trap passed: uninitialized table element
out/test/spec/bulk-memory-operations/linking.wast:190: assert_trap passed: undefined table index
out/test/spec/bulk-memory-operations/linking.wast:207: assert_unlinkable passed:
error: elem segment is out of bounds: [10, 11) >= max value 10
out/test/spec/bulk-memory-operations/linking.wast:216: assert_unlinkable passed:
error: unknown module field "mem"
0000027: error: OnImportMemory callback failed
out/test/spec/bulk-memory-operations/linking.wast:225: assert_trap passed: uninitialized table element
out/test/spec/bulk-memory-operations/linking.wast:230: assert_unlinkable passed:
error: elem segment is out of bounds: [12, 13) >= max value 10
out/test/spec/bulk-memory-operations/linking.wast:241: assert_unlinkable passed:
Expand Down
17 changes: 17 additions & 0 deletions test/spec/bulk-memory-operations/memory_copy.txt
View file
Expand Up @@ -10,6 +10,17 @@ test() =>
test() =>
test() =>
test() =>
out/test/spec/bulk-memory-operations/memory_copy.wast:349: assert_trap passed: out of bounds memory access
out/test/spec/bulk-memory-operations/memory_copy.wast:730: assert_trap passed: out of bounds memory access
out/test/spec/bulk-memory-operations/memory_copy.wast:1113: assert_trap passed: out of bounds memory access
out/test/spec/bulk-memory-operations/memory_copy.wast:1494: assert_trap passed: out of bounds memory access
out/test/spec/bulk-memory-operations/memory_copy.wast:1877: assert_trap passed: out of bounds memory access
out/test/spec/bulk-memory-operations/memory_copy.wast:2238: assert_trap passed: out of bounds memory access
out/test/spec/bulk-memory-operations/memory_copy.wast:2619: assert_trap passed: out of bounds memory access
out/test/spec/bulk-memory-operations/memory_copy.wast:2980: assert_trap passed: out of bounds memory access
out/test/spec/bulk-memory-operations/memory_copy.wast:3351: assert_trap passed: out of bounds memory access
out/test/spec/bulk-memory-operations/memory_copy.wast:3712: assert_trap passed: out of bounds memory access
out/test/spec/bulk-memory-operations/memory_copy.wast:4093: assert_trap passed: out of bounds memory access
out/test/spec/bulk-memory-operations/memory_copy.wast:4447: assert_invalid passed:
error: memory.copy requires an imported or defined memory.
0000030: error: OnMemoryCopyExpr callback failed
Expand Down Expand Up @@ -204,9 +215,15 @@ out/test/spec/bulk-memory-operations/memory_copy.wast:4887: assert_invalid passe
000004b: error: OnMemoryCopyExpr callback failed
test() =>
test() =>
out/test/spec/bulk-memory-operations/memory_copy.wast:4950: assert_trap passed: out of bounds memory access
out/test/spec/bulk-memory-operations/memory_copy.wast:4956: assert_trap passed: out of bounds memory access
out/test/spec/bulk-memory-operations/memory_copy.wast:4962: assert_trap passed: out of bounds memory access
out/test/spec/bulk-memory-operations/memory_copy.wast:4968: assert_trap passed: out of bounds memory access
test() =>
test() =>
out/test/spec/bulk-memory-operations/memory_copy.wast:5004: assert_trap passed: out of bounds memory access
test() =>
out/test/spec/bulk-memory-operations/memory_copy.wast:5016: assert_trap passed: out of bounds memory access
test() =>
test() =>
4548/4548 tests passed.
Expand Down
6 changes: 6 additions & 0 deletions test/spec/bulk-memory-operations/memory_fill.txt
View file
Expand Up @@ -3,8 +3,11 @@
;;; ARGS*: --enable-bulk-memory
(;; STDOUT ;;;
test() =>
out/test/spec/bulk-memory-operations/memory_fill.wast:43: assert_trap passed: out of bounds memory access
out/test/spec/bulk-memory-operations/memory_fill.wast:61: assert_trap passed: out of bounds memory access
test() =>
test() =>
out/test/spec/bulk-memory-operations/memory_fill.wast:117: assert_trap passed: out of bounds memory access
test() =>
test() =>
out/test/spec/bulk-memory-operations/memory_fill.wast:174: assert_invalid passed:
Expand Down Expand Up @@ -199,5 +202,8 @@ out/test/spec/bulk-memory-operations/memory_fill.wast:607: assert_invalid passed
out/test/spec/bulk-memory-operations/memory_fill.wast:614: assert_invalid passed:
error: type mismatch in memory.fill, expected [i32, i32, i32] but got [f64, f64, f64]
000004a: error: OnMemoryFillExpr callback failed
out/test/spec/bulk-memory-operations/memory_fill.wast:637: assert_trap passed: out of bounds memory access
out/test/spec/bulk-memory-operations/memory_fill.wast:661: assert_trap passed: out of bounds memory access
out/test/spec/bulk-memory-operations/memory_fill.wast:685: assert_trap passed: out of bounds memory access
92/92 tests passed.
;;; STDOUT ;;)
14 changes: 14 additions & 0 deletions test/spec/bulk-memory-operations/memory_init.txt
View file
Expand Up @@ -12,14 +12,22 @@ out/test/spec/bulk-memory-operations/memory_init.wast:189: assert_invalid passed
out/test/spec/bulk-memory-operations/memory_init.wast:195: assert_invalid passed:
error: invalid data_segment_index: 4 (max 1)
000002c: error: OnDataDropExpr callback failed
out/test/spec/bulk-memory-operations/memory_init.wast:208: assert_trap passed: data segment dropped
out/test/spec/bulk-memory-operations/memory_init.wast:216: assert_trap passed: data segment dropped
out/test/spec/bulk-memory-operations/memory_init.wast:223: assert_trap passed: data segment dropped
out/test/spec/bulk-memory-operations/memory_init.wast:226: assert_invalid passed:
error: memory.init requires an imported or defined memory.
000002f: error: OnMemoryInitExpr callback failed
out/test/spec/bulk-memory-operations/memory_init.wast:232: assert_invalid passed:
error: invalid data_segment_index: 1 (max 1)
0000034: error: OnMemoryInitExpr callback failed
test() =>
out/test/spec/bulk-memory-operations/memory_init.wast:252: assert_trap passed: out of bounds memory access
out/test/spec/bulk-memory-operations/memory_init.wast:259: assert_trap passed: out of bounds memory access
out/test/spec/bulk-memory-operations/memory_init.wast:266: assert_trap passed: out of bounds memory access
out/test/spec/bulk-memory-operations/memory_init.wast:273: assert_trap passed: out of bounds memory access
test() =>
out/test/spec/bulk-memory-operations/memory_init.wast:287: assert_trap passed: out of bounds memory access
test() =>
test() =>
out/test/spec/bulk-memory-operations/memory_init.wast:304: assert_invalid passed:
Expand Down Expand Up @@ -211,5 +219,11 @@ out/test/spec/bulk-memory-operations/memory_init.wast:792: assert_invalid passed
out/test/spec/bulk-memory-operations/memory_init.wast:800: assert_invalid passed:
error: type mismatch in memory.init, expected [i32, i32, i32] but got [f64, f64, f64]
0000048: error: OnMemoryInitExpr callback failed
out/test/spec/bulk-memory-operations/memory_init.wast:825: assert_trap passed: out of bounds memory access
out/test/spec/bulk-memory-operations/memory_init.wast:852: assert_trap passed: out of bounds memory access
out/test/spec/bulk-memory-operations/memory_init.wast:879: assert_trap passed: out of bounds memory access
out/test/spec/bulk-memory-operations/memory_init.wast:906: assert_trap passed: out of bounds memory access
out/test/spec/bulk-memory-operations/memory_init.wast:933: assert_trap passed: out of bounds memory access
out/test/spec/bulk-memory-operations/memory_init.wast:960: assert_trap passed: out of bounds memory access
227/227 tests passed.
;;; STDOUT ;;)

0 comments on commit b51f2cd

Please sign in to comment.