Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Some tests fail #1868

Closed
yurivict opened this issue Sep 13, 2020 · 13 comments
Closed

Some tests fail #1868

yurivict opened this issue Sep 13, 2020 · 13 comments

Comments

@yurivict
Copy link 

===>  Testing for ispc-1.14.1
Testing ispc: ../../../../../local/bin/ispc

Using test compiler: Intel(r) Implicit SPMD Program Compiler (Intel(r) ISPC), 1.14.1 (build commit  @ 20200912, LLVM 9.0.1)
Using C/C++ compiler: FreeBSD clang version 10.0.1 (git@github.com:llvm/llvm-project.git llvmorg-10.0.1-0-gef32c611aa2)

Running 8 jobs in parallel. Running 1430 tests.
Test ./tests/launch-1.ispc failed (return code -11)                     
Test ./tests/launch-2.ispc failed (return code -11)                     
Test ./tests/launch-3.ispc failed (return code -11)                     
Test ./tests/launch-4.ispc failed (return code -11)                     
Test ./tests/launch-5.ispc failed (return code -11)                     
Test ./tests/launch-6.ispc failed (return code -11)                     
Test ./tests/launch-7.ispc failed (return code -11)                     
Test ./tests/launch-8.ispc failed (return code -11)                     
Test ./tests/launch-9.ispc failed (return code -11)                     
Test bool: uniform: true, false; varying: [false,true,true,false], [true,false,false,true]
Test bool: uniform: true, false; varying: [false,true,true,false], [true,false,false,true]

Test bool: uniform: true, false; varying: [false,true,true,false], [true,false,false,true]; in simd cf: [_________,_________,true,false], [_________,_________,false,true].
Test bool: uniform: true, false; varying: [false,true,true,false], [true,false,false,true]; in simd cf: [_________,_________,true,false], [_________,_________,false,true].

Hello World!1430 [./tests/print_bool_simd.ispc]                         
Hello World!

Test uniform: small ones: -5, 5, -6, 6, -7, 7, 9.750000; big ones: -10000000005, 10000000005, 18.250000.
Test uniform: small ones: -5, 5, -6, 6, -7, 7, 9.750000; big ones: -10000000005, 10000000005, 18.250000.

Test varying: small ones: [-1,-2,-3,-4], [1,2,3,4], [-2,-3,-4,-5], [2,3,4,5], [-3,-4,-5,-6], [3,4,5,6], [5.750000,6.750000,7.750000,8.750000]; big ones: [-10000000001,-10000000002,-10000000003,-10000000004], [10000000001,10000000002,10000000003,10000000004], [14.250000,15.250000,16.250000,17.250000].
Test varying: small ones: [-1,-2,-3,-4], [1,2,3,4], [-2,-3,-4,-5], [2,3,4,5], [-3,-4,-5,-6], [3,4,5,6], [5.750000,6.750000,7.750000,8.750000]; big ones: [-10000000001,-10000000002,-10000000003,-10000000004], [10000000001,10000000002,10000000003,10000000004], [14.250000,15.250000,16.250000,17.250000].

Test varying simd: small ones: [((1)),2,((3)),4]
Test varying simd: small ones: [((1)),2,((3)),4]

Test ./tests/uniform-1.ispc failed (return code -11)                     
 Done 1430 / 1430 [./tests_errors/void-qualifiers-3.ispc]                
PASSRATE 99% 
1416 / 1430 tests PASSED
0 / 1430 tests FAILED compilation
10 / 1430 tests FAILED execution
	./tests/launch-1.ispc
	./tests/launch-2.ispc
	./tests/launch-3.ispc
	./tests/launch-4.ispc
	./tests/launch-5.ispc
	./tests/launch-6.ispc
	./tests/launch-7.ispc
	./tests/launch-8.ispc
	./tests/launch-9.ispc
	./tests/uniform-1.ispc
4 / 1430 tests SKIPPED
	./tests/gen-task-count.ispc
	./tests/gen-task-index-1.ispc
	./tests/gen-task-index.ispc
	./tests/transcendentals-10-0.ispc
NEW RUNFAILS:
	./tests/launch-1.ispc
	./tests/launch-2.ispc
	./tests/launch-3.ispc
	./tests/launch-4.ispc
	./tests/launch-5.ispc
	./tests/launch-6.ispc
	./tests/launch-7.ispc
	./tests/launch-8.ispc
	./tests/launch-9.ispc
	./tests/uniform-1.ispc
*** Error code 1

Stop.
make: stopped in /usr/ports/devel/ispc

Version 1.14.1
FreeBSD 12.2
clang-10
@dbabokin
Copy link
Collaborator

Have you also run make check-all?

@yurivict
Copy link
Author 

$ make test-check-all
[0/1] cd /usr/ports/devel/ispc/work/.build/tests && /usr/local/bin/python3.7 /usr/ports/devel/ispc/work/ispc-1.14.1/utils/lit/lit.py --path=/usr/ports/devel/ispc/work/.build/bin/. --path=/usr/local/llvm90/bin /usr/ports/devel/ispc/work/ispc-1.14.1/tests/lit-tests -Dispc_test_exec_root=/usr/ports/devel/ispc/work/.build/bin/./tests -Dispc_llvm_version_number=9.0.1 -Dx86_enabled=ON -Darm_enabled=ON -Dwasm_enabled=OFF -Dgenx_enabled=OFF -Dwindows_enabled=OFF
Config:
LLVM_10_0+: NO
WINDOWS_ENABLED: NO
X86_ENABLED: YES
ARM_ENABLED: YES
WASM_ENABLED: NO
GENX_ENABLED: NO
-- Testing: 73 tests, 8 threads --
PASS: ispc :: 1437.ispc (1 of 73)
FAIL: ispc :: 1447.ispc (2 of 73)
PASS: ispc :: 1319.ispc (3 of 73)
PASS: ispc :: 1419.ispc (4 of 73)
PASS: ispc :: 1210.ispc (5 of 73)
FAIL: ispc :: 1505.ispc (6 of 73)
PASS: ispc :: 1538.ispc (7 of 73)
PASS: ispc :: 1499.ispc (8 of 73)
PASS: ispc :: 1253.ispc (9 of 73)
PASS: ispc :: 1425.ispc (10 of 73)
PASS: ispc :: 1408.ispc (11 of 73)
PASS: ispc :: 1548.ispc (12 of 73)
PASS: ispc :: 1595.ispc (13 of 73)
FAIL: ispc :: 1596.ispc (14 of 73)
PASS: ispc :: 1576.ispc (15 of 73)
FAIL: ispc :: 1588_stdin.ispc (16 of 73)
PASS: ispc :: 1627.ispc (17 of 73)
PASS: ispc :: 1628.ispc (18 of 73)
FAIL: ispc :: 1654.ispc (19 of 73)
PASS: ispc :: 1608.ispc (20 of 73)
FAIL: ispc :: 1688.ispc (21 of 73)
PASS: ispc :: 1631.ispc (22 of 73)
PASS: ispc :: 1673.ispc (23 of 73)
PASS: ispc :: 1609.ispc (24 of 73)
PASS: ispc :: 1323.ispc (25 of 73)
PASS: ispc :: 1711.ispc (26 of 73)
FAIL: ispc :: 1739.ispc (27 of 73)
PASS: ispc :: 1717.ispc (28 of 73)
PASS: ispc :: 1469.ispc (29 of 73)
PASS: ispc :: 1719.ispc (30 of 73)
PASS: ispc :: 1729.ispc (31 of 73)
UNSUPPORTED: ispc :: 1771.ispc (32 of 73)
PASS: ispc :: 1738.ispc (33 of 73)
PASS: ispc :: 1763.ispc (34 of 73)
UNSUPPORTED: ispc :: 1844.ispc (35 of 73)
PASS: ispc :: 1747.ispc (36 of 73)
PASS: ispc :: 1762.ispc (37 of 73)
FAIL: ispc :: arg_parsing_errors.ispc (38 of 73)
FAIL: ispc :: 1767.ispc (39 of 73)
PASS: ispc :: 1788.ispc (40 of 73)
PASS: ispc :: 1806.ispc (41 of 73)
PASS: ispc :: add.ispc (42 of 73)
PASS: ispc :: bool_struct_member.ispc (43 of 73)
PASS: ispc :: exportVaryingPointer.ispc (44 of 73)
PASS: ispc :: avx512skx-registers.ispc (45 of 73)
UNSUPPORTED: ispc :: gather_test.ispc (46 of 73)
UNSUPPORTED: ispc :: genAddrSpace.ispc (47 of 73)
UNSUPPORTED: ispc :: genx_gather_coalescing.ispc (48 of 73)
PASS: ispc :: abs.ispc (49 of 73)
PASS: ispc :: 1620.ispc (50 of 73)
PASS: ispc :: noinline.ispc (51 of 73)
FAIL: ispc :: noinline_error.ispc (52 of 73)
PASS: ispc :: avx512skx-i32x8.ispc (53 of 73)
PASS: ispc :: print_bool.ispc (54 of 73)
PASS: ispc :: refBoolArgSelect.ispc (55 of 73)
PASS: ispc :: interleaved.ispc (56 of 73)
PASS: ispc :: gatherScaleConst.ispc (57 of 73)
FAIL: ispc :: varying_init_error.ispc (58 of 73)
UNSUPPORTED: ispc :: vecadd.ispc (59 of 73)
FAIL: ispc :: response_files.ispc (60 of 73)
UNSUPPORTED: ispc :: vectorcall-2.ispc (61 of 73)
UNSUPPORTED: ispc :: vectorcall-3.ispc (62 of 73)
UNSUPPORTED: ispc :: vectorcall-4.ispc (63 of 73)
UNSUPPORTED: ispc :: vectorcall-5.ispc (64 of 73)
UNSUPPORTED: ispc :: vectorcall-6.ispc (65 of 73)
PASS: ispc :: uint.ispc (66 of 73)
PASS: ispc :: int_min.ispc (67 of 73)
PASS: ispc :: x86_arch.ispc (68 of 73)
PASS: ispc :: vectorcall-1.ispc (69 of 73)
PASS: ispc :: deps.ispc (70 of 73)
PASS: ispc :: 1470.ispc (71 of 73)
PASS: ispc :: ps4_cpu.ispc (72 of 73)
PASS: ispc :: cpus_x86.ispc (73 of 73)
Testing Time: 26.20s
********************
Failing Tests (12):
    ispc :: 1447.ispc
    ispc :: 1505.ispc
    ispc :: 1588_stdin.ispc
    ispc :: 1596.ispc
    ispc :: 1654.ispc
    ispc :: 1688.ispc
    ispc :: 1739.ispc
    ispc :: 1767.ispc
    ispc :: arg_parsing_errors.ispc
    ispc :: noinline_error.ispc
    ispc :: response_files.ispc
    ispc :: varying_init_error.ispc

  Expected Passes    : 50
  Unsupported Tests  : 11
  Unexpected Failures: 12
FAILED: tests/CMakeFiles/check-all 
cd /usr/ports/devel/ispc/work/.build/tests && /usr/local/bin/python3.7 /usr/ports/devel/ispc/work/ispc-1.14.1/utils/lit/lit.py --path=/usr/ports/devel/ispc/work/.build/bin/. --path=/usr/local/llvm90/bin /usr/ports/devel/ispc/work/ispc-1.14.1/tests/lit-tests -Dispc_test_exec_root=/usr/ports/devel/ispc/work/.build/bin/./tests -Dispc_llvm_version_number=9.0.1 -Dx86_enabled=ON -Darm_enabled=ON -Dwasm_enabled=OFF -Dgenx_enabled=OFF -Dwindows_enabled=OFF
ninja: build stopped: subcommand failed.
*** Error code 1

@dbabokin
Copy link
Collaborator

It gives a clue how bad is using "stock" LLVM. All of these fails are bugs fixed in LLVM trunk and there are patches back-ported to LLVM 10 and in some cases to LLVM 9. Every test from these list is responsible for different problem.

It's probably better not to distribute ISPC, which fails at so many tests, than provide a broken ISPC to users.

@yurivict
Copy link
Author

It's probably better not to distribute ISPC, which fails at so many tests, than provide a broken ISPC to users.

It's even better to switch to the llvm-devel which is currently llvm-12.0.d20200811

@yurivict
Copy link
Author

ISPC doesn't build with the latest LLVM: #1870

@dbabokin
Copy link
Collaborator

Devel is not a good idea. LLVM API is constantly changing, we are tracking these changes and making the changes accordingly. Right now we build with LLVM trunk, but not building with past snapshots of trunk. Also, we are not fully validating with random trunk snapshot and they are know to be less stable.

@cloudhan
Copy link
Contributor

I think it might be better to include a private definition in patched llvm and when compile with stock llvm make a explicit compile error with explanation why it refuse to compile and link to a FAQ.

@dbabokin
Copy link
Collaborator

@cloudhan I thought about it, but distro maintainers extremely don't like the idea to use any LLVM builds other than their canonical one. I see their reasons, but it still doesn't work well for us. In the past we were requiring LLVM build with assertions and dumps enabled, so distro maintainers were hacking our code to disable this checks.

I probably end up adding what you suggested to be able to warn a user that the build is likely not passing all the tests and uses wrong LLVM build.

@yurivict
Copy link
Author

You should also clearly document in README what version or revision of LLVM a particular release works with.

@dbabokin
Copy link
Collaborator

We document LLVM version that we used for the specific release in ReleaseNotes. This is the best version to use (assuming all the patches are applied). We also maintain compatibility with other versions (for example for 1.14.1 we can build with LLVM 8.0-12.0), but this is mostly to keep compiler functional across multiple LLVMs to be able to analyze regressions. The range of "compatible versions" are defined here and you basically won't be able to build with completely unsupported version. But we don't validate ISPC against all the versions, just do basic testing.

Also, for every release we publish a Dockerfile, which reproduces Linux build. It contains LLVM version. For example, here's Dockerfile for v1.14.1 release.

@yurivict
Copy link
Author

If the build always assumes extra patches this means that ISPC isn't really compatible with these LLVM versions.

This development model would most likely cause broken ISPC packages on most systems.

@dbabokin
Copy link
Collaborator

That's why I don't like the idea of using "stock" LLVM build for ISPC build. One the bright side, most likely other packages, which depend on ISPC are mostly not affected.

@pbrubaker
Copy link
Collaborator

Closing as inactive. If this is still an issue please reopen.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@yurivict @cloudhan @dbabokin @pbrubaker