is it expected that check.py reports errors and crashes?

[guoyejun]

Describe the bug
$DPCPP_HOME/llvm/buildbot/check.py reports errors and crashes, is it expected? thanks.

To Reproduce

docker run --name sycl_build --device=/dev/dri -it -v /home/gta/sycl_workspace:/src ghcr.io/intel/llvm/ubuntu2004_intel_drivers /bin/bash

export DPCPP_HOME=/src

python $DPCPP_HOME/llvm/buildbot/configure.py
python $DPCPP_HOME/llvm/buildbot/compile.py

python $DPCPP_HOME/llvm/buildbot/check.py > check.log 2>&1

root@69cca42c10a9:/src# cat check.log
[0/2] Re-checking globbed directories...
[1/3] Preparing lit tests
[2/3] cd /src/llvm/clang/bindings/python && /usr/bin/cmake -E env CLANG_LIBRARY_PATH=/src/llvm/build/lib /usr/bin/python3.8 -m unittest discover
..............................................................................................................................
----------------------------------------------------------------------
Ran 126 tests in 0.992s

OK
[2/3] Running all regression tests
-- Testing: 79213 tests, 96 workers --
Testing: llvm-lit: /src/llvm/llvm/utils/lit/lit/llvm/config.py:438: note: using clang: /src/llvm/build/bin/clang
llvm-lit: /src/llvm/llvm/utils/lit/lit/llvm/config.py:438: note: using clang: /src/llvm/build/bin/clang
 0.. 10.. 20.. 30.. 40.. 50.. 60.. 70
FAIL: Clang :: SemaSYCL/sycl-empty-header.cpp (15915 of 79213)
******************** TEST 'Clang :: SemaSYCL/sycl-empty-header.cpp' FAILED ********************
Script:
--
: 'RUN: at line 4';   mkdir -p /src/llvm/build/tools/clang/test/SemaSYCL/Output/sycl-empty-header.cpp.tmp_dir
: 'RUN: at line 5';   /src/llvm/build/bin/clang -cc1 -internal-isystem /src/llvm/build/lib/clang/15.0.0/include -nostdsysteminc -fsycl-is-device -fsycl-int-header=/src/llvm/build/tools/clang/test/SemaSYCL/Output/sycl-empty-header.cpp.tmp.h -save-temps=cwd /src/llvm/clang/test/SemaSYCL/sycl-empty-header.cpp
: 'RUN: at line 6';   ls /src/llvm/build/tools/clang/test/SemaSYCL/Output/sycl-empty-header.cpp.tmp.h
: 'RUN: at line 7';   rm -f /src/llvm/build/tools/clang/test/SemaSYCL/Output/sycl-empty-header.cpp.tmp.h
: 'RUN: at line 8';   touch /src/llvm/build/tools/clang/test/SemaSYCL/Output/sycl-empty-header.cpp.tmp.fail.h
: 'RUN: at line 9';   chmod 400 /src/llvm/build/tools/clang/test/SemaSYCL/Output/sycl-empty-header.cpp.tmp.fail.h
: 'RUN: at line 10';   /src/llvm/build/bin/clang -cc1 -internal-isystem /src/llvm/build/lib/clang/15.0.0/include -nostdsysteminc -fsycl-is-device -fsycl-int-header=/src/llvm/build/tools/clang/test/SemaSYCL/Output/sycl-empty-header.cpp.tmp.fail.h /src/llvm/clang/test/SemaSYCL/sycl-empty-header.cpp 2>&1 | /src/llvm/build/bin/FileCheck /src/llvm/clang/test/SemaSYCL/sycl-empty-header.cpp --check-prefix=SYCL-BADFILE
: 'RUN: at line 11';   rm -f /src/llvm/build/tools/clang/test/SemaSYCL/Output/sycl-empty-header.cpp.tmp.fail.h
--
Exit Code: 2

Command Output (stdout):
--
/src/llvm/build/tools/clang/test/SemaSYCL/Output/sycl-empty-header.cpp.tmp.h

--
Command Output (stderr):
--
FileCheck error: '<stdin>' is empty.
FileCheck command line:  /src/llvm/build/bin/FileCheck /src/llvm/clang/test/SemaSYCL/sycl-empty-header.cpp --check-prefix=SYCL-BADFILE

--

********************
Testing:  0.. 10.. 20.. 30.. 40.. 50.. 60.. 70
FAIL: LLVM :: tools/llvm-dwarfdump/X86/output.s (16902 of 79213)
******************** TEST 'LLVM :: tools/llvm-dwarfdump/X86/output.s' FAILED ********************
Script:
--
: 'RUN: at line 1';   rm -f /src/llvm/build/test/tools/llvm-dwarfdump/X86/Output/output.s.tmp1.txt /src/llvm/build/test/tools/llvm-dwarfdump/X86/Output/output.s.tmp2.txt /src/llvm/build/test/tools/llvm-dwarfdump/X86/Output/output.s.tmp3.txt
: 'RUN: at line 2';   /src/llvm/build/bin/llvm-mc /src/llvm/llvm/test/tools/llvm-dwarfdump/X86/brief.s -filetype obj -triple x86_64-apple-darwin -o /src/llvm/build/test/tools/llvm-dwarfdump/X86/Output/output.s.tmp.o
: 'RUN: at line 4';   /src/llvm/build/bin/llvm-dwarfdump -o=- /src/llvm/build/test/tools/llvm-dwarfdump/X86/Output/output.s.tmp.o | /src/llvm/build/bin/FileCheck /src/llvm/llvm/test/tools/llvm-dwarfdump/X86/output.s
: 'RUN: at line 6';   /src/llvm/build/bin/llvm-dwarfdump -o=/src/llvm/build/test/tools/llvm-dwarfdump/X86/Output/output.s.tmp1.txt /src/llvm/build/test/tools/llvm-dwarfdump/X86/Output/output.s.tmp.o
: 'RUN: at line 7';   /src/llvm/build/bin/FileCheck /src/llvm/llvm/test/tools/llvm-dwarfdump/X86/output.s --input-file /src/llvm/build/test/tools/llvm-dwarfdump/X86/Output/output.s.tmp1.txt
: 'RUN: at line 9';   touch /src/llvm/build/test/tools/llvm-dwarfdump/X86/Output/output.s.tmp2.txt
: 'RUN: at line 10';   /src/llvm/build/bin/llvm-dwarfdump -o=/src/llvm/build/test/tools/llvm-dwarfdump/X86/Output/output.s.tmp2.txt /src/llvm/build/test/tools/llvm-dwarfdump/X86/Output/output.s.tmp.o
: 'RUN: at line 11';   /src/llvm/build/bin/FileCheck /src/llvm/llvm/test/tools/llvm-dwarfdump/X86/output.s --input-file /src/llvm/build/test/tools/llvm-dwarfdump/X86/Output/output.s.tmp2.txt
: 'RUN: at line 13';   touch /src/llvm/build/test/tools/llvm-dwarfdump/X86/Output/output.s.tmp3.txt
: 'RUN: at line 14';   chmod 444 /src/llvm/build/test/tools/llvm-dwarfdump/X86/Output/output.s.tmp3.txt
: 'RUN: at line 15';   not /src/llvm/build/bin/llvm-dwarfdump -o=/src/llvm/build/test/tools/llvm-dwarfdump/X86/Output/output.s.tmp3.txt /src/llvm/build/test/tools/llvm-dwarfdump/X86/Output/output.s.tmp.o 2>&1 | /src/llvm/build/bin/FileCheck /src/llvm/llvm/test/tools/llvm-dwarfdump/X86/output.s  --check-prefix=ERROR1 -DFILE=/src/llvm/build/test/tools/llvm-dwarfdump/X86/Output/output.s.tmp3.txt -DMSG='Permission denied'
: 'RUN: at line 17';   not /src/llvm/build/bin/llvm-dwarfdump -o= /src/llvm/build/test/tools/llvm-dwarfdump/X86/Output/output.s.tmp.o 2>&1 | /src/llvm/build/bin/FileCheck /src/llvm/llvm/test/tools/llvm-dwarfdump/X86/output.s  --check-prefix=ERROR2 -DMSG='No such file or directory'
--
Exit Code: 2

Command Output (stderr):
--
/src/llvm/llvm/test/tools/llvm-dwarfdump/X86/brief.s:33:2: warning: .macosx_version_min used while targeting darwin
        .macosx_version_min 10, 12
        ^
FileCheck error: '<stdin>' is empty.
FileCheck command line:  /src/llvm/build/bin/FileCheck /src/llvm/llvm/test/tools/llvm-dwarfdump/X86/output.s --check-prefix=ERROR1 -DFILE=/src/llvm/build/test/tools/llvm-dwarfdump/X86/Output/output.s.tmp3.txt -DMSG=Permission denied

--

********************
Testing:  0.. 10.. 20.. 30.. 40.. 50.. 60.. 70.. 80
FAIL: LLVM :: tools/llvm-ifs/fail-file-write.test (30668 of 79213)
******************** TEST 'LLVM :: tools/llvm-ifs/fail-file-write.test' FAILED ********************
Script:
--
: 'RUN: at line 4';   rm -rf /src/llvm/build/test/tools/llvm-ifs/Output/fail-file-write.test.tmp.TestDir
: 'RUN: at line 5';   mkdir /src/llvm/build/test/tools/llvm-ifs/Output/fail-file-write.test.tmp.TestDir
: 'RUN: at line 6';   touch /src/llvm/build/test/tools/llvm-ifs/Output/fail-file-write.test.tmp.TestDir/Output.TestFile
: 'RUN: at line 7';   chmod 400 /src/llvm/build/test/tools/llvm-ifs/Output/fail-file-write.test.tmp.TestDir
: 'RUN: at line 8';   not /src/llvm/build/bin/llvm-ifs --output-elf=/src/llvm/build/test/tools/llvm-ifs/Output/fail-file-write.test.tmp.TestDir/Output.TestFile /src/llvm/llvm/test/tools/llvm-ifs/fail-file-write.test 2>&1 | /src/llvm/build/bin/FileCheck -DMSG='Permission denied' /src/llvm/llvm/test/tools/llvm-ifs/fail-file-write.test --check-prefix=ERR
: 'RUN: at line 9';   chmod 777 /src/llvm/build/test/tools/llvm-ifs/Output/fail-file-write.test.tmp.TestDir
: 'RUN: at line 10';   rm -rf /src/llvm/build/test/tools/llvm-ifs/Output/fail-file-write.test.tmp.TestDir
--
Exit Code: 2

Command Output (stderr):
--
FileCheck error: '<stdin>' is empty.
FileCheck command line:  /src/llvm/build/bin/FileCheck -DMSG=Permission denied /src/llvm/llvm/test/tools/llvm-ifs/fail-file-write.test --check-prefix=ERR

--

********************
Testing:  0.. 10.. 20.. 30.. 40.. 50.. 60.. 70.. 80.. 90.
FAIL: LLVM :: tools/llvm-ar/error-opening-permission.test (50943 of 79213)
******************** TEST 'LLVM :: tools/llvm-ar/error-opening-permission.test' FAILED ********************
Script:
--
: 'RUN: at line 5';   rm -rf /src/llvm/build/test/tools/llvm-ar/Output/error-opening-permission.test.tmp && mkdir -p /src/llvm/build/test/tools/llvm-ar/Output/error-opening-permission.test.tmp
: 'RUN: at line 6';   echo file1 > /src/llvm/build/test/tools/llvm-ar/Output/error-opening-permission.test.tmp/1.txt
: 'RUN: at line 9';   /src/llvm/build/bin/llvm-ar rc /src/llvm/build/test/tools/llvm-ar/Output/error-opening-permission.test.tmp/permission.b /src/llvm/build/test/tools/llvm-ar/Output/error-opening-permission.test.tmp/1.txt
: 'RUN: at line 10';   chmod 100 /src/llvm/build/test/tools/llvm-ar/Output/error-opening-permission.test.tmp/permission.b
: 'RUN: at line 11';   not /src/llvm/build/bin/llvm-ar p /src/llvm/build/test/tools/llvm-ar/Output/error-opening-permission.test.tmp/permission.b 2>&1 |    /src/llvm/build/bin/FileCheck /src/llvm/llvm/test/tools/llvm-ar/error-opening-permission.test --check-prefix=NO-PERMISSION -DARCHIVE=/src/llvm/build/test/tools/llvm-ar/Output/error-opening-permission.test.tmp/permission.b -DMSG='Permission denied'
--
Exit Code: 1

Command Output (stderr):
--
/src/llvm/llvm/test/tools/llvm-ar/error-opening-permission.test:14:18: error: NO-PERMISSION: expected string not found in input
# NO-PERMISSION: error: unable to open '[[ARCHIVE]]': [[MSG]]
                 ^
<stdin>:1:1: note: scanning from here
file1
^
<stdin>:1:1: note: with "ARCHIVE" equal to "/src/llvm/build/test/tools/llvm-ar/Output/error-opening-permission\\.test\\.tmp/permission\\.b"
file1
^
<stdin>:1:1: note: with "MSG" equal to "Permission denied"
file1
^

Input file: <stdin>
Check file: /src/llvm/llvm/test/tools/llvm-ar/error-opening-permission.test

-dump-input=help explains the following input dump.

Input was:
<<<<<<
            1: file1
check:14'0     X~~~~~ error: no match found
check:14'1            with "ARCHIVE" equal to "/src/llvm/build/test/tools/llvm-ar/Output/error-opening-permission\\.test\\.tmp/permission\\.b"
check:14'2            with "MSG" equal to "Permission denied"
>>>>>>

--

********************
Testing:  0.. 10.. 20.. 30.. 40.. 50.. 60.. 70.. 80.. 90..
********************
Failed Tests (4):
  Clang :: SemaSYCL/sycl-empty-header.cpp
  LLVM :: tools/llvm-ar/error-opening-permission.test
  LLVM :: tools/llvm-dwarfdump/X86/output.s
  LLVM :: tools/llvm-ifs/fail-file-write.test


Testing Time: 86.64s
  Skipped          :    64
  Unsupported      : 22650
  Passed           : 56407
  Expectedly Failed:    88
  Failed           :     4
FAILED: CMakeFiles/check-all
cd /src/llvm/build && /usr/bin/python3.8 /src/llvm/build/./bin/llvm-lit --verbose -sv --param USE_Z3_SOLVER=0 /src/llvm/build/tools/clang/test /src/llvm/build/tools/llvm-spirv/test /src/llvm/build/tools/libdevice/test /src/llvm/build/utils/lit /src/llvm/build/test
ninja: build stopped: subcommand failed.
args:Namespace(base_branch=None, branch=None, build_number=None, builder_dir=None, obj_dir=None, pr_number=None, src_dir=None, test_suite='check-all')
[Cmake Command]: cmake --build /src/llvm/build -- check-all -j 96
Traceback (most recent call last):
  File "/src/llvm/buildbot/check.py", line 64, in <module>
    ret = main()
  File "/src/llvm/buildbot/check.py", line 61, in main
    return do_check(args)
  File "/src/llvm/buildbot/check.py", line 38, in do_check
    subprocess.check_call(cmake_cmd, cwd=abs_obj_dir, env=env_tmp)
  File "/usr/lib/python3.8/subprocess.py", line 364, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['cmake', '--build', '/src/llvm/build', '--', 'check-all', '-j', '96']' returned non-zero exit status 1.

Environment (please complete the following information):

• OS: Linux
• Target device and vendor: Intel GPU
• DPC++ version: [e.g. commit hash or output of clang++ --version]

# git log --abbrev-commit -1
commit 7b2fb0272f69 (HEAD -> sycl, origin/sycl, origin/HEAD)
Author: Joe Todd <joeatodd@users.noreply.github.com>
Date:   Wed Apr 6 08:59:27 2022 +0100

[AlexeySachkov]

Hi @guoyejun, the expectation is that all LIT tests are always pass.

However, there could be cases where we introduce some errors or uncover some problems by testing in new environments. You can find the latest status of testing on a nightly build here: SYCL Nightly Builds. As you can see, they are mostly caused by newly added Ubuntu 22.04 environment (ignore opaque pointers section for now: that is still work in progress, failures are expected there)

[guoyejun]

thanks @AlexeySachkov, let me close it.