tests: stl-lit.py -q says AttributeError: 'tuple' object has no attribute 'isFailure'

[StephanTLavavej]

I tried running the tests with lit's documented -q option. Aside from discovering that it ignores -o testing_x64.log (which isn't what I wanted, but fine), it emits a Python error. Note that I was running this in a branch with test failures, so those are expected:

S:\GitHub\STL\out\build\x64>python tests\utils\stl-lit\stl-lit.py ..\..\..\tests\std -q -o testing_x64.log
********************
Failing Tests (36):
  std :: tests/Dev10_860410_bitset_ctors:26
  std :: tests/Dev10_860410_bitset_ctors:27
  std :: tests/Dev10_860410_bitset_ctors:28
  std :: tests/Dev11_0343056_pair_tuple_ctor_sfinae:28
  std :: tests/Dev11_0447546_facet_allocation:26
  std :: tests/Dev11_0447546_facet_allocation:27
  std :: tests/Dev11_0447546_facet_allocation:28
  std :: tests/Dev11_0494593_time_put_wchar_t:26
  std :: tests/Dev11_0494593_time_put_wchar_t:27
  std :: tests/Dev11_0494593_time_put_wchar_t:28
  std :: tests/Dev11_0835323_to_string:26
  std :: tests/Dev11_0835323_to_string:27
  std :: tests/Dev11_0835323_to_string:28
  std :: tests/Dev11_1066931_filesystem_rename_noop:26
  std :: tests/Dev11_1066931_filesystem_rename_noop:27
  std :: tests/Dev11_1066931_filesystem_rename_noop:28
  std :: tests/Dev11_1180290_filesystem_error_code:26
  std :: tests/Dev11_1180290_filesystem_error_code:27
  std :: tests/Dev11_1180290_filesystem_error_code:28
  std :: tests/P0218R1_filesystem:22
  std :: tests/P0218R1_filesystem:23
  std :: tests/P0218R1_filesystem:24
  std :: tests/P0433R2_deduction_guides:22
  std :: tests/P0433R2_deduction_guides:23
  std :: tests/P0433R2_deduction_guides:24
  std :: tests/VSO_0000000_nullptr_stream_out:27
  std :: tests/VSO_0000000_nullptr_stream_out:28
  std :: tests/VSO_0000000_path_stream_parameter:26
  std :: tests/VSO_0000000_path_stream_parameter:27
  std :: tests/VSO_0000000_path_stream_parameter:28
  std :: tests/VSO_0000000_regex_interface:26
  std :: tests/VSO_0000000_regex_interface:27
  std :: tests/VSO_0000000_regex_interface:28
  std :: tests/VSO_0121275_filesystem_canonical_should_handle_many_double_dots:24
  std :: tests/VSO_0121275_filesystem_canonical_should_handle_many_double_dots:25
  std :: tests/VSO_0121275_filesystem_canonical_should_handle_many_double_dots:26

Traceback (most recent call last):
  File "tests\utils\stl-lit\stl-lit.py", line 37, in <module>
    main(builtin_parameters)
  File "S:/GitHub/STL/llvm-project/llvm\utils\lit\lit\main.py", line 100, in main
    print_results(discovered_tests, elapsed, opts)
  File "S:/GitHub/STL/llvm-project/llvm\utils\lit\lit\main.py", line 292, in print_results
    print_summary(tests_by_code, opts.quiet, elapsed)
  File "S:/GitHub/STL/llvm-project/llvm\utils\lit\lit\main.py", line 315, in print_summary
    codes = [c for c in result_codes if not quiet or c.isFailure]
  File "S:/GitHub/STL/llvm-project/llvm\utils\lit\lit\main.py", line 315, in <listcomp>
    codes = [c for c in result_codes if not quiet or c.isFailure]
AttributeError: 'tuple' object has no attribute 'isFailure'

After looking at the code, I believe this is an upstream issue.

https://github.com/llvm/llvm-project/blob/a6be4d17e349f834e4d365f68e0435a1c4334a81/llvm/utils/lit/lit/main.py#L315
says:

codes = [c for c in result_codes if not quiet or c.isFailure]

result_codes appears to be what in C++ I would call an array of tuples (and the error message does refer to its element as a 'tuple' object):
https://github.com/llvm/llvm-project/blob/a6be4d17e349f834e4d365f68e0435a1c4334a81/llvm/utils/lit/lit/main.py#L268-L281

result_codes = [
    # Passes
    (lit.Test.EXCLUDED,    'Excluded Tests',      'Excluded'),
    (lit.Test.SKIPPED,     'Skipped Tests',       'Skipped'),
    (lit.Test.UNSUPPORTED, 'Unsupported Tests',   'Unsupported'),
    (lit.Test.PASS,        'Expected Passes',     ''),
    (lit.Test.FLAKYPASS,   'Passes With Retry',   ''),
    (lit.Test.XFAIL,       'Expected Failures',   'Expected Failing'),
    # Failures
    (lit.Test.UNRESOLVED,  'Unresolved Tests',    'Unresolved'),
    (lit.Test.TIMEOUT,     'Individual Timeouts', 'Timed Out'),
    (lit.Test.FAIL,        'Unexpected Failures', 'Failing'),
    (lit.Test.XPASS,       'Unexpected Passes',   'Unexpected Passing')
]

I believe that the bug is asking c.isFailure on the tuple c, instead of asking that of the tuple's first element - the ResultCode that provides isFailure:
https://github.com/llvm/llvm-project/blob/a6be4d17e349f834e4d365f68e0435a1c4334a81/llvm/utils/lit/lit/Test.py#L9-L40

[frederick-vs-ja]

Is this resolved by commit llvm/llvm-project@5c24411?

[StephanTLavavej]

@frederick-vs-ja Yep, this works in microsoft/STL main now! I observe that -q is respected, -o testing_x64.log is respected, and passing vs. failing tests behave as expected.