TEST llvm-cm X86/multi_func.s FAILED

[ronghongbo]

Hello, after building tflite, there is an error with llvm-cm:

(env) $                         ninja check-llvm-tools-llvm-cm
[0/1] Running llvm-cm tests
llvm-lit: /home/hrong1/llvm-src/llvm-project/llvm/utils/lit/lit/llvm/config.py:502: note: using yaml2obj: /home/hrong1/llvm-src/cmake-build/bin/yaml2obj
llvm-lit: /home/hrong1/llvm-src/llvm-project/llvm/utils/lit/lit/llvm/config.py:502: note: using llvm-cm: /home/hrong1/llvm-src/cmake-build/bin/llvm-cm
llvm-lit: /home/hrong1/llvm-src/llvm-project/llvm/utils/lit/lit/llvm/config.py:502: note: using split-file: /home/hrong1/llvm-src/cmake-build/bin/split-file
llvm-lit: /home/hrong1/llvm-src/llvm-project/llvm/utils/lit/lit/llvm/config.py:502: note: using llvm-mc: /home/hrong1/llvm-src/cmake-build/bin/llvm-mc
FAIL: llvm-cm :: X86/multi_func.s (11 of 11)
******************** TEST 'llvm-cm :: X86/multi_func.s' FAILED ********************
Exit Code: 1

Command Output (stdout):
--
# RUN: at line 2
/home/hrong1/llvm-src/cmake-build/bin/llvm-mc -o /home/hrong1/llvm-src/cmake-build/X86/Output/multi_func.s.tmp.o --filetype=obj -triple=x86_64-unknown-linux-gnu /home/hrong1/gematria/llvm_cm/test/X86/multi_func.s
# executed command: /home/hrong1/llvm-src/cmake-build/bin/llvm-mc -o /home/hrong1/llvm-src/cmake-build/X86/Output/multi_func.s.tmp.o --filetype=obj -triple=x86_64-unknown-linux-gnu /home/hrong1/gematria/llvm_cm/test/X86/multi_func.s
# RUN: at line 3
/home/hrong1/llvm-src/cmake-build/bin/llvm-cm /home/hrong1/llvm-src/cmake-build/X86/Output/multi_func.s.tmp.o -csv=/home/hrong1/gematria/llvm_cm/test/X86/Inputs/multi-func.csv -granite_model=/home/hrong1/gematria/llvm_cm/test/X86/Inputs/gb-token-mit-2022_12_02.tflite -evaluator=granite | /home/hrong1/llvm-src/cmake-build/bin/FileCheck /home/hrong1/gematria/llvm_cm/test/X86/multi_func.s
# executed command: /home/hrong1/llvm-src/cmake-build/bin/llvm-cm /home/hrong1/llvm-src/cmake-build/X86/Output/multi_func.s.tmp.o -csv=/home/hrong1/gematria/llvm_cm/test/X86/Inputs/multi-func.csv -granite_model=/home/hrong1/gematria/llvm_cm/test/X86/Inputs/gb-token-mit-2022_12_02.tflite -evaluator=granite
# .---command stderr------------
# | Unexpected node token: 'RIP'
# `-----------------------------
# executed command: /home/hrong1/llvm-src/cmake-build/bin/FileCheck /home/hrong1/gematria/llvm_cm/test/X86/multi_func.s
# .---command stderr------------
# | /home/hrong1/gematria/llvm_cm/test/X86/multi_func.s:8:15: error: CHECK-NEXT: expected string not found in input
# | # CHECK-NEXT: Calculated Frequency: 8.342712e+03
# |               ^
# | <stdin>:1:11: note: scanning from here
# | <reverse>:
# |           ^
# | <stdin>:2:1: note: possible intended match here
# | Calculated Frequency: 8.342695e+03
# | ^
# |
# | Input file: <stdin>
# | Check file: /home/hrong1/gematria/llvm_cm/test/X86/multi_func.s
# |
# | -dump-input=help explains the following input dump.
# |
# | Input was:
# | <<<<<<
# |           1: <reverse>:
# | next:8'0               X~ error: no match found
# |           2: Calculated Frequency: 8.342695e+03
# | next:8'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# | next:8'1     ?                                   possible intended match
# |           3: <tallestBillboard>:
# | next:8'0     ~~~~~~~~~~~~~~~~~~~~~
# |           4: Calculated Frequency: 2.928508e+05
# | next:8'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# |           5: <isMatch>:
# | next:8'0     ~~~~~~~~~~~~
# |           6: Calculated Frequency: 8.204262e+02
# | next:8'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# |           7: <bubbleSort>:
# | next:8'0     ~~~~~~~~~~~~~~~
# |           .
# |           .
# |           .
# | >>>>>>
# `-----------------------------
# error: command failed with exit status: 1

--

********************
********************
Failed Tests (1):
  llvm-cm :: X86/multi_func.s


Testing Time: 0.59s

Total Discovered Tests: 11
  Passed: 10 (90.91%)
  Failed:  1 (9.09%)
FAILED: tools/gematria/llvm_cm/CMakeFiles/check-llvm-tools-llvm-cm /home/hrong1/llvm-src/cmake-build/tools/gematria/llvm_cm/CMakeFiles/check-llvm-tools-llvm-cm
cd /home/hrong1/llvm-src/cmake-build/tools/gematria/llvm_cm && /home/hrong1/gematria/env/bin/python3 /home/hrong1/llvm-src/cmake-build/./bin/llvm-lit -sv /home/hrong1/llvm-src/cmake-build/tools/gematria/llvm_cm

[boomanaiden154]

Do you have more information on the LLVM version that you built with/your environment? Everything works fine for me with tip of tree LLVM.

[ronghongbo]

I pulled the latest LLVM project from https://github.com/llvm/llvm-project.git. Here is the head:

$ git log
commit 82f479ba315a417b6cd01a8c2efdc15c26689f2e (HEAD -> main, origin/main, origin/HEAD)
Author: Usama Hameed <u_hameed@apple.com>
Date:   Mon Apr 15 19:42:45 2024 -0700

    Add asan tests for libsanitizers. (#88349)

    This patch tests LLDB integration with libsanitizers for ASan.

    rdar://111856681

[boomanaiden154]

Looking at the log more, it seems like this is a floating point precision issue. For example, the value given for the reverse function in your case is 8.342695e+03 instead of the expected 8.342712e+03. Given that result requires the evaluation of the GRANITE model, some FP differences depending on the platform might be expected, although I'm not too familiar with what guarantees Tensorflow/TFLite makes.

That means the functionality is working fine, there are just some minor output differences. The llvm-cm tests need to be cleaned up at some point (probably rewritten in IR rather than assembly to make them significantly more maintainable), so I'll make sure to figure out a better floating point matching strategy when doing that.

[ronghongbo]

Good point! Thanks, Aiden.