[libc] `ninja check-libc`: AttributeError: module 'yaml' has no attribute 'safe_load'

[vinay-deshmukh]

I was following instructions on: https://libc.llvm.org/full_host_build.html#id3

(with slightly modified cmake command, i.e. without Sphinx):

cmake \
-B ~/build-folder \
-S runtimes \
   -G Ninja \
   -DCMAKE_C_COMPILER=clang \
   -DCMAKE_CXX_COMPILER=clang++ \
   -DLLVM_ENABLE_RUNTIMES="libc;compiler-rt" \
   -DLLVM_LIBC_FULL_BUILD=ON \
   -DCMAKE_BUILD_TYPE=Debug \
   -DLLVM_LIBC_INCLUDE_SCUDO=ON \
   -DCOMPILER_RT_BUILD_SCUDO_STANDALONE_WITH_LLVM_LIBC=ON \
   -DCOMPILER_RT_BUILD_GWP_ASAN=OFF                       \
   -DCOMPILER_RT_SCUDO_STANDALONE_BUILD_SHARED=OFF        \
   -DCMAKE_EXPORT_COMPILE_COMMANDS=ON \
   -DLIBC_CMAKE_VERBOSE_LOGGING=ON

And then:

ninja -C ~/build-folder check-libc

Which fails with:

Traceback (most recent call last):
  File "~/llvm-project/libc/hdrgen/yaml_to_classes.py", line 284, in <module>
    main()
  File "~/llvm-project/libc/hdrgen/yaml_to_classes.py", line 261, in main
    header = load_yaml_file(args.yaml_file, header_class, args.entry_points)
  File "~/llvm-project/libc/hdrgen/yaml_to_classes.py", line 123, in load_yaml_file
    yaml_data = yaml.safe_load(f)
AttributeError: module 'yaml' has no attribute 'safe_load'
ninja: build stopped: subcommand failed.

Near:

llvm-project/libc/hdrgen/yaml_to_classes.py

Lines 120 to 121 in 5f0db7c

	with open(yaml_file, "r") as f:
	yaml_data = yaml.safe_load(f)

When I add the following code, just before the yaml.safe_load,

        print(dir(yaml))
        print((yaml).__path__)

I get the output:

['__doc__', '__file__', '__loader__', '__name__', '__package__', '__path__', '__spec__']
_NamespacePath(['~/llvm-project/libc/hdrgen/yaml'])

Which probably means, that import yaml is trying to "import" the https://github.com/llvm/llvm-project/tree/main/libc/hdrgen/yaml directory instead of PyYaml

However, given that

Note: I do have PyYaml installed within a conda environment which is activated, so I don't think this is an issue with my python setup:

For instance, the following works as expected:

❯ /usr/bin/env python3
Python 3.13.1 | packaged by Anaconda, Inc. | (main, Dec 11 2024, 10:35:08) [Clang 14.0.6 ] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import yaml
>>> 'safe_load' in dir(yaml)
True

Given that the CI has been passing for months, it seems that I might be missing some sort of extra setup so that /usr/bin/env python3 looks at the right path to import yaml (which might need to be added to https://libc.llvm.org/full_host_build.html#configure-the-build-for-development)

Also looked at:
https://libc.llvm.org/dev/header_generation.html#common-errors, but this page didn't seem to mention my issue

[jhuber6]

https://libc.llvm.org/dev/header_generation.html#instructions install PyYaml to your system or in a virtual environment.

[vinay-deshmukh]

I do have that package installed as I mentioned in my earlier comment.

This is my original environment's packages:

Original environment

❯ conda activate llvm
❯ conda list
# packages in environment at /Users/vinay/miniconda3/envs/llvm:
#
# Name                    Version                   Build  Channel
bzip2                     1.0.8                h6c40b1e_6
ca-certificates           2024.11.26           hecd8cb5_0
expat                     2.6.4                h6d0c2b6_0
libcxx                    14.0.6               h9765a3e_0
libffi                    3.4.4                hecd8cb5_1
libmpdec                  4.0.0                h46256e1_0
ncurses                   6.4                  hcec6c5f_0
openssl                   3.0.15               h46256e1_0
pip                       24.2            py313hecd8cb5_0
python                    3.13.1          h1ec50cd_100_cp313
python_abi                3.13                    0_cp313
pyyaml                    6.0.2           py313h46256e1_0
readline                  8.2                  hca72f7f_0
setuptools                75.1.0          py313hecd8cb5_0
sqlite                    3.45.3               h6c40b1e_0
tk                        8.6.14               h4d00af3_0
tzdata                    2024b                h04d1e81_0
wheel                     0.44.0          py313hecd8cb5_0
xz                        5.4.6                h6c40b1e_1
yaml                      0.2.5                haf1e3a3_0
zlib                      1.2.13               h4b97444_1

And to be redundantly safe, here's the env I created after referring to the above link:

Very specific py3.8, pyyaml 5.1 env

❯ conda list
# packages in environment at /Users/vinay/miniconda3/envs/llpy:
#
# Name                    Version                   Build  Channel
ca-certificates           2024.11.26           hecd8cb5_0
libcxx                    14.0.6               h9765a3e_0
libffi                    3.4.4                hecd8cb5_1
ncurses                   6.4                  hcec6c5f_0
openssl                   3.0.15               h46256e1_0
pip                       24.2             py38hecd8cb5_0
python                    3.8.20               hce00570_0
pyyaml                    5.1.2            py38h1de35cc_0
readline                  8.2                  hca72f7f_0
setuptools                75.1.0           py38hecd8cb5_0
sqlite                    3.45.3               h6c40b1e_0
tk                        8.6.14               h4d00af3_0
wheel                     0.44.0           py38hecd8cb5_0
xz                        5.4.6                h6c40b1e_1
yaml                      0.1.7                hc338f04_2
zlib                      1.2.13               h4b97444_1

ninja check-hdrgen passes, but ninja libc still fails as earlier:

❯ ninja -C ~/build-folder check-hdrgen
ninja: Entering directory `/Users/vinay/build-folder'
[1/1] cd /Users/vinay/build-folder/libc/hdrgen && /opt/homebrew/bin/ctest -R hdrgen_integration_test --output-on-failure
Test project /Users/vinay/build-folder/libc/hdrgen
    Start 1: hdrgen_integration_test
1/1 Test #1: hdrgen_integration_test ..........   Passed    0.23 sec

100% tests passed, 0 tests failed out of 1

Total Test time (real) =   0.24 sec

and if I open an interpreter with usr/bin/env, my import yaml works as shown above, implying the package is already installed in my virtual environment.

[jhuber6]

I guess it is possible that the yaml/ directory is confusing pyyaml somehow? You could try renaming the folder and in the CMake.

[vinay-deshmukh]

@jhuber6 Thanks for the suggestion! I renamed the yaml/ folder to yaml_data/ and update the corresponding .cmake and ninja -C build libc is working.

Related: https://stackoverflow.com/a/9287306

I can raise a patch for the aforementioned change so such issues can be circumvented

[jhuber6]

Sure, @nickdesaulniers shall we rename it to libc/hdrgen/spec or something?

[vinay-deshmukh]

I've created #120931 for now as per the previous comment. If that looks good, we can go ahead or I can rename it as needed.

[vinay-deshmukh]

Fixed by #121443