Description
Hi. I want to build with HWAsan and join c / rust / python code. I prepared example based on a public project https://github.com/pyca/cryptography
Steps to repro
- Clone cryptography
- Create
config.toml
- Create
Dockerfile
- Create
hello.py
- Build it
- Run
hello.py
Setup cryptography
git clone --recursive --branch=43.0.0 --depth=1 --single-branch git@github.com:pyca/cryptography.git
cd cryptography
config.toml
Create config.toml
with content:
[build]
target="aarch64-unknown-linux-gnu"
rustflags = [
"-g",
"-Z", "sanitizer=hwaddress",
"-Z", "external-clangrt",
"-L", "/usr/lib/llvm-20/lib/clang/20/lib/linux/",
"-l", "clang_rt.hwasan-aarch64",
"-C", "link-arg=-fuse-ld=/usr/bin/ld.lld-20",
"-C", "linker=/usr/bin/clang-20",
"-C", "lto=no"
]
Dockerfile
Create Dockerfile
with content:
FROM ubuntu:24.04 AS env
ENV DEBIAN_FRONTEND="noninteractive"
RUN apt-get update -y && \
apt-get install -y \
autoconf \
cmake \
curl \
gnupg \
libffi-dev \
libssl-dev \
lsb-release \
ninja-build \
patchelf \
pkg-config \
python3-dbg \
python3-dev \
python3-pip \
python3-venv \
software-properties-common \
wget
RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- --default-toolchain nightly -y
ENV PATH=/root/.cargo/bin:$PATH
RUN wget https://apt.llvm.org/llvm.sh && \
chmod +x llvm.sh && \
./llvm.sh 20 && \
ln -s /usr/bin/lld-20 /usr/local/bin/lld && \
ln -s /usr/bin/clang-20 /usr/local/bin/clang && \
ln -s /usr/bin/clang++-20 /usr/local/bin/clang++ && \
ln -s /usr/bin/clang-20 /usr/local/bin/cc && \
ln -s /usr/bin/clang++-20 /usr/local/bin/c++ && \
rm /usr/bin/ld && \
ln -s /usr/lib/llvm-20/bin/ld.lld /usr/bin/ld
RUN python3 -m venv /venv
RUN echo "/usr/lib/llvm-20/lib/clang/20/lib/linux" > /etc/ld.so.conf.d/clang.conf && ldconfig
ENV CC=/usr/bin/clang-20
ENV CXX=/usr/bin/clang++-20
ENV CFLAGS="-g -fsanitize=hwaddress -shared-libsan -mllvm -hwasan-globals=0 -std=c23"
ENV CCFLAGS="-g -fsanitize=hwaddress -shared-libsan -mllvm -hwasan-globals=0 -std=c23"
ENV CXXFLAGS="-g -fsanitize=hwaddress -shared-libsan -mllvm -hwasan-globals=0 -std=c++23"
# ENV CPPFLAGS="-g -fsanitize=hwaddress -shared-libsan -mllvm -hwasan-globals=0" | ?
ENV LDFLAGS="-fsanitize=hwaddress -shared-libsan"
ENV LDSHARED="/usr/bin/clang-20 -shared"
ENV RUSTFLAGS="-g -Zsanitizer=hwaddress -C linker=/usr/bin/clang-20 -C link-arg=-fuse-ld=/usr/bin/ld.lld-20 -C lto=no -Zexternal-clangrt -C target-feature=+tagged-globals"
ENV CARGO_BUILD_TARGET="aarch64-unknown-linux-gnu"
COPY config.toml /.cargo/
FROM env AS run
WORKDIR /src
COPY . .
Create hello.py
(from examples) with content:
from cryptography.fernet import Fernet
key = Fernet.generate_key()
f = Fernet(key)
message = b"A really secret message. Not for prying eyes."
private_data = f.encrypt(message)
public_data = f.decrypt(token)
print(f"Message: {message}")
print(f"Private data: {private_data}")
print(f"Public data: {public_data}")
Build
$ docker build . -q
$ docker run --rm -it <image>
$ source /venv/bin/activate
$ cargo build
$ pip3 install setuptools
Collecting setuptools
Downloading setuptools-73.0.1-py3-none-any.whl.metadata (6.6 kB)
Downloading setuptools-73.0.1-py3-none-any.whl (2.3 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.3/2.3 MB 1.6 MB/s eta 0:00:00
Installing collected packages: setuptools
Successfully installed setuptools-73.0.1
$ pip3 install --no-binary ":all:" cffi --no-clean -vv
---------------- CUT ----------------
ld: error: undefined symbol: __hwasan_init
>>> referenced by _configtest.c
>>> _configtest.o:(hwasan.module_ctor)
clang-20: error: linker command failed with exit code 1 (use -v to see invocation)
Note: will not use '__sync_synchronize()' in the C code
***** The above error message can be safely ignored.
---------------- CUT ----------------
ld: error: undefined symbol: __hwasan_init
>>> referenced by _configtest.c
>>> _configtest.o:(hwasan.module_ctor)
clang-20: error: linker command failed with exit code 1 (use -v to see invocation)
Note: will not use '__sync_synchronize()' in the C code
***** The above error message can be safely ignored.
---------------- CUT ----------------
ld: error: undefined symbol: __hwasan_init
>>> referenced by _configtest.c
>>> _configtest.o:(hwasan.module_ctor)
clang-20: error: linker command failed with exit code 1 (use -v to see invocation)
Note: will not use '__sync_synchronize()' in the C code
***** The above error message can be safely ignored.
---------------- CUT ----------------
building '_cffi_backend' extension
creating build/temp.linux-aarch64-cpython-312
creating build/temp.linux-aarch64-cpython-312/src
creating build/temp.linux-aarch64-cpython-312/src/c
/usr/bin/clang-20 -fno-strict-overflow -Wsign-compare -DNDEBUG -g -O2 -Wall -g -fsanitize=hwaddress -shared-libsan -mllvm -hwasan-globals=0 -std=c23 -fPIC -DFFI_BUILDING=1 -DUSE__THREAD -I/venv/include -I/usr/include/python3.12 -c src/c/_cffi_backend.c -o build/temp.linux-aarch64-cpython-312/src/c/_cffi_backend.o
src/c/_cffi_backend.c:4579:22: warning: 'Py_FileSystemDefaultEncoding' is deprecated [-Wdeprecated-declarations]
4579 | Py_FileSystemDefaultEncoding, &filename_or_null, &flags))
| ^
/usr/include/python3.12/fileobject.h:22:1: note: 'Py_FileSystemDefaultEncoding' has been explicitly marked deprecated here
22 | Py_DEPRECATED(3.12) PyAPI_DATA(const char *) Py_FileSystemDefaultEncoding;
| ^
/usr/include/python3.12/pyport.h:317:54: note: expanded from macro 'Py_DEPRECATED'
317 | #define Py_DEPRECATED(VERSION_UNUSED) __attribute__((__deprecated__))
| ^
In file included from src/c/_cffi_backend.c:8027:
In file included from src/c/cffi1_module.c:20:
src/c/call_python.c:211:5: warning: "no definition for read_barrier(), missing synchronization for multi-thread initialization in embedded mode" [-W#warnings]
211 | # warning "no definition for read_barrier(), missing synchronization for\
| ^
2 warnings generated.
/usr/bin/clang-20 -shared -fsanitize=hwaddress -shared-libsan -g -fsanitize=hwaddress -shared-libsan -mllvm -hwasan-globals=0 -std=c23 build/temp.linux-aarch64-cpython-312/src/c/_cffi_backend.o -L/usr/lib/aarch64-linux-gnu -lffi -o build/lib.linux-aarch64-cpython-312/_cffi_backend.cpython-312-aarch64-linux-gnu.so
clang-20: warning: argument unused during compilation: '-mllvm -hwasan-globals=0' [-Wunused-command-line-argument]
---------------- CUT ----------------
Created wheel for pycparser: filename=pycparser-2.22-py3-none-any.whl size=117552 sha256=d57055b6dddc795bb4eca6fc3754bb5ed521035680dd552d86560baed33ef091
Stored in directory: /root/.cache/pip/wheels/36/53/17/c0ae2e096d359a9a8faf47fd7ded8f4c878af41a3c66cb5199
Successfully built cffi pycparser
Installing collected packages: pycparser, cffi
---------------- CUT ----------------
Successfully installed cffi-1.17.0 pycparser-2.22
Removed build tracker: '/tmp/pip-build-tracker-62wvhedt'
Run with pip isolation (by default):
$ pip3 install -e . --no-binary ":all:" --no-clean -vv
Using pip 24.0 from /venv/lib/python3.12/site-packages/pip (python 3.12)
---------------- CUT ----------------
= note: ld.lld-20: error: undefined symbol: __hwasan_tls
>>> referenced by mod.rs:536 (/rustc/eff09483c67e6fc96c8098ba46dce476162754c5/library/core/src/ptr/mod.rs:536)
>>> /tmp/pip-install-85vkgny3/maturin_6b5877bea7064e2bbabad687f355ccbb/target/aarch64-unknown-linux-gnu/release/deps/maturin-38e63050942023b0.maturin.5af50ad9ae4ddb95-cgu.01.rcgu.o:(core::ptr::drop_in_place$LT$$LP$core..any..TypeId$C$alloc..boxed..Box$LT$dyn$u20$core..any..Any$u2b$core..marker..Sync$u2b$core..marker..Send$GT$$RP$$GT$::h0a3864e356d1b87e)
>>> referenced by function.rs:250 (/rustc/eff09483c67e6fc96c8098ba46dce476162754c5/library/core/src/ops/function.rs:250)
>>> /tmp/pip-install-85vkgny3/maturin_6b5877bea7064e2bbabad687f355ccbb/target/aarch64-unknown-linux-gnu/release/deps/maturin-38e63050942023b0.maturin.5af50ad9ae4ddb95-cgu.00.rcgu.o:(core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::haffe049bd93073b5)
>>> referenced by mod.rs:536 (/rustc/eff09483c67e6fc96c8098ba46dce476162754c5/library/core/src/ptr/mod.rs:536)
>>> /tmp/pip-install-85vkgny3/maturin_6b5877bea7064e2bbabad687f355ccbb/target/aarch64-unknown-linux-gnu/release/deps/maturin-38e63050942023b0.maturin.5af50ad9ae4ddb95-cgu.01.rcgu.o:(core::ptr::drop_in_place$LT$$LP$core..any..TypeId$C$alloc..boxed..Box$LT$dyn$u20$core..any..Any$u2b$core..marker..Sync$u2b$core..marker..Send$GT$$RP$$GT$::h0a3864e356d1b87e)
>>> referenced 39817 more times
ld.lld-20: error: undefined symbol: __hwasan_loadN
>>> referenced by function.rs:250 (/rustc/eff09483c67e6fc96c8098ba46dce476162754c5/library/core/src/ops/function.rs:250)
>>> /tmp/pip-install-85vkgny3/maturin_6b5877bea7064e2bbabad687f355ccbb/target/aarch64-unknown-linux-gnu/release/deps/maturin-38e63050942023b0.maturin.5af50ad9ae4ddb95-cgu.00.rcgu.o:(core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::haffe049bd93073b5)
>>> referenced by intrinsics.rs:3325 (/rustc/eff09483c67e6fc96c8098ba46dce476162754c5/library/core/src/intrinsics.rs:3325)
>>> /tmp/pip-install-85vkgny3/maturin_6b5877bea7064e2bbabad687f355ccbb/target/aarch64-unknown-linux-gnu/release/deps/maturin-38e63050942023b0.maturin.5af50ad9ae4ddb95-cgu.01.rcgu.o:(_$LT$hashbrown..raw..RawTable$LT$T$C$A$GT$$u20$as$u20$core..ops..drop..Drop$GT$::drop::h059dda3c88ea07a7)
>>> referenced by intrinsics.rs:3325 (/rustc/eff09483c67e6fc96c8098ba46dce476162754c5/library/core/src/intrinsics.rs:3325)
>>> /tmp/pip-install-85vkgny3/maturin_6b5877bea7064e2bbabad687f355ccbb/target/aarch64-unknown-linux-gnu/release/deps/maturin-38e63050942023b0.maturin.5af50ad9ae4ddb95-cgu.01.rcgu.o:(_$LT$hashbrown..raw..RawTable$LT$T$C$A$GT$$u20$as$u20$core..ops..drop..Drop$GT$::drop::h059dda3c88ea07a7)
>>> referenced 3842 more times
---------------- CUT ----------------
error: `cargo build --manifest-path Cargo.toml --message-format=json-render-diagnostics --target aarch64-unknown-linux-gnu --release -v --no-default-features --locked` failed with code 101
We can see that ENVs and /.cargo/config.toml
are ignored and a lot of HWasan errors showed. I dont know how to fix it correctly (it is problem one; possibly, question to python community).
Run without pip isolation (but it is unreal to do in real world application):
$ pip3 install -e . --no-binary ":all:" --no-clean -vv --no-build-isolation
---------------- CUT ----------------
Successfully installed cryptography-43.0.0
Removed build tracker: '/tmp/pip-build-tracker-bvpypdpj'
Run hello.py
$ python3 hello.py
Traceback (most recent call last):
File "/src/hello.py", line 1, in <module>
from cryptography.fernet import Fernet
File "/src/src/cryptography/fernet.py", line 14, in <module>
from cryptography.exceptions import InvalidSignature
File "/src/src/cryptography/exceptions.py", line 9, in <module>
from cryptography.hazmat.bindings._rust import exceptions as rust_exceptions
ImportError: /src/src/cryptography/hazmat/bindings/_rust.abi3.so: undefined symbol: __hwasan_tls
We can see that ENVs and /.cargo/config.toml
are not ignored, but __hwasan_tls
(it is problem #2).
Additional information
$ rustc --version --verbose
rustc 1.82.0-nightly (eff09483c 2024-08-22)
binary: rustc
commit-hash: eff09483c67e6fc96c8098ba46dce476162754c5
commit-date: 2024-08-22
host: aarch64-unknown-linux-gnu
release: 1.82.0-nightly
LLVM version: 19.1.0
$ clang --version
Ubuntu clang version 20.0.0 (++20240821083450+84fa7b438e1f-1~exp1~20240821203619.364)
Target: aarch64-unknown-linux-gnu
Thread model: posix
InstalledDir: /usr/lib/llvm-20/bin
Few things which helps me it other cases:
-
disable link-time-optimisation (otherwise link fails with R_AARCH64_ADR_PREL_PG_HI21 out of range)
-
use only
lld
as linker (otherwise link fails R_AARCH64_ADR_PREL_PG_HI21 out of range) -
use shared (otherwise ASan conflicts or link fails)
-
use external clang runtime (otherwise ASan conflicts)
-
do not use gcc (sometimes it timeouts)
-
pass target-feature=+tagged-globals (otherwise R_AARCH64_ADR_PREL_PG_HI21 out of range)
-
pass -mllvm -hwasan-globals=0 (otherwise FP crashes)
-
use global cargo config with environment variables (in case env-only, args do not passed to cargo/rustc)(pip isolation)