build with musl #57

GopherJ · 2019-12-02T09:57:42Z

Hello, I tried to compile some examples with rust-musl-builder but it seems to be not working.

first try

rust-musl-builder cargo build --example futures_consume --release

second try

rust-musl-builder cargo build --example futures_consume --release  --features "build_bindgen"

is there a workaround?

The text was updated successfully, but these errors were encountered:

fpagliughi · 2019-12-02T15:17:02Z

Oh. I have no idea. I've never tried this. Do you need to specify a --target or is it considered a native build inside the container? What sort of errors are you getting? Are they in the Rust crate or when compiling the C library?

GopherJ · 2019-12-02T15:46:49Z

Hello,

it's considered a native build inside the container.

error: failed to run custom build command for `paho-mqtt-sys v0.2.1 (/home/rust/src/paho-mqtt-sys)`
process didn't exit successfully: `/home/rust/src/target/release/build/paho-mqtt-sys-e87ed646c81d4087/build-script-build` (exit code: 101)
--- stdout
debug:Running the bundled build for Paho C
cargo:rerun-if-changed=build.rs
running: "cmake" "/home/rust/src/paho-mqtt-sys/paho.mqtt.c/" "-DPAHO_BUILD_STATIC=on" "-DPAHO_ENABLE_TESTING=off" "-DPAHO_WITH_SSL=on" "-DCMAKE_INSTALL_PREFIX=/home/rust/src/target/x86_64-unknown-linux-musl/release/build/paho-mqtt-sys-8d156c5b0c0125c7/out" "-DCMAKE_C_FLAGS= -ffunction-sections -fdata-sections -fPIC -m64 -static" "-DCMAKE_C_COMPILER=/usr/bin/musl-gcc" "-DCMAKE_CXX_FLAGS= -ffunction-sections -fdata-sections -fPIC -m64 -static" "-DCMAKE_CXX_COMPILER=/usr/bin/musl-g++" "-DCMAKE_BUILD_TYPE=Release"
-- The C compiler identification is GNU 5.4.0
-- Check for working C compiler: /usr/bin/musl-gcc
-- Check for working C compiler: /usr/bin/musl-gcc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- CMake version: 3.5.1
-- CMake system name: Linux
-- Timestamp is 2019-12-02T15:45:50Z
-- OpenSSL hints: 
-- OpenSSL headers found at /usr/local/musl/include
-- OpenSSL library found at /usr/local/musl/lib/libssl.a
-- OpenSSL Crypto library found at /usr/local/musl/lib/libcrypto.a
-- Configuring done
-- Generating done
-- Build files have been written to: /home/rust/src/target/x86_64-unknown-linux-musl/release/build/paho-mqtt-sys-8d156c5b0c0125c7/out/build
running: "cmake" "--build" "." "--target" "install" "--config" "Release" "--"
Scanning dependencies of target common_ssl_obj
Scanning dependencies of target common_obj
[  9%] Building C object src/CMakeFiles/common_obj.dir/MQTTProtocolClient.c.o
[  9%] Building C object src/CMakeFiles/common_obj.dir/MQTTPacket.c.o
[  9%] Building C object src/CMakeFiles/common_obj.dir/Clients.c.o
[  9%] Building C object src/CMakeFiles/common_obj.dir/MQTTPacketOut.c.o
[  9%] Building C object src/CMakeFiles/common_obj.dir/StackTrace.c.o
[  9%] Building C object src/CMakeFiles/common_obj.dir/utf-8.c.o
[ 10%] Building C object src/CMakeFiles/common_ssl_obj.dir/MQTTProtocolClient.c.o
[ 12%] Building C object src/CMakeFiles/common_obj.dir/Messages.c.o
[ 13%] Building C object src/CMakeFiles/common_obj.dir/Tree.c.o
[ 15%] Building C object src/CMakeFiles/common_obj.dir/Socket.c.o
[ 16%] Building C object src/CMakeFiles/common_obj.dir/Log.c.o
[ 18%] Building C object src/CMakeFiles/common_obj.dir/MQTTPersistence.c.o
[ 19%] Building C object src/CMakeFiles/common_obj.dir/Thread.c.o
[ 21%] Building C object src/CMakeFiles/common_obj.dir/MQTTProtocolOut.c.o
[ 22%] Building C object src/CMakeFiles/common_obj.dir/MQTTPersistenceDefault.c.o
[ 24%] Building C object src/CMakeFiles/common_ssl_obj.dir/Clients.c.o
[ 25%] Building C object src/CMakeFiles/common_obj.dir/SocketBuffer.c.o
[ 27%] Building C object src/CMakeFiles/common_obj.dir/Heap.c.o
[ 28%] Building C object src/CMakeFiles/common_obj.dir/LinkedList.c.o
[ 30%] Building C object src/CMakeFiles/common_ssl_obj.dir/utf-8.c.o
[ 31%] Building C object src/CMakeFiles/common_obj.dir/MQTTProperties.c.o
[ 33%] Building C object src/CMakeFiles/common_obj.dir/MQTTReasonCodes.c.o
[ 34%] Building C object src/CMakeFiles/common_obj.dir/Base64.c.o
[ 36%] Building C object src/CMakeFiles/common_obj.dir/SHA1.c.o
[ 37%] Building C object src/CMakeFiles/common_obj.dir/WebSocket.c.o
[ 39%] Building C object src/CMakeFiles/common_ssl_obj.dir/StackTrace.c.o
[ 40%] Building C object src/CMakeFiles/common_ssl_obj.dir/MQTTPacket.c.o
[ 42%] Building C object src/CMakeFiles/common_ssl_obj.dir/MQTTPacketOut.c.o
[ 43%] Building C object src/CMakeFiles/common_ssl_obj.dir/Messages.c.o
[ 45%] Building C object src/CMakeFiles/common_ssl_obj.dir/Tree.c.o
[ 46%] Building C object src/CMakeFiles/common_ssl_obj.dir/Socket.c.o
[ 48%] Building C object src/CMakeFiles/common_ssl_obj.dir/Log.c.o
[ 50%] Building C object src/CMakeFiles/common_ssl_obj.dir/MQTTPersistence.c.o
[ 51%] Building C object src/CMakeFiles/common_ssl_obj.dir/Thread.c.o
[ 53%] Building C object src/CMakeFiles/common_ssl_obj.dir/MQTTProtocolOut.c.o
[ 53%] Built target common_obj
[ 54%] Building C object src/CMakeFiles/common_ssl_obj.dir/MQTTPersistenceDefault.c.o
Scanning dependencies of target paho-mqtt3c-static
[ 57%] Building C object src/CMakeFiles/paho-mqtt3c-static.dir/MQTTClient.c.o
[ 57%] Building C object src/CMakeFiles/common_ssl_obj.dir/SocketBuffer.c.o
[ 59%] Building C object src/CMakeFiles/common_ssl_obj.dir/Heap.c.o
[ 60%] Building C object src/CMakeFiles/common_ssl_obj.dir/LinkedList.c.o
[ 62%] Building C object src/CMakeFiles/common_ssl_obj.dir/MQTTProperties.c.o
[ 63%] Building C object src/CMakeFiles/common_ssl_obj.dir/MQTTReasonCodes.c.o
[ 65%] Building C object src/CMakeFiles/common_ssl_obj.dir/Base64.c.o
Scanning dependencies of target paho-mqtt3a-static
[ 66%] Building C object src/CMakeFiles/paho-mqtt3a-static.dir/MQTTAsync.c.o
[ 68%] Building C object src/CMakeFiles/common_ssl_obj.dir/SHA1.c.o
[ 69%] Building C object src/CMakeFiles/common_ssl_obj.dir/WebSocket.c.o
Scanning dependencies of target paho-mqtt3a
[ 71%] Building C object src/CMakeFiles/paho-mqtt3a.dir/MQTTAsync.c.o
Scanning dependencies of target paho-mqtt3c
[ 72%] Building C object src/CMakeFiles/paho-mqtt3c.dir/MQTTClient.c.o
[ 72%] Built target common_ssl_obj
Scanning dependencies of target paho-mqtt3as-static
[ 74%] Building C object src/CMakeFiles/paho-mqtt3as-static.dir/MQTTAsync.c.o
[ 75%] Linking C static library libpaho-mqtt3c-static.a
[ 75%] Built target paho-mqtt3c-static
Scanning dependencies of target paho-mqtt3cs-static
[ 77%] Building C object src/CMakeFiles/paho-mqtt3cs-static.dir/MQTTClient.c.o
[ 78%] Linking C shared library libpaho-mqtt3c.so
src/CMakeFiles/paho-mqtt3c.dir/build.make:138: recipe for target 'src/libpaho-mqtt3c.so.1.3.1' failed
CMakeFiles/Makefile2:475: recipe for target 'src/CMakeFiles/paho-mqtt3c.dir/all' failed
[ 80%] Building C object src/CMakeFiles/paho-mqtt3cs-static.dir/SSLSocket.c.o
[ 81%] Building C object src/CMakeFiles/paho-mqtt3as-static.dir/SSLSocket.c.o
[ 83%] Linking C static library libpaho-mqtt3a-static.a
[ 84%] Linking C shared library libpaho-mqtt3a.so
src/CMakeFiles/paho-mqtt3a.dir/build.make:138: recipe for target 'src/libpaho-mqtt3a.so.1.3.1' failed
CMakeFiles/Makefile2:544: recipe for target 'src/CMakeFiles/paho-mqtt3a.dir/all' failed
[ 84%] Built target paho-mqtt3a-static
[ 86%] Linking C static library libpaho-mqtt3cs-static.a
[ 86%] Built target paho-mqtt3cs-static
[ 87%] Linking C static library libpaho-mqtt3as-static.a
[ 87%] Built target paho-mqtt3as-static
Makefile:149: recipe for target 'all' failed

--- stderr
CMake Warning:
  Manually-specified variables were not used by the project:

    CMAKE_CXX_COMPILER
    CMAKE_CXX_FLAGS


/home/rust/src/paho-mqtt-sys/paho.mqtt.c/src/StackTrace.c: In function 'StackTrace_entry':
/home/rust/src/paho-mqtt-sys/paho.mqtt.c/src/StackTrace.c:110:34: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast]
   Log_stackTrace(trace_level, 9, (int)my_thread->id, my_thread->current_depth, name, line, NULL);
                                  ^
/home/rust/src/paho-mqtt-sys/paho.mqtt.c/src/StackTrace.c: In function 'StackTrace_exit':
/home/rust/src/paho-mqtt-sys/paho.mqtt.c/src/StackTrace.c:134:36: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast]
    Log_stackTrace(trace_level, 10, (int)my_thread->id, my_thread->current_depth, name, line, NULL);
                                    ^
/home/rust/src/paho-mqtt-sys/paho.mqtt.c/src/StackTrace.c:136:36: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast]
    Log_stackTrace(trace_level, 11, (int)my_thread->id, my_thread->current_depth, name, line, (int*)rc);
                                    ^
/home/rust/src/paho-mqtt-sys/paho.mqtt.c/src/StackTrace.c: In function 'StackTrace_entry':
/home/rust/src/paho-mqtt-sys/paho.mqtt.c/src/StackTrace.c:110:34: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast]
   Log_stackTrace(trace_level, 9, (int)my_thread->id, my_thread->current_depth, name, line, NULL);
                                  ^
/home/rust/src/paho-mqtt-sys/paho.mqtt.c/src/StackTrace.c: In function 'StackTrace_exit':
/home/rust/src/paho-mqtt-sys/paho.mqtt.c/src/StackTrace.c:134:36: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast]
    Log_stackTrace(trace_level, 10, (int)my_thread->id, my_thread->current_depth, name, line, NULL);
                                    ^
/home/rust/src/paho-mqtt-sys/paho.mqtt.c/src/StackTrace.c:136:36: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast]
    Log_stackTrace(trace_level, 11, (int)my_thread->id, my_thread->current_depth, name, line, (int*)rc);
                                    ^
/usr/bin/ld: /usr/lib/x86_64-linux-musl/libc.a(__stack_chk_fail.o): relocation R_X86_64_32 against `__stack_chk_guard' can not be used when making a shared object; recompile with -fPIC
/usr/lib/x86_64-linux-musl/libc.a: error adding symbols: Bad value
collect2: error: ld returned 1 exit status
make[2]: *** [src/libpaho-mqtt3c.so.1.3.1] Error 1
make[1]: *** [src/CMakeFiles/paho-mqtt3c.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....
/usr/bin/ld: /usr/lib/x86_64-linux-musl/libc.a(__stack_chk_fail.o): relocation R_X86_64_32 against `__stack_chk_guard' can not be used when making a shared object; recompile with -fPIC
/usr/lib/x86_64-linux-musl/libc.a: error adding symbols: Bad value
collect2: error: ld returned 1 exit status
make[2]: *** [src/libpaho-mqtt3a.so.1.3.1] Error 1
make[1]: *** [src/CMakeFiles/paho-mqtt3a.dir/all] Error 2
make: *** [all] Error 2
thread 'main' panicked at '
command did not execute successfully, got: exit code: 2

build script failed, must exit now', /home/rust/.cargo/registry/src/github.com-1ecc6299db9ec823/cmake-0.1.42/src/lib.rs:861:5
note: Run with `RUST_BACKTRACE=1` environment variable to display a backtrace.

warning: build failed, waiting for other jobs to finish...
error: build failed

@fpagliughi

fpagliughi · 2019-12-03T18:24:04Z

Hmmm. It seems to be choking on the build of the shared C libraries. Does the C lib not specify "-fPIC" for building the shared libs?!?

We don't use or need the shared libraries for the Rust build, but there doesn't seem to be a way to turn off building them in the CMake.

I wonder if you can modify build.rs to force CMake to use the position-independent build using the cflag option. Maybe try to update build.rs at around line 156, like this:

        cmk_cfg
            .cflag("-fPIC")    // <- add this line
            .define("PAHO_BUILD_STATIC", "on")
            .define("PAHO_ENABLE_TESTING", "off")
            .define("PAHO_WITH_SSL", ssl);

If that works, let me know. Maybe it's worth adding to the crate for everyone? Although, if that works it would be better for us to send a PR to the C library.

fpagliughi · 2019-12-03T18:40:05Z

Another option would be to see if you can build and install the Paho C libs independently, then tell cargo not to compile them when building the Rust crate (turn off the "bundled" feature).

GopherJ · 2019-12-05T08:42:11Z

hello @fpagliughi , I tried to add the line, but still errors

GopherJ · 2019-12-05T08:47:01Z

but it seems that I get less errors:

--- stderr
/home/rust/src/paho-mqtt-sys/paho.mqtt.c/src/StackTrace.c: In function 'StackTrace_entry':
/home/rust/src/paho-mqtt-sys/paho.mqtt.c/src/StackTrace.c:110:34: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast]
   Log_stackTrace(trace_level, 9, (int)my_thread->id, my_thread->current_depth, name, line, NULL);
                                  ^
/home/rust/src/paho-mqtt-sys/paho.mqtt.c/src/StackTrace.c: In function 'StackTrace_exit':
/home/rust/src/paho-mqtt-sys/paho.mqtt.c/src/StackTrace.c:134:36: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast]
    Log_stackTrace(trace_level, 10, (int)my_thread->id, my_thread->current_depth, name, line, NULL);
                                    ^
/home/rust/src/paho-mqtt-sys/paho.mqtt.c/src/StackTrace.c:136:36: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast]
    Log_stackTrace(trace_level, 11, (int)my_thread->id, my_thread->current_depth, name, line, (int*)rc);
                                    ^
/home/rust/src/paho-mqtt-sys/paho.mqtt.c/src/StackTrace.c: In function 'StackTrace_entry':
/home/rust/src/paho-mqtt-sys/paho.mqtt.c/src/StackTrace.c:110:34: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast]
   Log_stackTrace(trace_level, 9, (int)my_thread->id, my_thread->current_depth, name, line, NULL);
                                  ^
/home/rust/src/paho-mqtt-sys/paho.mqtt.c/src/StackTrace.c: In function 'StackTrace_exit':
/home/rust/src/paho-mqtt-sys/paho.mqtt.c/src/StackTrace.c:134:36: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast]
    Log_stackTrace(trace_level, 10, (int)my_thread->id, my_thread->current_depth, name, line, NULL);
                                    ^
/home/rust/src/paho-mqtt-sys/paho.mqtt.c/src/StackTrace.c:136:36: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast]
    Log_stackTrace(trace_level, 11, (int)my_thread->id, my_thread->current_depth, name, line, (int*)rc);
                                    ^
/usr/bin/ld: /usr/lib/x86_64-linux-musl/libc.a(__stack_chk_fail.o): relocation R_X86_64_32 against `__stack_chk_guard' can not be used when making a shared object; recompile with -fPIC
/usr/lib/x86_64-linux-musl/libc.a: error adding symbols: Bad value
collect2: error: ld returned 1 exit status
make[2]: *** [src/libpaho-mqtt3cs.so.1.3.1] Error 1
make[1]: *** [src/CMakeFiles/paho-mqtt3cs.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....
/usr/bin/ld: /usr/lib/x86_64-linux-musl/libc.a(__stack_chk_fail.o): relocation R_X86_64_32 against `__stack_chk_guard' can not be used when making a shared object; recompile with -fPIC
/usr/lib/x86_64-linux-musl/libc.a: error adding symbols: Bad value
collect2: error: ld returned 1 exit status
make[2]: *** [src/libpaho-mqtt3c.so.1.3.1] Error 1
make[1]: *** [src/CMakeFiles/paho-mqtt3c.dir/all] Error 2
/usr/bin/ld: /usr/lib/x86_64-linux-musl/libc.a(__stack_chk_fail.o): relocation R_X86_64_32 against `__stack_chk_guard' can not be used when making a shared object; recompile with -fPIC
/usr/lib/x86_64-linux-musl/libc.a: error adding symbols: Bad value
collect2: error: ld returned 1 exit status
make[2]: *** [src/libpaho-mqtt3as.so.1.3.1] Error 1
make[1]: *** [src/CMakeFiles/paho-mqtt3as.dir/all] Error 2
/usr/bin/ld: /usr/lib/x86_64-linux-musl/libc.a(__stack_chk_fail.o): relocation R_X86_64_32 against `__stack_chk_guard' can not be used when making a shared object; recompile with -fPIC
/usr/lib/x86_64-linux-musl/libc.a: error adding symbols: Bad value
collect2: error: ld returned 1 exit status
make[2]: *** [src/libpaho-mqtt3a.so.1.3.1] Error 1
make[1]: *** [src/CMakeFiles/paho-mqtt3a.dir/all] Error 2
make: *** [all] Error 2
thread 'main' panicked at '
command did not execute successfully, got: exit code: 2

build script failed, must exit now', /home/rust/.cargo/registry/src/github.com-1ecc6299db9ec823/cmake-0.1.42/src/lib.rs:861:5
note: Run with `RUST_BACKTRACE=1` environment variable to display a backtrace.

fpagliughi · 2019-12-08T15:13:53Z

Oh, wait. Is it linking to a static MUSL library that won't relocate? Can you build the C library independently in your environment? If not, then the Rust wrapper certainly won't be able to build it either. If that's the case, then perhaps this is an issue to bring up in the Paho C repository.

rcastill · 2020-03-19T19:34:26Z

Hi @fpagliughi, I'm having a similiar problem. The error seems to be related to not being able to link dynamic libraries (using cross as compilation tool).

$ cross build --release --target aarch64-unknown-linux-musl
   Compiling paho-mqtt-sys v0.2.1 (https://github.com/eclipse/paho.mqtt.rust#f92e43a9)
error: failed to run custom build command for `paho-mqtt-sys v0.2.1 (https://github.com/eclipse/paho.mqtt.rust#f92e43a9)`

Caused by:
  process didn't exit successfully: `/target/release/build/paho-mqtt-sys-0db00f74924c5f85/build-script-build` (exit code: 101)
--- stdout
debug:Running the bundled build for Paho C
cargo:rerun-if-changed=build.rs
running: "cmake" "/cargo/git/checkouts/paho.mqtt.rust-44b07fd42c95bba6/f92e43a/paho-mqtt-sys/paho.mqtt.c/" "-DPAHO_BUILD_STATIC=on" "-DPAHO_ENABLE_TESTING=off" "-DPAHO_WITH_SSL=off" "-DCMAKE_INSTALL_PREFIX=/target/aarch64-unknown-linux-musl/release/build/paho-mqtt-sys-6485bf04d813526d/out" "-DCMAKE_C_FLAGS= -ffunction-sections -fdata-sections -fPIC -static" "-DCMAKE_C_COMPILER=/usr/local/bin/aarch64-linux-musl-gcc" "-DCMAKE_CXX_FLAGS= -ffunction-sections -fdata-sections -fPIC -static" "-DCMAKE_CXX_COMPILER=/usr/local/bin/aarch64-linux-musl-g++" "-DCMAKE_BUILD_TYPE=Release"
-- CMake version: 3.16.2
-- CMake system name: Linux
-- Timestamp is 2020-03-19T16:16:33Z
-- Configuring done
-- Generating done
-- Build files have been written to: /target/aarch64-unknown-linux-musl/release/build/paho-mqtt-sys-6485bf04d813526d/out/build
running: "cmake" "--build" "." "--target" "install" "--config" "Release" "--"
[ 68%] Built target common_obj
Scanning dependencies of target paho-mqtt3c
Scanning dependencies of target paho-mqtt3c-static
Scanning dependencies of target paho-mqtt3a-static
Scanning dependencies of target paho-mqtt3a
[ 78%] Building C object src/CMakeFiles/paho-mqtt3a-static.dir/MQTTAsync.c.o
[ 81%] Building C object src/CMakeFiles/paho-mqtt3c-static.dir/MQTTClient.c.o
[ 81%] Building C object src/CMakeFiles/paho-mqtt3a.dir/MQTTAsync.c.o
[ 81%] Building C object src/CMakeFiles/paho-mqtt3c.dir/MQTTClient.c.o
[ 87%] Linking C shared library libpaho-mqtt3c.so
[ 87%] Linking C static library libpaho-mqtt3c-static.a
[ 87%] Built target paho-mqtt3c-static
[ 90%] Linking C shared library libpaho-mqtt3a.so
[ 93%] Linking C static library libpaho-mqtt3a-static.a
[ 93%] Built target paho-mqtt3a-static
[ 93%] Built target paho-mqtt3c
[ 93%] Built target paho-mqtt3a
[ 96%] Linking C executable MQTTVersion
src/CMakeFiles/MQTTVersion.dir/build.make:85: recipe for target 'src/MQTTVersion' failed
CMakeFiles/Makefile2:133: recipe for target 'src/CMakeFiles/MQTTVersion.dir/all' failed
Makefile:151: recipe for target 'all' failed

--- stderr
/usr/local/bin/../lib/gcc/aarch64-linux-musl/6.4.0/../../../../aarch64-linux-musl/bin/ld: attempted static link of dynamic object `libpaho-mqtt3a.so.1.3.1'
collect2: error: ld returned 1 exit status
make[2]: *** [src/MQTTVersion] Error 1
make[1]: *** [src/CMakeFiles/MQTTVersion.dir/all] Error 2
make: *** [all] Error 2
thread 'main' panicked at '
command did not execute successfully, got: exit code: 2

build script failed, must exit now', /cargo/registry/src/github.com-1ecc6299db9ec823/cmake-0.1.41/src/lib.rs:835:5
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace.

We don't use or need the shared libraries for the Rust build, but there doesn't seem to be a way to turn off building them in the CMake.

Looking around I stumbled upon this PR on the C repo that allows to select only static building. Maybe it would be nice to add that to this repo?

fpagliughi · 2020-03-31T00:08:27Z

Yes, that's the plan. I'm trying to help get the new Paho C library tested and out the door, then I will update the Rust crate to use it, and finally get MQTT v5 support out the door!
...in addition to fixing a bunch of bugs and build issues.

GopherJ · 2020-04-30T15:24:55Z

@fpagliughi hello, is there a progress on this? This is becoming critical when we would like to use the release binary in different version of linux distributions. (some has correct version of openssl, some not)

fpagliughi · 2020-04-30T17:23:16Z

My last questions went unanswered:

Is it linking to a static MUSL library that won't relocate? Can you build the C library independently in your environment?

In the meantime, Paho Rust v0.7.x was released, using the static Paho C lib. Did that make a difference?

Looking through the Paho C issues, it appears that the solution for #63 could fix this as well. If you want to test it out and report back, that would help. If it fixes the problem, let me know and I can get it right in.

Basically, add the CMake flag PAHO_HIGH_PERFORMANCE to the C lib build in build.rs, like this:

cmk_cfg
    .define("PAHO_BUILD_SHARED", "off")
    .define("PAHO_BUILD_STATIC", "on")
    .define("PAHO_ENABLE_TESTING", "off")
    .define("PAHO_HIGH_PERFORMANCE", "on")    // <- Add this
    .define("PAHO_WITH_SSL", ssl);

GopherJ · 2020-04-30T19:37:44Z

@fpagliughi Thanks for yor quick reply and sorry that I was actually out of this topic during last months...

So I just tried the latest one: 0.7.1, add added:

    .define("PAHO_HIGH_PERFORMANCE", "on")    // <- Add this

in paho-mqtt-sys

Then I still ran into the same errors on my local machine and also in musl container...I suspect that something is missing in my environment,

Could you have a look at the follow error messages to see If you can help?

TARGET = Some("x86_64-unknown-linux-musl")
OPT_LEVEL = Some("3")
TARGET = Some("x86_64-unknown-linux-musl")
HOST = Some("x86_64-unknown-linux-gnu")
TARGET = Some("x86_64-unknown-linux-musl")
TARGET = Some("x86_64-unknown-linux-musl")
HOST = Some("x86_64-unknown-linux-gnu")
CC_x86_64-unknown-linux-musl = None
CC_x86_64_unknown_linux_musl = None
TARGET_CC = None
CC = None
HOST = Some("x86_64-unknown-linux-gnu")
CROSS_COMPILE = None
TARGET = Some("x86_64-unknown-linux-musl")
HOST = Some("x86_64-unknown-linux-gnu")
CFLAGS_x86_64-unknown-linux-musl = None
CFLAGS_x86_64_unknown_linux_musl = None
TARGET_CFLAGS = None
CFLAGS = None
DEBUG = Some("false")
running: "musl-gcc" "-O3" "-ffunction-sections" "-fdata-sections" "-fPIC" "-m64" "-static" "-I" "/usr/local/musl/include/" "-Wall" "-Wextra" "-o" "/home/rust/src/alert_collector/target/x86_64-unknown-linux-musl/release/build/openssl-sys-extras-ac53a0da05b2da63/out/src/openssl_shim.o" "-c" "src/openssl_shim.c"
cargo:warning=src/openssl_shim.c: In function 'SSL_CTX_set_ecdh_auto_shim':
cargo:warning=src/openssl_shim.c:121:41: warning: unused parameter 'ctx' [-Wunused-parameter]
cargo:warning= int SSL_CTX_set_ecdh_auto_shim(SSL_CTX *ctx, int onoff) {
cargo:warning=                                         ^~~
cargo:warning=src/openssl_shim.c: In function 'DH_new_from_params':
cargo:warning=src/openssl_shim.c:132:7: error: dereferencing pointer to incomplete type 'DH {aka struct dh_st}'
cargo:warning=     dh->p = p;
cargo:warning=       ^~
cargo:warning=src/openssl_shim.c: In function 'X509_get_extensions_shim':
cargo:warning=src/openssl_shim.c:143:13: error: dereferencing pointer to incomplete type 'X509 {aka struct x509_st}'
cargo:warning=     return x->cert_info ? x->cert_info->extensions : NULL;
cargo:warning=             ^~
cargo:warning=src/openssl_shim.c:144:1: warning: control reaches end of non-void function [-Wreturn-type]
cargo:warning= }
cargo:warning= ^
exit code: 1

--- stderr
thread 'main' panicked at '

Internal error occurred: Command "musl-gcc" "-O3" "-ffunction-sections" "-fdata-sections" "-fPIC" "-m64" "-static" "-I" "/usr/local/musl/include/" "-Wall" "-Wextra" "-o" "/home/rust/src/alert_collector/target/x86_64-unknown-linux-musl/release/build/openssl-sys-extras-ac53a0da05b2da63/out/src/openssl_shim.o" "-c" "src/openssl_shim.c" with args "musl-gcc" did not execute successfully (status code exit code: 1).

', /home/rust/.cargo/registry/src/github.com-1ecc6299db9ec823/gcc-0.3.55/src/lib.rs:1672:5
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

warning: build failed, waiting for other jobs to finish...
error: failed to run custom build command for `openssl v0.7.14`

Caused by:
  process didn't exit successfully: `/home/rust/src/alert_collector/target/release/build/openssl-3726d8fa975c5572/build-script-build` (exit code: 101)
--- stdout
TARGET = Some("x86_64-unknown-linux-musl")
OPT_LEVEL = Some("3")
TARGET = Some("x86_64-unknown-linux-musl")
HOST = Some("x86_64-unknown-linux-gnu")
TARGET = Some("x86_64-unknown-linux-musl")
TARGET = Some("x86_64-unknown-linux-musl")
HOST = Some("x86_64-unknown-linux-gnu")
CC_x86_64-unknown-linux-musl = None
CC_x86_64_unknown_linux_musl = None
TARGET_CC = None
CC = None
HOST = Some("x86_64-unknown-linux-gnu")
CROSS_COMPILE = None
TARGET = Some("x86_64-unknown-linux-musl")
HOST = Some("x86_64-unknown-linux-gnu")
CFLAGS_x86_64-unknown-linux-musl = None
CFLAGS_x86_64_unknown_linux_musl = None
TARGET_CFLAGS = None
CFLAGS = None
DEBUG = Some("false")
running: "musl-gcc" "-O3" "-ffunction-sections" "-fdata-sections" "-fPIC" "-m64" "-static" "-I" "/usr/local/musl/include/" "-Wall" "-Wextra" "-o" "/home/rust/src/alert_collector/target/x86_64-unknown-linux-musl/release/build/openssl-8a8fb78b4ecf7e44/out/src/c_helpers.o" "-c" "src/c_helpers.c"
cargo:warning=src/c_helpers.c: In function 'rust_SSL_clone':
cargo:warning=src/c_helpers.c:4:5: warning: implicit declaration of function 'CRYPTO_add'; did you mean 'CRYPTO_free'? [-Wimplicit-function-declaration]
cargo:warning=     CRYPTO_add(&ssl->references, 1, CRYPTO_LOCK_SSL);
cargo:warning=     ^~~~~~~~~~
cargo:warning=     CRYPTO_free
cargo:warning=src/c_helpers.c:4:20: error: dereferencing pointer to incomplete type 'SSL {aka struct ssl_st}'
cargo:warning=     CRYPTO_add(&ssl->references, 1, CRYPTO_LOCK_SSL);
cargo:warning=                    ^~
cargo:warning=src/c_helpers.c:4:37: error: 'CRYPTO_LOCK_SSL' undeclared (first use in this function); did you mean 'CRYPTO_LOCK'?
cargo:warning=     CRYPTO_add(&ssl->references, 1, CRYPTO_LOCK_SSL);
cargo:warning=                                     ^~~~~~~~~~~~~~~
cargo:warning=                                     CRYPTO_LOCK
cargo:warning=src/c_helpers.c:4:37: note: each undeclared identifier is reported only once for each function it appears in
cargo:warning=src/c_helpers.c: In function 'rust_SSL_CTX_clone':
cargo:warning=src/c_helpers.c:8:20: error: dereferencing pointer to incomplete type 'SSL_CTX {aka struct ssl_ctx_st}'
cargo:warning=     CRYPTO_add(&ctx->references,1,CRYPTO_LOCK_SSL_CTX);
cargo:warning=                    ^~
cargo:warning=src/c_helpers.c:8:35: error: 'CRYPTO_LOCK_SSL_CTX' undeclared (first use in this function); did you mean 'CRYPTO_EX_INDEX_SSL_CTX'?
cargo:warning=     CRYPTO_add(&ctx->references,1,CRYPTO_LOCK_SSL_CTX);
cargo:warning=                                   ^~~~~~~~~~~~~~~~~~~
cargo:warning=                                   CRYPTO_EX_INDEX_SSL_CTX
cargo:warning=src/c_helpers.c: In function 'rust_X509_clone':
cargo:warning=src/c_helpers.c:12:21: error: dereferencing pointer to incomplete type 'X509 {aka struct x509_st}'
cargo:warning=     CRYPTO_add(&x509->references,1,CRYPTO_LOCK_X509);
cargo:warning=                     ^~
cargo:warning=src/c_helpers.c:12:36: error: 'CRYPTO_LOCK_X509' undeclared (first use in this function); did you mean 'CRYPTO_LOCK'?
cargo:warning=     CRYPTO_add(&x509->references,1,CRYPTO_LOCK_X509);
cargo:warning=                                    ^~~~~~~~~~~~~~~~
cargo:warning=                                    CRYPTO_LOCK
exit code: 1

--- stderr
thread 'main' panicked at '

Internal error occurred: Command "musl-gcc" "-O3" "-ffunction-sections" "-fdata-sections" "-fPIC" "-m64" "-static" "-I" "/usr/local/musl/include/" "-Wall" "-Wextra" "-o" "/home/rust/src/alert_collector/target/x86_64-unknown-linux-musl/release/build/openssl-8a8fb78b4ecf7e44/out/src/c_helpers.o" "-c" "src/c_helpers.c" with args "musl-gcc" did not execute successfully (status code exit code: 1).

', /home/rust/.cargo/registry/src/github.com-1ecc6299db9ec823/gcc-0.3.55/src/lib.rs:1672:5
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

warning: build failed, waiting for other jobs to finish...
error: build failed

GopherJ · 2020-04-30T19:49:07Z

To reproduce:

step 1:

git clone https://github.com/eclipse/paho.mqtt.rust

step 2:

cd paho.mqtt.rust/paho-mqtt-sys
vim build.rs

// add the following line
//     .define("PAHO_HIGH_PERFORMANCE", "on")    // <- Add this

step 3:

cd ../

docker run --rm  \
    -v "$(pwd)":/home/rust/src \
    -v cargo-git:/home/rust/.cargo/git \
    -v cargo-registry:/home/rust/.cargo/registry \
    -v "$(pwd)/target":/home/rust/src/target \
    ekidd/rust-musl-builder:1.43.0 \
    bash -c 'sudo chown -R rust:rust /home/rust/.cargo/git /home/rust/.cargo/registry /home/rust/src/target && cargo build --release --manifest-path=Cargo.toml'

GopherJ · 2020-04-30T20:21:01Z

This error is something similar to :

/paho.mqtt.rust/paho-mqtt-sys/paho.mqtt.c/src/SSLSocket.h:29:10: fatal error: openssl/ssl.h: No such file or directory
 #include <openssl/ssl.h>
          ^~~~~~~~~~~~~~~
compilation terminated.
make[2]: *** [src/CMakeFiles/common_ssl_obj_static.dir/MQTTProtocolClient.c.o] Error 1
make[1]: *** [src/CMakeFiles/common_ssl_obj_static.dir/all] Error 2
make: *** [all] Error 2
thread 'main' panicked at '
command did not execute successfully, got: exit code: 2

fpagliughi · 2020-04-30T21:28:35Z

Well it would be interesting to first see if this can all be built without SSL first. If that works, it would be a good sign, and then we can move on to figure out what can be done about OpenSSL

GopherJ · 2020-04-30T22:00:31Z

@fpagliughi I tried to compile with --no-default-features flag but still got errors.

➜  paho.mqtt.rust git:(master) ✗ docker run --rm \
    -v "$(pwd)":/home/rust/src \
    -v cargo-git:/home/rust/.cargo/git \
    -v cargo-registry:/home/rust/.cargo/registry \
    -v "$(pwd)/target":/home/rust/src/target \
    ekidd/rust-musl-builder:1.42.0-openssl10 \
    bash -c 'sudo chown -R rust:rust /home/rust/.cargo/git /home/rust/.cargo/registry /home/rust/src/target && cargo build --no-default-features --release --manifest-path=Cargo.toml'

   Compiling paho-mqtt-sys v0.3.0 (/home/rust/src/paho-mqtt-sys)
   Compiling num-derive v0.3.0
error: failed to run custom build command for `paho-mqtt-sys v0.3.0 (/home/rust/src/paho-mqtt-sys)`

Caused by:
  process didn't exit successfully: `/home/rust/src/target/release/build/paho-mqtt-sys-15c730ecfa8c52ea/build-script-build` (exit code: 101)
--- stdout
debug:Running the un-bundled build for Paho C
cargo:rerun-if-changed=paho.mqtt.c/
debug:link Using non-SSL library
cargo:rerun-if-env-changed=PAHO_MQTT_C_DIR
cargo:rerun-if-env-changed=PAHO_MQTT_C_INCLUDE_DIR
cargo:rerun-if-env-changed=PAHO_MQTT_C_LIB_DIR
cargo:rustc-link-lib=paho-mqtt3a
debug:Target: x86_64-unknown-linux-musl

--- stderr
thread 'main' panicked at 'No generated bindings exist for the version/target: bindings/bindings_paho_mqtt_c_1.3.2-x86_64-unknown-linux-musl.rs', paho-mqtt-sys/build.rs:101:13
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

warning: build failed, waiting for other jobs to finish...
error: build failed

fpagliughi · 2020-05-27T04:16:18Z

Sorry this is taking so long, but seems to be coming together. I got the non-SSL version working on Mint 19 (Ubuntu 18.04) using the code that's in the develop branch, and a minor tweak to the CMake file in the C lib.

I'm hoping that #72/#74 will get the SSL version working as well.

The result is this...

$ ./target/x86_64-unknown-linux-musl/release/examples/async_publish
Publishing a message on the 'test' topic

$ ldd target/x86_64-unknown-linux-musl/release/examples/async_publish
	not a dynamic executable

So the example apps run as fully-static executables.

The underlying C library is now built statically by default. The dynamic C lib is not built at all. The new C release 1.3.2 allows us to do this.

But, for some reason, v1.3.2 specified C++ support in the CMake file, which has since been removed. For this, I commented out the line in the bundled C lib CMakeLists.txt:34:

#ENABLE_LANGUAGE(CXX)

I added the Rust musl target for my platform:

$ rustup target add x86_64-unknown-linux-musl

I added the musl C compiler to my system:

$ sudo apt install musl-tools

$ musl-gcc --version
cc (Ubuntu 9.3.0-11ubuntu0~18.04.1) 9.3.0
Copyright (C) 2019 Free Software Foundation, Inc.

Then I compiled the Rust crate and examples for the release build without SSL, using the modified, bundled C lib:

$ cargo build --no-default-features --features="bundled" --release --target=x86_64-unknown-linux-musl --examples

That worked.

GopherJ · 2020-05-27T16:14:30Z

@fpagliughi It looks awesome, I haven't tested yet, I'll test recently this feature

Antonito · 2020-06-18T08:31:40Z

I also experienced problems building the crate with musl.

I found out the paho-mqtt-sys/paho-mqtt.c doesn't point to the same commit as paho-mqtt.c@1.32. It seems it's because the tag changed at some point (see eclipse/paho.mqtt.c#861 (comment)).

Specifically, I found 2 issues:

The CMakeLists.txt has C++ support, as mentioned by @fpagliughi
The src/CMakeLists.txt doesn't check for LIB_ANL support (but v1.3.2 does!)

I managed to build it, without SSL supports by manually building the v1.3.2/1.3.3/1.3.4 C Library & installing it on my system and then building with the --no-default-features flag.

Rust toolchain setup:

rustup target add x86_64-unknown-linux-musl
rustup component add rustfmt --toolchain 1.44.0-x86_64-unknown-linux-gnu

Building the C library & installing it to /usr/local/:

RUN curl -LO https://github.com/eclipse/paho.mqtt.c/archive/v1.3.2.tar.gz                   \
    && tar xzf v1.3.2.tar.gz && cd paho.mqtt.c-1.3.2/                                       \
    && sed -i '/ENABLE_LANGUAGE(CXX)/d' ./CMakeLists.txt                                    \
    && cmake                                                                                \
    "-DPAHO_WITH_SSL=off" "-DPAHO_BUILD_DOCUMENTATION=off" "-DPAHO_BUILD_SAMPLES=off"       \
    "-DPAHO_BUILD_SHARED=off" "-DPAHO_ENABLE_TESTING=off" "-DPAHO_HIGH_PERFORMANCE=on"      \
    "-DPAHO_BUILD_STATIC=on"                                                                \
    "-DCMAKE_C_FLAGS=-ffunction-sections -fdata-sections -fPIC -m64"                        \
    "-DCMAKE_C_COMPILER=/usr/bin/musl-gcc"                                                  \
    "-DCMAKE_ASM_COMPILER=/usr/bin/musl-gcc" "-DCMAKE_BUILD_TYPE=Release"                   \
    .                                                                                       \
    && make && make install                                                                 \
    && cd ..                                                                                \
    && rm -rf v1.3.2.tar.gz paho.mqtt.c-1.3.2

(tested on Docker rust:1.44 & rust:1.44-alpine3.10)

The library compiles & links fine, but when I try to build & run an executable with it, I face linking errors (missing references to C's standard library, such as snprintf, rand..etc), so I guess it didn't link correctly with musl 🤔

On the other hand, I had no problem building & running with glibc.

fpagliughi · 2020-12-31T16:47:36Z

With v0.9, everything seems to be in place to do the musl builds, like I described above for the non-SSL version. That's pretty cool.

$ cargo build --no-default-features --features="bundled" --target=x86_64-unknown-linux-musl --examples

^ This works. Fully static executables.

Now, with the openssl-sys crate, I was hoping to be able to get the SSL version working out-of-the-box as well. Now the library builds, but I'm getting linker issues in the applications, such as if I try to build the example apps:

$ cargo build --features=vendored-ssl --target=x86_64-unknown-linux-musl --examples
...
uild/install/lib/libssl.a(ssl_conf.o): In function `ssl_conf_cmd_lookup':
          /home/fmp/mqtt/paho-rust/target/x86_64-unknown-linux-musl/debug/build/openssl-sys-e3faff97e20dd452/out/openssl-build/build/src/ssl/ssl_conf.c:777: undefined reference to `strcasecmp'
          /home/fmp/mqtt/paho-rust/target/x86_64-unknown-linux-musl/debug/build/openssl-sys-e3faff97e20dd452/out/openssl-build/install/lib/libssl.a(ssl_conf.o): In function `cmd_ECDHParameters':
          /home/fmp/mqtt/paho-rust/target/x86_64-unknown-linux-musl/debug/build/openssl-sys-e3faff97e20dd452/out/openssl-build/build/src/ssl/ssl_conf.c:233: undefined reference to `strcasecmp'
...
          /home/fmp/mqtt/paho-rust/target/x86_64-unknown-linux-musl/debug/build/openssl-sys-e3faff97e20dd452/out/openssl-build/install/lib/libcrypto.a(conf_def.o): In function `get_next_file':
          /home/fmp/mqtt/paho-rust/target/x86_64-unknown-linux-musl/debug/build/openssl-sys-e3faff97e20dd452/out/openssl-build/build/src/crypto/conf/conf_def.c:717: undefined reference to `strcasecmp'
          /home/fmp/mqtt/paho-rust/target/x86_64-unknown-linux-musl/debug/build/openssl-sys-e3faff97e20dd452/out/openssl-build/build/src/crypto/conf/conf_def.c:718: undefined reference to `strcasecmp'
          collect2: error: ld returned 1 exit status
          

error: aborting due to previous error

error: build failed

Has anyone had any luck with this?
@svanharmelen ? Any ideas?

fpagliughi · 2022-04-16T15:57:20Z

Sorry, I haven't made any progress with this. I'll push it off again to try and solve by the next release.

But any new thoughts or ideas would be fully appreciated.

FantomJAC · 2022-06-24T03:23:51Z

I have the exact same problem and here is my workaround.

Add build.rs with the below contents:

fn main() {
    // Linker workaround for openssl with musl
    println!("cargo:rustc-link-arg=-lc");
}

Somehow the linker doesn't pick up libc, so this script force adding -lc when the linker is invoked.

FantomJAC · 2022-06-27T21:51:33Z

I was also able to workaround this by adding .cargo/config with the following contents:

[target.x86_64-unknown-linux-musl]
rustflags = ["-C", "link-arg=-lc"]

msdrigg · 2022-10-01T15:43:32Z

@FantomJAC 's solution worked for me too

fpagliughi · 2022-10-03T21:05:52Z

Do you think it would work to add the link instruction to the -sys library's build.rs or does it need to specify the linking in the app's build?

fpagliughi · 2022-10-04T00:09:22Z

^ Nevermind, that didn't seem to work.

Perhaps I should just document these two work-around options in the README?

fpagliughi · 2023-01-09T12:36:39Z

I just put the workarounds into the README. If anyone has a better idea, please reopen this or make a new issue or PR.

fpagliughi mentioned this issue May 27, 2020

Fully static builds with musl eclipse/paho.mqtt.c#901

Closed

slinkydeveloper mentioned this issue Nov 10, 2020

#9 Encoders for MQTT cloudevents/sdk-rust#104

Open

fpagliughi added this to the v0.9 milestone Nov 30, 2020

fpagliughi modified the milestones: v0.9, v0.9.1 Dec 28, 2020

fpagliughi modified the milestones: v0.9.1, v0.10 Jan 1, 2021

fpagliughi modified the milestones: v0.10, v0.10.1 Jan 20, 2022

fpagliughi removed this from the v0.10.1 milestone Apr 14, 2022

fpagliughi added this to the v0.11.0 milestone Apr 14, 2022

fpagliughi modified the milestones: v0.11.0, v0.12.0 Apr 16, 2022

fpagliughi closed this as completed Jan 9, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

build with musl #57

build with musl #57

GopherJ commented Dec 2, 2019

fpagliughi commented Dec 2, 2019

GopherJ commented Dec 2, 2019 •

edited

fpagliughi commented Dec 3, 2019 •

edited

fpagliughi commented Dec 3, 2019

GopherJ commented Dec 5, 2019

GopherJ commented Dec 5, 2019

fpagliughi commented Dec 8, 2019

rcastill commented Mar 19, 2020

fpagliughi commented Mar 31, 2020 •

edited

GopherJ commented Apr 30, 2020 •

edited

fpagliughi commented Apr 30, 2020 •

edited

GopherJ commented Apr 30, 2020 •

edited

GopherJ commented Apr 30, 2020

GopherJ commented Apr 30, 2020

fpagliughi commented Apr 30, 2020

GopherJ commented Apr 30, 2020 •

edited

fpagliughi commented May 27, 2020

GopherJ commented May 27, 2020

Antonito commented Jun 18, 2020

fpagliughi commented Dec 31, 2020 •

edited

fpagliughi commented Apr 16, 2022

FantomJAC commented Jun 24, 2022

FantomJAC commented Jun 27, 2022

msdrigg commented Oct 1, 2022

fpagliughi commented Oct 3, 2022

fpagliughi commented Oct 4, 2022

fpagliughi commented Jan 9, 2023

build with musl #57

build with musl #57

Comments

GopherJ commented Dec 2, 2019

fpagliughi commented Dec 2, 2019

GopherJ commented Dec 2, 2019 • edited

fpagliughi commented Dec 3, 2019 • edited

fpagliughi commented Dec 3, 2019

GopherJ commented Dec 5, 2019

GopherJ commented Dec 5, 2019

fpagliughi commented Dec 8, 2019

rcastill commented Mar 19, 2020

fpagliughi commented Mar 31, 2020 • edited

GopherJ commented Apr 30, 2020 • edited

fpagliughi commented Apr 30, 2020 • edited

GopherJ commented Apr 30, 2020 • edited

GopherJ commented Apr 30, 2020

GopherJ commented Apr 30, 2020

fpagliughi commented Apr 30, 2020

GopherJ commented Apr 30, 2020 • edited

fpagliughi commented May 27, 2020

GopherJ commented May 27, 2020

Antonito commented Jun 18, 2020

fpagliughi commented Dec 31, 2020 • edited

fpagliughi commented Apr 16, 2022

FantomJAC commented Jun 24, 2022

FantomJAC commented Jun 27, 2022

msdrigg commented Oct 1, 2022

fpagliughi commented Oct 3, 2022

fpagliughi commented Oct 4, 2022

fpagliughi commented Jan 9, 2023

GopherJ commented Dec 2, 2019 •

edited

fpagliughi commented Dec 3, 2019 •

edited

fpagliughi commented Mar 31, 2020 •

edited

GopherJ commented Apr 30, 2020 •

edited

fpagliughi commented Apr 30, 2020 •

edited

GopherJ commented Apr 30, 2020 •

edited

GopherJ commented Apr 30, 2020 •

edited

fpagliughi commented Dec 31, 2020 •

edited