Problem: The addition of sha1 sources breaks static linking

[myd7349]

Please use this template for reporting suspected bugs or requests for help.

Issue description

I encountered a zyre REGRESSION error when I was trying to update vcpkg's zeromq to lastest: microsoft/vcpkg#8119
In #3579, external/sha1/sha1.h and external/sha1/sha1.c are added. However, czmq also has its own copy of these two files: https://github.com/zeromq/czmq/tree/master/src/foreign/sha1

If both libzmq and czmq are built as static libraries, then the compiler will be unhappy:

/ci/myagent/_work/1/s/installed/x64-linux/debug/lib/libzmq.a(sha1.c.o):/ci/myagent/_work/1/s/buildtrees/zeromq/src/407c1e69ad-6a0d86f40e/external/sha1/sha1.c:225: first defined here
/ci/myagent/_work/1/s/installed/x64-linux/debug/lib/libczmq.a(zdigest.c.o): In function `sha1_pad':
/ci/myagent/_work/1/s/buildtrees/czmq/src/ff5f9b6893-f1fd59e8c7/src/foreign/sha1/sha1.inc_c:235: multiple definition of `sha1_pad'
/ci/myagent/_work/1/s/installed/x64-linux/debug/lib/libzmq.a(sha1.c.o):/ci/myagent/_work/1/s/buildtrees/zeromq/src/407c1e69ad-6a0d86f40e/external/sha1/sha1.c:236: first defined here
/ci/myagent/_work/1/s/installed/x64-linux/debug/lib/libczmq.a(zdigest.c.o): In function `sha1_loop':
/ci/myagent/_work/1/s/buildtrees/czmq/src/ff5f9b6893-f1fd59e8c7/src/foreign/sha1/sha1.inc_c:278: multiple definition of `sha1_loop'
/ci/myagent/_work/1/s/installed/x64-linux/debug/lib/libzmq.a(sha1.c.o):/ci/myagent/_work/1/s/buildtrees/zeromq/src/407c1e69ad-6a0d86f40e/external/sha1/sha1.c:279: first defined here
/ci/myagent/_work/1/s/installed/x64-linux/debug/lib/libczmq.a(zdigest.c.o): In function `sha1_result':
/ci/myagent/_work/1/s/buildtrees/czmq/src/ff5f9b6893-f1fd59e8c7/src/foreign/sha1/sha1.inc_c:306: multiple definition of `sha1_result'
/ci/myagent/_work/1/s/installed/x64-linux/debug/lib/libzmq.a(sha1.c.o):/ci/myagent/_work/1/s/buildtrees/zeromq/src/407c1e69ad-6a0d86f40e/external/sha1/sha1.c:307: first defined here
collect2: error: ld returned 1 exit status
[19/25] : && /usr/bin/c++  -fPIC -g  -rdynamic CMakeFiles/ztester_gossip.dir/src/ztester_gossip.c.o  -o ztester_gossip  libzyre.a /ci/myagent/_work/1/s/installed/x64-linux/debug/lib/libzmq.a /ci/myagent/_work/1/s/installed/x64-linux/debug/lib/libczmq.a /ci/myagent/_work/1/s/installed/x64-linux/debug/lib/libzmq.a -lrt -lpthread && :
FAILED: ztester_gossip 

Environment

• libzmq version (e756743):
• OS: Ubuntu 19.04

Minimal test code / Steps to reproduce the issue

What's the actual result? (include assertion message & call stack if applicable)

What's the expected result?

[bluca]

@somdoron is there an external library that could be used to get those functions instead of embedding it?

[somdoron]

@somdoron is there an external library that could be used to get those functions instead of embedding it?

Plenty I guess, but that will make a dependency. I wonder if defining LIBZMQ_PRIVATE to __attribute__ ((visibility ("hidden"))) and marking the methods will solve it.

Anyway, the easiest solution is to change the function names to something like zmq_.

[bluca]

Isn't the visibility only used for dynamic linking? I'd very much rather not mangle external code, as it makes it harder to maintain.
IMHO the best solution would be to add the dependency, make it optional and only if not found use the embedded code. Sort of like libsodium vs tweetnacl.

[somdoron]

NSS and OpenSSL are an option:
https://nss-crypto.org/

[somdoron]

But it won't solve the problem in case the dependency for czmq.
During czmq compilation can we check if the function already defined by zeromq and not compile it so?
we can use the NETINET6_SHA1_H definition maybe.

[bluca]

openssl is not possible unfortunately as the license is not compatible with the GPL.
To check NETINET6_SHA1_H it would require again to change the external code right?

[somdoron]

No, it is already defined that way, what we need to do is to check during the compilation of czmq if the flag is defined, and only if not defined include sha1.c

[somdoron]

It is actually the same file, between czmq and libzmq.

[somdoron]

we can also check the libzmq version and drop the file if static compilation and version is higher than last released version

[bluca]

No, it is already defined that way, what we need to do is to check during the compilation of czmq if the flag is defined, and only if not defined include sha1.c

I'm not sure how this would be done? It would require inspecting the ar, which will get messy especially when cross compiling

[somdoron]

We can use NSS, but it won't solve the problem when the dependency is not found

[sigiesec]

I am not sure if the problem also exists under Windows, but at least for Linux it should be possible to declare the sha* functions as weak symbols with #pragma weak, in either or both of libzmq and czmq.

[somdoron]

@bluca can we use LibreSSL? it seems like the license is ok and the API is the same as OpenSSL.

[bluca]

The license is the same as (old) openssl as it's a fork, and thus still incompatible with the GPL

[somdoron]

I will use NSS, working on it

[somdoron]

@bluca is apache 2.0 license works?

[bluca]

it's compatible with GPL3 yes, but it is not compatible with GPL2

[somdoron]

I have the NSS ready, however building it on windows is not going to be easy...

[somdoron]

@myd7349 following PR should allow to build with external library (NSS):
#3677

Supporting both cmake and autotools.

I'm not sure how to make vcpkg use the external library or disable websocket transport so the build will succeed.

[myd7349]

@somdoron Thanks!
Currently, NSS-proto is not included in vcpkg yet(microsoft/vcpkg#7172). So, disabling websocket may be the only choice. I will look into it tomorrow.

[myd7349]

I have added a new feature in zeromq's control file: websockets-sha1:

./vcpkg install zeromq[websockets-sha1]

will build zeromq with sha1 sources and enable WebSocket transport support.

./vcpkg install zeromq

will exclude sha1 sources and disable WebSocket support.

[TomzBench]

Hello - I am reading through and don't follow how to fix... I am compiling czmq and libzmq as static library and run into this error.

EDIT - I added DISABLE_WS:BOOL=ON to libzmq and now compiles...