Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Failed to build czmq on Windows via CMake #1972

Closed
myd7349 opened this issue Dec 18, 2018 · 12 comments · Fixed by #2012
Closed

Failed to build czmq on Windows via CMake #1972

myd7349 opened this issue Dec 18, 2018 · 12 comments · Fixed by #2012

Comments

@myd7349
Copy link
Contributor

myd7349 commented Dec 18, 2018

Hey, guys! I am back again.

I am trying to add czmq to vcpkg these days(microsoft/vcpkg#4979). When building czmq with CMake, I encountered several problems.

  1. CMAKE_CURRENT_SOURCE_DIR isn't in CMAKE_MODULE_PATH.

    https://github.com/zeromq/czmq/blob/master/CMakeLists.txt#L19

    so Findlibzmq.cmake, Findlibsodium.cmake, .etc. may not be called.

  2. When built as a static library, libzmq will set an additional compiler flag -DZMQ_STATIC through set_target_properties:

    https://github.com/zeromq/libzmq/blob/master/CMakeLists.txt#L1060

    so if we want to build czmq as a static library on Windows, we should define ZMQ_STATIC manually(Maybe we can define it in Findlibzmq.cmake). Otherwise, the build will fail: undefined reference in mingw when link static library. libzmq#3318

  3. Findlibzmq.cmake is not be able to find libzmq on Windows.

    libzmq dll/lib built with MSVC is named using the Boost convention(something like libzmq-mt-4_3_1.lib, libzmq-mt-4_3_1.dll). So find_library (LIBZMQ_LIBRARIES NAMES zmq HINTS ${PC_LIBZMQ_LIBRARY_HINTS}) will not work.

    Here is a patch for solving this problem: https://bitbucket.org/ignitionrobotics/ign-transport/pull-requests/19/detec-zmq-under-windows-using-a-findzmq/diff

  4. Findlibsodium.cmake will not found libsodium on Windows

    When built with MSVC, libsodium will generate a library named libsodium.dll/lib. In this case, find_library(LIBSODIUM_LIBRARY NAMES sodium) is not able to find the library. I found that find_library(LIBSODIUM_LIBRARY NAMES sodium libsodium) works.

    By the way, Findlibcurl.cmake has this problem too.

  5. Maybe we can add some options like:

    WITH_LIBSODIUM
    WITH_LIBCURL
    WITH_LZ4
    WITH_UUID
    BUILD_TESTS (For example: Create a configuration flag to disable tests. googleapis/google-cloud-cpp#1617)
    BUILD_TOOLS (zmakecret)

@ras0219-msft
Copy link

We can (and should) fix (2) upstream in vcpkg by encoding ZMQ_STATIC into the zmq headers.

@ghost
Copy link

ghost commented Dec 19, 2018

You don't need to add BUILD_TESTS option. include(CTest) automatically calls enable_testing() and creates a BUILD_TESTING option for you. Depending on how you've structured your CMakeLists, you may need to wrap certain blocks in IF (BUILD_TESTING).

@myd7349
Copy link
Contributor Author

myd7349 commented Dec 19, 2018

@ras0219-msft Just like this?

if(VCPKG_LIBRARY_LINKAGE STREQUAL "static")
    file(READ ${CURRENT_PACKAGES_DIR}/include/zmq.h _contents)
    string(REPLACE "defined ZMQ_STATIC" "1 //defined ZMQ_STATIC" _contents "${_contents}")
    file(WRITE ${CURRENT_PACKAGES_DIR}/include/zmq.h "${_contents}")
endif()

I am considering to add these line to zeromq/portfile.cmake through microsoft/vcpkg#4214.

@remyabel Thanks for telling me that. :-)

@bluca
Copy link
Member

bluca commented Dec 19, 2018

Note that libsodium is not a dependency of CZMQ, so there's no need for an option for it

@stale
Copy link

stale bot commented Mar 21, 2019

This issue has been automatically marked as stale because it has not had recent activity for 90 days. It will be closed if no further activity occurs within 21 days. Thank you for your contributions.

@stale stale bot added the stale label Mar 21, 2019
@myd7349
Copy link
Contributor Author

myd7349 commented Apr 2, 2019

Do not close yet. I'd like to tackle this issue.

@myd7349
Copy link
Contributor Author

myd7349 commented Apr 19, 2019

A quick fix of this problem:

First, build and install libzmq on Windows:

cd libzmq
mkdir build
cd build
cmake .. -DCMAKE_INSTALL_PREFIX=C:/libzmq
cmake --build . --target install

Then, clone czmq and replace Findlibzmq.cmake with this one: https://github.com/myd7349/Ongoing-Study/blob/master/cpp/CMake/find_libzmq_ng/Findlibzmq.cmake

cd czmq
mkdir build
cd build
cmake .. -DCMAKE_PREFIX_PATH=C:/libzmq

The CI configuration file has this line: https://github.com/zeromq/czmq/blob/master/appveyor.yml#L53
That is, it renames libs like libzmq-v141-mt-gd-4_3_2.lib to zmq.lib.

@myd7349
Copy link
Contributor Author

myd7349 commented Apr 19, 2019

Findlibzmq.cmake is generated by these code: https://github.com/zeromq/zproject/blob/master/zproject_cmake.gsl#L640-L697 via gsl -target:* project.xml.

@myd7349
Copy link
Contributor Author

myd7349 commented Apr 19, 2019

I think we may have three ways to fix the libzmq not found issue on Windows:

  1. Remove <use project = "libzmq" /> from czmq/project.xml, and maintain Findlibzmq.cmake seperately. Then we can fix this problem by patching Findlibzmq.cmake manually.
  2. Change these code https://github.com/zeromq/zproject/blob/master/zproject_cmake.gsl#L640-L697 to handle zmq specially so that a special version of Findlibzmq.cmake is generated like the one I gave above.
  3. Prefer ZeroMQ's CMake target instead of Findlibzmq.cmake.
    ZeroMQ itself provides a CMake target(libzmq for a shared build, and libzmq-static for a static build). So we may consider replacing Findlibzmq.cmake with find_package(ZeroMQ CONFIG REQUIRED).
    This solution also require us to remove <use project = "libzmq" /> from czmq/project.xml, and then add find_package(ZeroMQ CONFIG REQUIRED) to this file: https://github.com/zeromq/zproject/blob/master/zproject_cmake.gsl

I personally think solution 3 is the best, since it uses modern cmake.

Update:
I noticed the existance of this #1939. So maybe I will fix this problem through solution 1 or 2.

@eplumer
Copy link

eplumer commented Apr 19, 2019

To close out on my question for #2004 which was redirected here I have attached the manual steps that I got working on my VS 2013 system. Thanks for pointing me to he appveyor.yml file for hints:

set ZMQ_ROOT=path_to_my_project_root
set CMAKE_GENERATOR="Visual Studio 12 2013 Win64"
set LIBZMQ_SOURCEDIR=%ZMQ_ROOT%\libzmq
set LIBZMQ_BUILDDIR=%LIBZMQ_SOURCEDIR%\BUILD
set CZMQ_SOURCEDIR=%ZMQ_ROOT%\czmq
set CZMQ_BUILDDIR=%CZMQ_SOURCEDIR%\BUILD

rmdir /s /q %LIBZMQ_BUILDDIR%
mkdir %LIBZMQ_BUILDDIR%
cd %LIBZMQ_BUILDDIR%

cmake -D CMAKE_CXX_FLAGS_RELEASE="/MT" -D CMAKE_CXX_FLAGS_DEBUG="/MTd" -G %CMAKE_GENERATOR% %LIBZMQ_SOURCEDIR%

C:\Progra~2\MSBuild\12.0\Bin\msbuild /v:minimal /maxcpucount:%NUMBER_OF_PROCESSORS% /p:Configuration=Release libzmq.vcxproj

C:\Progra~2\MSBuild\12.0\Bin\msbuild /v:minimal /maxcpucount:%NUMBER_OF_PROCESSORS% /p:Configuration=Debug libzmq.vcxproj

copy /Y %LIBZMQ_BUILDDIR%\lib\Release\libzmq-v120-mt-4_3_1.lib %LIBZMQ_BUILDDIR%\lib\Release\zmq.lib
copy /Y %LIBZMQ_BUILDDIR%\lib\Debug\libzmq-v120-mt-gd-4_3_1.lib %LIBZMQ_BUILDDIR%\lib\Debug\zmq.lib

rmdir /s /q %CZMQ_BUILDDIR%
mkdir %CZMQ_BUILDDIR%
cd %CZMQ_BUILDDIR%

cmake -G %CMAKE_GENERATOR% -D CMAKE_INCLUDE_PATH="%LIBZMQ_SOURCEDIR%\include" -D CMAKE_LIBRARY_PATH="%LIBZMQ_BUILDDIR%\lib\Release" -D CMAKE_C_FLAGS_RELEASE="/MT" -D CMAKE_CXX_FLAGS_RELEASE="/MT" -D CMAKE_C_FLAGS_DEBUG="/MTd" %CZMQ_SOURCEDIR%

Open Visual Studio 2013 and Rebuild "Release | x64" for "%CZMQ_BUILDDIR%\czmq.sln"

copy /Y %LIBZMQ_BUILDDIR%\bin\Release\libzmq-v120-mt-4_3_1.dll %CZMQ_BUILDDIR%\Release\

cmake -G %CMAKE_GENERATOR% -D CMAKE_INCLUDE_PATH="%LIBZMQ_SOURCEDIR%\include" -D CMAKE_LIBRARY_PATH="%LIBZMQ_BUILDDIR%\lib\Debug" -D CMAKE_C_FLAGS_RELEASE="/MT" -D CMAKE_CXX_FLAGS_RELEASE="/MT" -D CMAKE_C_FLAGS_DEBUG="/MTd" %CZMQ_SOURCEDIR%

Open Visual Studio 2013 and Rebuild "Debug | x64" for "%CZMQ_BUILDDIR%\czmq.sln"

copy /Y %LIBZMQ_BUILDDIR%\bin\Debug\libzmq-v120-mt-gd-4_3_1.dll %CZMQ_BUILDDIR%\Debug\
copy /Y %LIBZMQ_BUILDDIR%\bin\Debug\libzmq-v120-mt-gd-4_3_1.pdb %CZMQ_BUILDDIR%\Debug\

myd7349 added a commit to myd7349/czmq that referenced this issue Apr 23, 2019
Findlibzmq.cmake should be regenerated to fix CMake build error on Win32 (zeromq#1972).
@myd7349
Copy link
Contributor Author

myd7349 commented Jun 9, 2019

Finally, I am able to build czmq (and almost all of its dependencies) successfully in vcpkg: microsoft/vcpkg#4979
czmq depends on many 3rd party libraries (optional or required), and it is hard to handle all these dependencies correctly with find_library.

ZeroMQ, for example, can use libsodium optionally. And ZeroMQ's CMake module exports a target named libzmq (or libzmq-static with a static build). The exported CMake target file may looks like this (Windows, Debug mode):

#----------------------------------------------------------------
# Generated CMake target import file for configuration "Debug".
#----------------------------------------------------------------

# Commands may need to know the format version.
set(CMAKE_IMPORT_FILE_VERSION 1)

# Import target "libzmq" for configuration "Debug"
set_property(TARGET libzmq APPEND PROPERTY IMPORTED_CONFIGURATIONS DEBUG)
set_target_properties(libzmq PROPERTIES
  IMPORTED_IMPLIB_DEBUG "${_IMPORT_PREFIX}/debug/lib/libzmq-mt-gd-4_3_2.lib"
  IMPORTED_LINK_INTERFACE_LIBRARIES_DEBUG "${_IMPORT_PREFIX}/debug/lib/libsodium.lib;ws2_32;rpcrt4;iphlpapi"
  IMPORTED_LOCATION_DEBUG "${_IMPORT_PREFIX}/debug/bin/libzmq-mt-gd-4_3_2.dll"
  )

list(APPEND _IMPORT_CHECK_TARGETS libzmq )
list(APPEND _IMPORT_CHECK_FILES_FOR_libzmq "${_IMPORT_PREFIX}/debug/lib/libzmq-mt-gd-4_3_2.lib" "${_IMPORT_PREFIX}/debug/bin/libzmq-mt-gd-4_3_2.dll" )

# Commands beyond this point should not need to know the version.
set(CMAKE_IMPORT_FILE_VERSION)

If the users use the exported CMake module in a CMake-based project like this:

find_package(ZeroMQ CONFIG REQUIRED)
target_link_libraries(main PRIVATE libzmq)

then they don't worry about the libsodium dependency (It is already recorded in the IMPORTED_LINK_INTERFACE_LIBRARIES_* property of libzmq target).

If the users use find_library instead, they have to handle the libsodium dependency manually, and all the other platform specific libraries ZeroMQ may depends on. So, I replace Findlibzmq.cmake with this one in microsoft/vcpkg#4979:

find_package(ZeroMQ CONFIG REQUIRED)

set(LIBZMQ_INCLUDE_DIRS ${ZeroMQ_INCLUDE_DIR})
set(LIBZMQ_LIBRARIES libzmq libzmq-static)

include(FindPackageHandleStandardArgs)
find_package_handle_standard_args(
    LIBZMQ
    REQUIRED_VARS LIBZMQ_LIBRARIES LIBZMQ_INCLUDE_DIRS
)

and apply this patch to the cmake/Config.cmake.in:

diff --git a/builds/cmake/Config.cmake.in b/builds/cmake/Config.cmake.in
index 9c15f36a..e1475cd6 100644
--- a/builds/cmake/Config.cmake.in
+++ b/builds/cmake/Config.cmake.in
@@ -1,4 +1,14 @@
 @PACKAGE_INIT@

+include(CMakeFindDependencyMacro)
+
+find_dependency(ZeroMQ)
+
+if (@CZMQ_WITH_LIBCURL@ AND @LIBCURL_FOUND@)
+    find_dependency(OpenSSL)
+    find_dependency(ZLIB)
+endif ()
+
+
 include("${CMAKE_CURRENT_LIST_DIR}/@PROJECT_NAME@Targets.cmake")
 check_required_components("@PROJECT_NAME@")

I noticed the existence of #1939, and totally understand @bluca's concern. So a better solution should take both of the two cases into consideration:

find_package(ZeroMQ CONFIG QUIET)
if (ZeroMQ_FOUND)
    set(ZMQ_CMAKE_MODULE_AVAILABLE 1)
else ()
    set(ZMQ_CMAKE_MODULE_AVAILABLE 0)
    # Fall back to find_library
    ......
endif ()

and in the cmake/Config.cmake.in:

if (@ZMQ_CMAKE_MODULE_AVAILABLE@)
    find_dependency(ZeroMQ)
endif ()

czmq's another optional dependency, libcurl, also faces this problem. libcurl itself may depends on OpenSSL, zlib, and others. So:

find_library (
    LIBCURL_LIBRARIES
    NAMES curl
    HINTS ${PC_LIBCURL_LIBRARY_HINTS}
)

is not enough when libcurl is built as a static library. Besides, on Windows, the lib name of libcurl is not always libcurl.lib, so I have to take care of them this way:

find_library(
    LIBCURL_LIBRARY_DEBUG
    NAMES libcurl-d_imp libcurl-d curl-d curl
)

find_library(
    LIBCURL_LIBRARY_RELEASE
    NAMES libcurl_imp libcurl curl
)

google-cloud-cpp handles libcurl dependency this way: https://github.com/googleapis/google-cloud-cpp/pull/2407/files

So you see, a generic Find*.cmake template sounds great, but it doesn't work all the time. If we can mantain those Find*.cmake standalone, then we may refine them freely. Otherwise, we have to use a bounch of if else in zproject_cmake.gsl to handle all the special cases when a generic solution is not enough (like what I did in zeromq/zproject#1168).

If we do not want to go that far currently:

  1. For *nix users, things seem not that bad. So, we do not have to change anything.
  2. For Windows users with vcpkg (VS2015+ required), they may install czmq via:
    vcpkg install czmq:x86-windows
    or
    vcpkg install czmq[curl]:x86-windows
    to build czmq with libcurl.
  3. For Windows users without vcpkg or with older versions of Visual Studio, they may use the deprecated Visual Studio solutions or CMake. And when they use CMake, they should handle czmq's indirect dependencies (libsodium, OpenSSL, zlib, etc.) themself or just forget about them if they don't need them.

@myd7349
Copy link
Contributor Author

myd7349 commented Jun 9, 2019

OK. Before I close this issue, I want to do two more things:

  1. Add CZMQ_WITH_LIBCURL, CZMQ_WITH_LZ4, CZMQ_WITH_UUID options to give the users more control over those optional 3rd dependencies so that I don't have to use a patch here: https://github.com/microsoft/vcpkg/pull/4979/files#diff-3aedb01dfe69985b0650dc857d7b6e1a
  2. Update CZMQ's install instruction for Windows users.

myd7349 added a commit to myd7349/zproject that referenced this issue Jun 9, 2019
This patch also:
1. Simplify Findlibzmq.cmake
2. Use both libname and linkname when they are not identical

Related issue: zeromq/czmq#1972
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants