Protobuf/grpc PR

[KindDragon]

Cannot build gRPC for vcpkg

vcpkg>vcpkg install grpc
-- CURRENT_INSTALLED_DIR=D:/Work/vcpkg/installed/x86-windows
-- DOWNLOADS=D:/Work/vcpkg/downloads
-- CURRENT_PACKAGES_DIR=D:/Work/vcpkg/packages/grpc_x86-windows
-- CURRENT_BUILDTREES_DIR=D:/Work/vcpkg/buildtrees/grpc
-- CURRENT_PORT_DIR=D:/Work/vcpkg/ports/grpc/.
-- Downloading https://github.com/grpc/grpc/archive/ef5a2eb2678d38d35ff79c4d692bf48603e1c439.zip...
-- Downloading https://github.com/grpc/grpc/archive/ef5a2eb2678d38d35ff79c4d692bf48603e1c439.zip... OK
-- Testing integrity of downloaded file...
-- Testing integrity of downloaded file... OK
-- Extracting source D:/Work/vcpkg/downloads/grpc-ef5a2eb267.zip
-- Extracting done
-- Configuring x86-windows-rel
CMake Error at scripts/cmake/vcpkg_execute_required_process.cmake:14 (message):
  Command failed: C:/Program
  Files/CMake/bin/cmake.exe;D:/Work/vcpkg/buildtrees/grpc/src/grpc-ef5a2eb2678d38d35ff79c4d692bf48603e1c439;-DgRPC_ZLIB_PROVIDER=package;-DgRPC_SSL_PROVIDER=package;-DgRPC_PROTOBUF_PROVIDER=package;-G;Visual
  Studio 14
  2015;-DCMAKE_VERBOSE_MAKEFILE=ON;-DCMAKE_BUILD_TYPE=Release;-DCMAKE_TOOLCHAIN_FILE=D:/Work/vcpkg/triplets/x86-windows.cmake;-DCMAKE_PREFIX_PATH=D:/Work/vcpkg/installed/x86-windows;-DCMAKE_INSTALL_PREFIX=D:/Work/vcpkg/packages/grpc_x86-windows


  Working Directory: D:/Work/vcpkg/buildtrees/grpc/x86-windows-rel

  See logs for more information:

      D:\Work\vcpkg\buildtrees\grpc\config-x86-windows-rel-out.log
      D:\Work\vcpkg\buildtrees\grpc\config-x86-windows-rel-err.log

Call Stack (most recent call first):
  scripts/cmake/vcpkg_configure_cmake.cmake:27 (vcpkg_execute_required_process)
  ports/grpc/portfile.cmake:9 (vcpkg_configure_cmake)
  scripts/ports.cmake:96 (include)


Error: build command failed

config-x86-windows-rel-out.log.txt
config-x86-windows-rel-err.log.txt

[msftclas]

Hi @KindDragon, I'm your friendly neighborhood Microsoft Pull Request Bot (You can call me MSBOT). Thanks for your contribution!
You've already signed the contribution license agreement. Thanks!

The agreement was validated by Microsoft and real humans are currently evaluating your PR.

TTYL, MSBOT;

[KindDragon]

I use commit from master (not latest version) because it has advanced CMake script https://github.com/grpc/grpc/blob/ef5a2eb2678d38d35ff79c4d692bf48603e1c439/CMakeLists.txt

[KindDragon]

Please help me if you can 😃

[ras0219-msft]

This may be related to #77. I'm going to work on fixing that issue first, then I'll take a look here.

[ras0219-msft]

I believe I've tweaked protobuf into working; here's the portfile I ended up with:

include(vcpkg_common_functions)
vcpkg_download_distfile(ARCHIVE_FILE
    URLS "https://github.com/google/protobuf/releases/download/v3.0.2/protobuf-cpp-3.0.2.tar.gz"
    FILENAME "protobuf-cpp-3.0.2.tar.gz"
    SHA512 5c99fa5d20815f9333a1e30d4da7621375e179abab6e4369ef0827b6ea6a679afbfec445dda21a72b4ab11e1bdd72c0f17a4e86b153ea8e2d3298dc3bcfcd643
)
vcpkg_download_distfile(TOOL_ARCHIVE_FILE
    URLS "https://github.com/google/protobuf/releases/download/v3.0.2/protoc-3.0.2-win32.zip"
    FILENAME "protoc-3.0.2-win32.zip"
    SHA512 51c67bd8bdc35810da70786d873935814679c58b74e653923671bdf06b8b69a1c9a0793d090b17d25e91ddafff1726bcfcdd243373dd47c4aeb9ea83fbabaeb0
)
vcpkg_extract_source_archive(${ARCHIVE_FILE})
vcpkg_extract_source_archive(${TOOL_ARCHIVE_FILE} ${CURRENT_BUILDTREES_DIR}/src/protobuf-3.0.2-win32)

vcpkg_configure_cmake(
    SOURCE_PATH ${CURRENT_BUILDTREES_DIR}/src/protobuf-3.0.2/cmake
    OPTIONS
        -Dprotobuf_BUILD_SHARED_LIBS=ON
        -Dprotobuf_WITH_ZLIB=ON
        -Dprotobuf_BUILD_TESTS=OFF
        -DCMAKE_INSTALL_CMAKEDIR=share/protobuf
)

vcpkg_install_cmake()

file(REMOVE_RECURSE ${CURRENT_PACKAGES_DIR}/debug/include)

file(READ ${CURRENT_PACKAGES_DIR}/share/protobuf/protobuf-targets-release.cmake RELEASE_MODULE)
string(REPLACE "\${_IMPORT_PREFIX}/bin/protoc.exe" "\${_IMPORT_PREFIX}/tools/protoc.exe" RELEASE_MODULE "${RELEASE_MODULE}")
file(WRITE ${CURRENT_PACKAGES_DIR}/share/protobuf/protobuf-targets-release.cmake "${RELEASE_MODULE}")

file(READ ${CURRENT_PACKAGES_DIR}/debug/share/protobuf/protobuf-targets-debug.cmake DEBUG_MODULE)
string(REPLACE "\${_IMPORT_PREFIX}" "\${_IMPORT_PREFIX}/debug" DEBUG_MODULE "${DEBUG_MODULE}")
string(REPLACE "\${_IMPORT_PREFIX}/debug/bin/protoc.exe" "\${_IMPORT_PREFIX}/tools/protoc.exe" DEBUG_MODULE "${DEBUG_MODULE}")
file(WRITE ${CURRENT_PACKAGES_DIR}/share/protobuf/protobuf-targets-debug.cmake "${DEBUG_MODULE}")

file(REMOVE_RECURSE ${CURRENT_PACKAGES_DIR}/debug/share)

file(REMOVE ${CURRENT_PACKAGES_DIR}/bin/protoc.exe)
file(REMOVE ${CURRENT_PACKAGES_DIR}/debug/bin/protoc.exe)

file(INSTALL ${CURRENT_BUILDTREES_DIR}/src/protobuf-3.0.2/LICENSE DESTINATION ${CURRENT_PACKAGES_DIR}/share/protobuf RENAME copyright)
file(INSTALL ${CURRENT_BUILDTREES_DIR}/src/protobuf-3.0.2-win32/bin/protoc.exe DESTINATION ${CURRENT_PACKAGES_DIR}/tools)
vcpkg_copy_pdbs()

[note: I needed to add a target extraction directory argument to the extract command, so you'll need to pull from master to use this]

Essentially, I discard the built protoc.exe and instead always download the official win32 version. This binary (which has statically linked its dependencies) is placed in tools\protoc.exe. I also patch all the protobuf-targets-*.cmake files to point to this new location.

After using this version of protobuf, grpc fails on missing some other libraries (nanopb, for example).

I have not successfully built anything using this package (I'm not familiar enough with protobuf to prove to myself that it works); if this does work, could you either post back here (I'll check it in separately) or submit a separate PR?

[KindDragon]

Thank you, working on grpc

[KindDragon]

Should work now, but with statically linked protobuf (because of protobuf bug)

[KindDragon]

grpc create some exe tools too, but they do not publish tools on GitHub :(

gen_hpack_tables.exe, gen_legal_metadata_characters.exe, gen_percent_encoding_tables.exe, grpc_cpp_plugin.exe, grpc_create_jwt.exe, grpc_csharp_plugin.exe, grpc_node_plugin.exe, grpc_objective_c_plugin.exe, grpc_print_google_default_creds_token.exe, grpc_python_plugin.exe, grpc_ruby_plugin.exe, grpc_verify_jwt.exe

[ras0219-msft]

Are any of these executables used for C++ code generation? Or are they general purpose diagnostics / other language integration?

[KindDragon]

grpc_cpp_plugin.exe used for that, so we need at least that tool

[ras0219-msft]

Since protobuf only works on x86 and x64 anyway, for now it should be fine to put the built grpc_cpp_plugin.exe into a tools\ directory similar to the protoc.exe above. If it requires DLLs, we can copy those applocal and put the whole thing in a subdirectory; something like tools\grpc\*.{dll,exe}

In the future, it would be better to find prebuilt copies of the tools (otherwise, when targeting arm, we'll need to do multiple builds).

[adam4813]

Is there any chance of merging the protobuf PR separately as this is the last library I need for my projects and would love to not duplicate work.

[adam4813]

Any reason for forcing ZLIB on?

[ras0219-msft]

At the moment, we don't have a good story for optional dependencies (we're working on it!). Since zlib is so ubiquitous, it's cheap to require and provides more functionality.

[KindDragon]

Because ZLIB disabled only for MSVC

if (MSVC)
  set(protobuf_WITH_ZLIB_DEFAULT OFF)
else (MSVC)...

I think this is only because protobuf authors don't have zlib on Windows, but we have now zlib in vcpkg.

[ras0219-msft]

Thanks for the extensive process here; I'm glad we're getting closer to having a really solid solution for tools.