[protobuf] Allow protobuf to be crosscompiled. #14165

Nemirtingas · 2020-10-22T11:53:45Z

Protobuf port now asks for the host protoc, so .proto files can be built.

Describe the pull request
It adds a check for the native protoc binary for cross-compilation like the case when you were cross-compiling on Windows for another OS.

What does your PR fix?
Fixes [protobuf] Handle cross-compilation on Linux host #10156 and possibly all cross-compilations for protoc.
Which triplets are supported/not supported?
Default Windows, Linux and MacOS. Didn't test the community pack;
Have you updated the CI baseline?
No
Does your PR follow the maintainer guide?
I think so.

Protobuf port now asks for the host protoc, so .proto files can be built.

JackBoosY

I think there is always a need to generate protobuf tools.

ports/protobuf/portfile.cmake

JackBoosY · 2020-10-23T07:18:25Z

ports/protobuf/portfile.cmake

-if(CMAKE_HOST_WIN32 AND NOT VCPKG_TARGET_ARCHITECTURE MATCHES "x64" AND NOT VCPKG_TARGET_ARCHITECTURE MATCHES "x86")
+if(CMAKE_HOST_WIN32)
+  set(HOST_PROTOBUF_TRIPLET "x86-windows")
+  if(NOT VCPKG_TARGET_ARCHITECTURE MATCHES "x64" AND NOT VCPKG_TARGET_ARCHITECTURE MATCHES "x86")
    set(protobuf_BUILD_PROTOC_BINARIES OFF)


I think the uwp triplet also need to generate binaries.

I don't know, it was disabled before my change, was there any reason why ?

Maybe the reviewer ignored it.

Actually, it is built on UWP.

NOT (VCPKG_TARGET_IS_WINDOWS AND NOT VCPKG_TARGET_IS_UWP) => NOT (TRUE AND NOT TRUE) => NOT (TRUE AND FALSE) => NOT (FALSE) => TRUE

Am i right ?

ports/protobuf/portfile.cmake

JackBoosY · 2020-10-23T07:19:19Z

ports/protobuf/portfile.cmake

+if(protobuf_BUILD_PROTOC_BINARIES)
+  file(GLOB EXECUTABLES ${CURRENT_PACKAGES_DIR}/bin/protoc*)
+  foreach(E IN LISTS EXECUTABLES)
+    file(INSTALL ${E} DESTINATION ${CURRENT_PACKAGES_DIR}/tools/${PORT}
+         PERMISSIONS OWNER_READ OWNER_WRITE OWNER_EXECUTE GROUP_READ GROUP_WRITE GROUP_EXECUTE WORLD_READ)
+  endforeach()
+else()
+  file(GLOB EXECUTABLES ${_VCPKG_INSTALLED_DIR}/${HOST_PROTOBUF_TRIPLET}/tools/${PORT}/protoc*)
+  foreach(E IN LISTS EXECUTABLES)
+    file(INSTALL ${E} DESTINATION ${CURRENT_PACKAGES_DIR}/tools/${PORT}/
+         PERMISSIONS OWNER_READ OWNER_WRITE OWNER_EXECUTE GROUP_READ GROUP_WRITE GROUP_EXECUTE WORLD_READ)
+  endforeach()
+endif()


We prefert to use vcpkg_copy_tools instead of this.

Problem is that its not generic enought. protoc is a symbolic link on OS that are not Windows:

ll installed/x64-linux/tools/protobuf/ lrwxrwxrwx 1 root root 15 oct. 22 14:11 protoc -> protoc-3.13.0.0* -rwxrwxr-- 1 root root 5602472 oct. 22 14:11 protoc-3.13.0.0*

So if I use vcpkg_copy_tools, it will only copy the symbolic link. And protoc has the version appended to it, thats why I use the wildcard instead of vcpkg_copy_tools.

So I think we should expand vcpkg_copy_tools instead of modifying it here. Please open an issue to report this.

So I think we should expand vcpkg_copy_tools instead of modifying it here. Please open an issue to report this.

I think this is already tracked in #11547 .

Interesting, I wont open a new issue then.

ports/protobuf/portfile.cmake

JackBoosY · 2020-10-23T07:20:44Z

ports/protobuf/portfile.cmake

+elseif(CMAKE_HOST_UNIX)
+  set(HOST_PROTOBUF_TRIPLET "x64-linux")
+  if(NOT VCPKG_TARGET_ARCHITECTURE MATCHES "x64" AND NOT VCPKG_TARGET_ARCHITECTURE MATCHES "x86")
+    set(protobuf_BUILD_PROTOC_BINARIES OFF)


See comment above.

ports/protobuf/portfile.cmake

Co-authored-by: Jack·Boos·Yu <47264268+JackBoosY@users.noreply.github.com>

Nemirtingas · 2020-10-23T16:47:57Z

I can't find any VCPKG_HOST_ARCHITECTURE, any idea ? ${CMAKE_SYSTEM_PROCESSOR} and ${CMAKE_HOST_SYSTEM_PROCESSOR} seem empty. I'd like to do something like VCPKG_TARGET_ARCHITECTURE == VCPKG_HOST_ARCHITECTURE then build binaries.

linquize

ARM and ARM64 linux support

ports/protobuf/portfile.cmake

strega-nil · 2020-11-12T22:32:31Z

Requires cross-talk with #14318

Nemirtingas · 2020-11-13T05:54:04Z

Requires cross-talk with #14318

Adding a new variable to VCPKG like VCPKG_HOST_TRIPLET would be nice for cross-compilation. Thats what I triied to achieve in this pr for protoc.

BillyONeal · 2020-12-17T20:50:40Z

Requires cross-talk with #14318

To clarify: this PR tries to solve the cross compilation problems for protobuf, by repeating much of our existing triplet system within the protobuf port; we aren't fans of doing something like that because it's fragile, and it suggests that vcpkg the tool needs a more complete solution to these building a tool that we use in the build scenarios. The other PR #14318 solves a smaller subset of target architectures but gets both protobuf and grpc, so neither is a true superset of the other.

Building some kind of support for "tool ports" which use the host rather than the target settings is certainly on our radar but we don't want to expand the existing workarounds drastically right now.

Leaving this PR and the other open because at such time we address this problem more completely we're going to want to make sure both these are addressed well.

Nemirtingas · 2020-12-18T06:23:54Z

Building some kind of support for "tool ports" which use the host rather than the target settings is certainly on our radar but we don't want to expand the existing workarounds drastically right now.

Hi, yes I can understand that.

NancyLi1013 · 2020-12-18T07:46:45Z

Could you please resolve the conflicts?

Nemirtingas · 2020-12-18T10:34:12Z

Could you please resolve the conflicts?

Is it actually worthing it ? BillyONeal seemed to say that this pr won't be merged until a suitable solution to build host tools is found. At least thats my understanding.

BillyONeal · 2020-12-18T17:37:29Z

@Nemirtingas I think @NancyLi1013 just saw this PR get pinged due to "reopening" and didn't follow the whole thread.

Nemirtingas · 2020-12-19T10:30:10Z

@Nemirtingas I think @NancyLi1013 just saw this PR get pinged due to "reopening" and didn't follow the whole thread.

Ah right, Thoses 2 "comment" buttons... I often hit the wrong one 🤐

NancyLi1013 · 2020-12-21T09:01:36Z

Sorry for my above comments. As @BillyONeal said, I didn't follow the whole thread.

Allow protobuf to be crosscompiled.

ecf96a1

Protobuf port now asks for the host protoc, so .proto files can be built.

Nemirtingas changed the title ~~Allow protobuf to be crosscompiled.~~ [protobuf] Allow protobuf to be crosscompiled. Oct 22, 2020

LilyWangL assigned NancyLi1013 Oct 23, 2020

NancyLi1013 added the category:port-feature The issue is with a library, which is requesting new capabilities that didn’t exist label Oct 23, 2020

JackBoosY suggested changes Oct 23, 2020

View reviewed changes

JackBoosY added the requires:author-response label Oct 23, 2020

Apply suggestions from code review

508884a

Co-authored-by: Jack·Boos·Yu <47264268+JackBoosY@users.noreply.github.com>

linquize reviewed Oct 24, 2020

View reviewed changes

Nemirtingas force-pushed the protoc_crosscompile branch from fa5194e to c615203 Compare October 24, 2020 14:59

Try to detect host ARCH

756068c

Nemirtingas force-pushed the protoc_crosscompile branch from c615203 to 756068c Compare October 24, 2020 15:05

GameDumper added 3 commits October 24, 2020 17:25

Fix brackets

c90d8b1

Fix x86 HOST and AMD64 W6432

403a07d

Remove trailing slash.

ba36c4d

JackBoosY suggested changes Oct 26, 2020

View reviewed changes

ports/protobuf/portfile.cmake Show resolved Hide resolved

ports/protobuf/portfile.cmake Show resolved Hide resolved

JackBoosY added the requires:discussion label Oct 26, 2020

JackBoosY requested a review from BillyONeal October 26, 2020 08:37

Remove duplicate condition

c36672a

JackBoosY removed the requires:author-response label Oct 29, 2020

traversaro mentioned this pull request Oct 31, 2020

ignition-msgs5: Add support for cross-compilation #11538

Closed

strega-nil mentioned this pull request Nov 12, 2020

[grpc,opencv] Fix cross compile issue. #14318

Closed

strega-nil added the depends:different-pr This PR or Issue depends on a PR which has been filed label Nov 12, 2020

BillyONeal removed the requires:discussion label Dec 17, 2020

Nemirtingas closed this Dec 18, 2020

Nemirtingas reopened this Dec 18, 2020

Nemirtingas closed this Apr 12, 2021

Nemirtingas deleted the protoc_crosscompile branch April 12, 2021 14:16

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[protobuf] Allow protobuf to be crosscompiled. #14165

[protobuf] Allow protobuf to be crosscompiled. #14165

Nemirtingas commented Oct 22, 2020

JackBoosY left a comment

JackBoosY Oct 23, 2020

Nemirtingas Oct 23, 2020

JackBoosY Oct 23, 2020

Nemirtingas Oct 23, 2020 •

edited

Loading

JackBoosY Oct 23, 2020

Nemirtingas Oct 23, 2020

JackBoosY Oct 23, 2020

traversaro Oct 23, 2020 •

edited

Loading

Nemirtingas Oct 23, 2020

JackBoosY Oct 23, 2020

Nemirtingas commented Oct 23, 2020 •

edited

Loading

linquize left a comment

strega-nil commented Nov 12, 2020

Nemirtingas commented Nov 13, 2020 •

edited

Loading

BillyONeal commented Dec 17, 2020

Nemirtingas commented Dec 18, 2020

NancyLi1013 commented Dec 18, 2020

Nemirtingas commented Dec 18, 2020

BillyONeal commented Dec 18, 2020

Nemirtingas commented Dec 19, 2020

NancyLi1013 commented Dec 21, 2020

[protobuf] Allow protobuf to be crosscompiled. #14165

[protobuf] Allow protobuf to be crosscompiled. #14165

Conversation

Nemirtingas commented Oct 22, 2020

JackBoosY left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Nemirtingas Oct 23, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

traversaro Oct 23, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Nemirtingas commented Oct 23, 2020 • edited Loading

linquize left a comment

Choose a reason for hiding this comment

strega-nil commented Nov 12, 2020

Nemirtingas commented Nov 13, 2020 • edited Loading

BillyONeal commented Dec 17, 2020

Nemirtingas commented Dec 18, 2020

NancyLi1013 commented Dec 18, 2020

Nemirtingas commented Dec 18, 2020

BillyONeal commented Dec 18, 2020

Nemirtingas commented Dec 19, 2020

NancyLi1013 commented Dec 21, 2020

Nemirtingas Oct 23, 2020 •

edited

Loading

traversaro Oct 23, 2020 •

edited

Loading

Nemirtingas commented Oct 23, 2020 •

edited

Loading

Nemirtingas commented Nov 13, 2020 •

edited

Loading