protobuf: add protoc option #1179

madebr · 2020-03-24T20:23:39Z

Specify library name and version: protobuf/3.9.1.0

This recipe allows protobuf to be used as a build requirement.
Add option to build protoc.

fixes #65 #1956

I've read the guidelines for contributing.
I've followed the PEP8 style guides for Python code in the recipes.
I've used the latest Conan client version.
I've tried at least one configuration locally with the
conan-center hook activated.

conan-center-bot · 2020-03-24T20:39:23Z

Some configurations of 'protobuf/3.9.1.0' failed in build 1 (e594974848d15c887147cf2644e65585413745d6):

Macos x86_64, Debug, apple-clang 9.1, libc++ . Options: protobuf:shared-True
Access to all the logs
Windows x86_64, Debug, Visual Studio 14, MTd. Options: protobuf:shared-False
- Hooks errors detected:
  - [HOOK - conan-center.py] post_package_info(): ERROR: [CMAKE FILE NOT IN BUILD FOLDERS (KB-H019)] The *.cmake files have to be placed in a folder declared as cpp_info.builddirs. Currently folders declared: ['lib\\cmake\\protobuf'] (https://github.com/conan-io/conan-center-index/wiki/Error-Knowledge-Base#KB-H019)
  - [HOOK - conan-center.py] post_package_info(): ERROR: [CMAKE FILE NOT IN BUILD FOLDERS (KB-H019)] Found files:
- Log download
- Log error Download
- Profile
Access to all the logs

madebr · 2020-03-25T00:09:24Z

    * `[HOOK - conan-center.py] post_package_info(): ERROR: [CMAKE FILE NOT IN BUILD FOLDERS (KB-H019)] The *.cmake files have to be placed in a folder declared as `cpp_info.builddirs`. Currently folders declared: ['lib\\cmake\\protobuf'] (https://github.com/conan-io/conan-center-index/wiki/Error-Knowledge-Base#KB-H019)`
    * `[HOOK - conan-center.py] post_package_info(): ERROR: [CMAKE FILE NOT IN BUILD FOLDERS (KB-H019)] Found files:`

@uilianries On Windows, the hook should normalize windows paths to cmake paths ("\\" -> "/")

conan-center-bot · 2020-03-25T00:28:12Z

Some configurations of 'protobuf/3.9.1.0' failed in build 2 (7b7b66d460066f522c12abc622f9216d82c130ad):

Macos x86_64, Debug, apple-clang 9.1, libc++ . Options: protobuf:shared-True
Access to all the logs
Windows x86_64, Debug, Visual Studio 14, MTd. Options: protobuf:shared-False
- Hooks errors detected:
  - [HOOK - conan-center.py] post_package_info(): ERROR: [CMAKE FILE NOT IN BUILD FOLDERS (KB-H019)] The *.cmake files have to be placed in a folder declared as cpp_info.builddirs. Currently folders declared: ['lib\\cmake\\protobuf'] (https://github.com/conan-io/conan-center-index/wiki/Error-Knowledge-Base#KB-H019)
  - [HOOK - conan-center.py] post_package_info(): ERROR: [CMAKE FILE NOT IN BUILD FOLDERS (KB-H019)] Found files:
- Log download
- Log error Download
- Profile
Access to all the logs

madebr · 2020-03-25T00:29:54Z

@jgsogo This recipe provides some libraries (libprotobuf/libprotoc/..) and an executable (protoc).
So this package can be used as a build requirement and a ordinary requirement.
In package_info I add the bin folder to the PATH environment variable.

Won't this cause problems when cross building (and adding it in both requirements and build_requirements)? Because PATH might be appended by a directory, containing executables for the host architecture..

conan-center-bot · 2020-03-25T00:39:31Z

Some configurations of 'protobuf/3.9.1.0' failed in build 3 (8dcff5b4b209f241499a32fdc1cecb43c4d8338c):

Linux x86_64, Release, gcc 5, libstdc++ . Options: protobuf:shared-True
- Hooks errors detected:
  - [HOOK - conan-center.py] pre_export(): ERROR: [TEST PACKAGE - RUN ENVIRONMENT (KB-H029)] The 'RunEnvironment()' build helper is no longer needed. It has been integrated into the self.run(..., run_environment=True) (https://github.com/conan-io/conan-center-index/wiki/Error-Knowledge-Base#KB-H029)
- Log download
- Log error Download
- Profile
Access to all the logs

madebr · 2020-03-25T00:42:01Z

    * `[HOOK - conan-center.py] pre_export(): ERROR: [TEST PACKAGE - RUN ENVIRONMENT (KB-H029)] The 'RunEnvironment()' build helper is no longer needed. It has been integrated into the self.run(..., run_environment=True) (https://github.com/conan-io/conan-center-index/wiki/Error-Knowledge-Base#KB-H029) `

@uilianries

The RunEnvironment helper is needed in the build() step of the cmake script.
So either:

the cmake helper needs to be improved (as does AutoToolsBuildEnvironment and Meson); or
the hook is too stringent.

This is the same problem as @ericLemanissier was seeing while packaging libsndfile.
See #1103 (comment)

Croydon · 2020-03-25T01:15:12Z

Maybe the hook should check if RunEnvironment is used inside of test() because there should be no reasons why it should be used there? 🤔

recipes/protobuf/3.9.x/conanfile.py

jgsogo · 2020-03-25T13:46:54Z

@jgsogo This recipe provides some libraries (libprotobuf/libprotoc/..) and an executable (protoc).
So this package can be used as a build requirement and a ordinary requirement.
In package_info I add the bin folder to the PATH environment variable.

Won't this cause problems when cross building (and adding it in both requirements and build_requirements)? Because PATH might be appended by a directory, containing executables for the host architecture..

Have a look to the implementation of the feature in the sources: https://github.com/jgsogo/conan/blob/4b977e322c308341003007b5641aadc3df36383b/conans/client/installer.py#L503

Recipes in the host context (requirements and build_requires forced to host) propagate only the cpp_info to deps_cpp_info
```
conan_file.deps_cpp_info.update(n.conanfile.cpp_info, n.ref.name)
```

Regular build requirements (those will be in the build context) propagate the env_info and the cpp_info to the deps_env_info:

env_info = EnvInfo()
env_info._values_ = n.conanfile.env_info._values_.copy()
# Add cpp_info.bin_paths/lib_paths to env_info (it is needed for runtime)
env_info.DYLD_LIBRARY_PATH.extend(n.conanfile.cpp_info.lib_paths)
env_info.DYLD_LIBRARY_PATH.extend(n.conanfile.cpp_info.framework_paths)
env_info.LD_LIBRARY_PATH.extend(n.conanfile.cpp_info.lib_paths)
env_info.PATH.extend(n.conanfile.cpp_info.bin_paths)
conan_file.deps_env_info.update(env_info, n.ref.name)

This is not the same as it was being done before. This is not breaking because using two profiles is an opt-in, but we know this can break some recipes or workflows, we really wan't to know how people is using the propagation in the projects and try to improve it for Conan 2.0. We might change it if needed based on your feedback.

recipes/protobuf/3.9.x/conanfile.py

recipes/protobuf/3.9.x/conandata.yml

madebr · 2020-03-25T14:30:25Z

Have a look to the implementation of the feature in the sources: https://github.com/jgsogo/conan/blob/4b977e322c308341003007b5641aadc3df36383b/conans/client/installer.py#L503

  ```python
  env_info = EnvInfo()
  env_info._values_ = n.conanfile.env_info._values_.copy()
  # Add cpp_info.bin_paths/lib_paths to env_info (it is needed for runtime)
  env_info.DYLD_LIBRARY_PATH.extend(n.conanfile.cpp_info.lib_paths)
  env_info.DYLD_LIBRARY_PATH.extend(n.conanfile.cpp_info.framework_paths)
  env_info.LD_LIBRARY_PATH.extend(n.conanfile.cpp_info.lib_paths)
  env_info.PATH.extend(n.conanfile.cpp_info.bin_paths)
  conan_file.deps_env_info.update(env_info, n.ref.name)
  ```

@jgsogo
Thanks for the reference code.
Adding these extra fields to cpp_infoseems like a good idea.
Can a recipe detect in its package_info in what context it is used? (build/host context)

uilianries · 2020-03-25T14:31:02Z

The RunEnvironment helper is needed in the build() step of the cmake script.
So either:
* the cmake helper needs to be improved (as does `AutoToolsBuildEnvironment` and `Meson`); or

* the hook is too stringent.
This is the same problem as @ericLemanissier was seeing while packaging libsndfile.
See #1103 (comment)

I'll push a new fix for hooks, thanks for reporting.

jgsogo · 2020-03-25T14:47:16Z

@madebr

Adding these extra fields to cpp_info seems like a good idea.

We are not adding extra to cpp_info, with the two-profiles approach we are propagating (for the build requires in the build context) the cpp_info and the env_info to the deps_env_info, and before entering the build() method of the consumers all those variables will populate the environment: DYLD_..., LD_..., PATH_....

But, the deps_cpp_info in the consumers will contain less information.

Can a recipe detect in its package_info in what context it is used? (build/host context)

Right now (v1.24) it can't. This is something we need to investigate and implement for the next releases.

conan-center-bot · 2020-03-28T00:32:47Z

Some configurations of 'protobuf/3.9.1' failed in build 4 (404bde744641c80f9002bd8ff2ed836fa343130f):

Linux x86_64, Release, gcc 5, libstdc++ . Options: protobuf:shared-True
- Hooks errors detected:
  - [HOOK - conan-center.py] pre_export(): ERROR: [TEST PACKAGE - RUN ENVIRONMENT (KB-H029)] The 'RunEnvironment()' build helper is no longer needed. It has been integrated into the self.run(..., run_environment=True) (https://github.com/conan-io/conan-center-index/wiki/Error-Knowledge-Base#KB-H029)
- Log download
- Log error Download
- Profile
Access to all the logs

madebr · 2020-04-27T14:37:14Z

@uilianries Please re-review.
I've added protoc support to protobuf/3.11.4 as well and force rebased on top of current master.

conan-center-bot · 2020-04-27T14:44:09Z

Some configurations of 'protobuf/3.9.1' failed in build 5 (246a00050b410e989b61f4956a4a2d91244821d1):

Linux x86_64, Release, gcc 5, libstdc++ . Options: protobuf:shared-True
- Hooks errors detected:
  - [HOOK - conan-center.py] pre_export(): ERROR: [TEST PACKAGE - RUN ENVIRONMENT (KB-H029)] The 'RunEnvironment()' build helper is no longer needed. It has been integrated into the self.run(..., run_environment=True) (https://github.com/conan-io/conan-center-index/wiki/Error-Knowledge-Base#KB-H029)
- Log download
- Log error Download
- Profile
Access to all the logs

conan-center-bot · 2020-04-28T14:26:47Z

Some configurations of 'protobuf/3.9.1' failed in build 6 (246a00050b410e989b61f4956a4a2d91244821d1):

Macos x86_64, Debug, apple-clang 9.1, libc++ . Options: protobuf:shared-True
Access to all the logs

jgsogo · 2020-04-29T17:18:06Z

For this add_custom_command we need to modify the .cmake file, it is the same when running tests using cmake --build ... --target test (conan-io/examples#17 (review)). From outside CMake (where Conan lives) we cannot add anything to the environment, it won't be used in the subprocess.

madebr · 2020-04-29T17:47:01Z

For this add_custom_command we need to modify the .cmake file, it is the same when running tests using cmake --build ... --target test (conan-io/examples#17 (review)). From outside CMake (where Conan lives) we cannot add anything to the environment, it won't be used in the subprocess.

Ok, I'll patch the cmake script.

But the RunEnvironment will be needed anyhow for linux (LD_LIBRARY_PATH)

The original code used the cmake imported target protobuf::protoc.
If the new conan components feature would be able to generate this executable target, it can also be used.
Would this be another way to solve this?

madebr · 2020-04-29T18:05:21Z

For the cmake_find_package_multi generator, I think we need to have the INTERFACE_LINK_DIRECTORIES property of the libraries populated.
Then we can use generator expressions to extract the property of the correct build type.

Now I would need to add ${Protobuf_LIB_DIRS_RELEASE}:${Protobuf_LIB_DIRS_DEBUG}:${Protobuf_LIB_DIRS_RELWITHDEBINFO}:${Protobuf_LIB_DIRS_MINSIZEREL}.
And then, this doesn't guarantee success if a user added an extra build type to its settings.yml.

conan-center-bot · 2020-04-29T19:49:17Z

All green in build 15 (e688f64750ec13f0553f95b3d39b2d4640b94a36)! 😊

protobuf/3.9.1: Generated 130 packages. All logs here
protobuf/3.11.4: Generated 130 packages. All logs here

conan-center-bot · 2020-06-18T13:17:30Z

All green in build 16 (e688f64750ec13f0553f95b3d39b2d4640b94a36)! 😊

protobuf/3.9.1: Generated 130 packages. All logs here
protobuf/3.11.4: Generated 130 packages. All logs here

Croydon · 2020-06-23T10:18:09Z

I don't understand if there are any issues left, but if they aren't breaking and can't be fixed quickly I would suggest to create an issue to fix it later and finally merge this

This becomes quite a blocker IMO

uilianries · 2020-06-23T19:31:03Z

@Croydon Agreed, Protobuf is one of most required packages

jgsogo

Agree, we can move this one forward and wait for issues.

madebr · 2020-06-24T14:13:38Z

Agree, we can move this one forward and wait for issues.

I love your optimism 😄

jgsogo · 2020-06-24T14:18:51Z

Agree, we can move this one forward and wait for issues.

I love your optimism 😄

🤣 well, I didn't mean there will be issues.... but, well, we all know there will be. I'll spend some days now experiencing with the cross-building feature, env propagation, settings,... so I'll have time to try this recipe and open the issues myself 😄

Croydon · 2020-06-24T15:41:06Z

Thanks everyone! 🥳

This was referenced Mar 24, 2020

Add arrow/1.0.0 recipe #1182

Merged

Add Protobuf protoc 3.9.1 #65

Closed

conan-center-bot added the Failed label Mar 24, 2020

uilianries reviewed Mar 25, 2020

View reviewed changes

recipes/protobuf/3.9.x/conanfile.py Outdated Show resolved Hide resolved

uilianries reviewed Mar 25, 2020

View reviewed changes

recipes/protobuf/3.9.x/conanfile.py Outdated Show resolved Hide resolved

uilianries reviewed Mar 25, 2020

View reviewed changes

recipes/protobuf/3.9.x/conandata.yml Outdated Show resolved Hide resolved

uilianries mentioned this pull request Mar 25, 2020

Allow RunEnvironment for test_package conan-io/hooks#174

Merged

This was referenced Mar 27, 2020

add bison/3.5.3 #1196

Merged

add flex/2.6.4 #1195

Merged

uilianries mentioned this pull request Apr 2, 2020

Added version 3.11.4 #1199

Merged

4 tasks

This was referenced Apr 25, 2020

Fix build options bincrafters/conan-protobuf#42

Closed

[protobuf/3.9.1] Can't build package with option protobuf:lite=True bincrafters/community#1008

Closed

madebr added 2 commits April 27, 2020 14:51

protobuf: cache cmake object

36ec236

protobuf: add protoc

246a000

madebr force-pushed the protobuf_protoc branch from 404bde7 to 246a000 Compare April 27, 2020 14:36

madebr mentioned this pull request Apr 29, 2020

[feature] CMake generator: custom target conan-io/conan#6069

Closed

1 task

protobuf: treat macos special

e688f64

conan-center-bot removed the Failed label Apr 29, 2020

This was referenced May 4, 2020

I have packaged current version 1.28.1 drawing inspiration from this recipe inexorgame-obsolete/conan-grpc#37

Closed

protobuf / protoc_installer / 3.9.1: build_modules errors bincrafters/community#1189

Closed

0x8000-0000 mentioned this pull request May 18, 2020

[question] Crosscompiling protobuf #1658

Closed

jgsogo mentioned this pull request May 26, 2020

[feature] Symlink Imports conan-io/conan#7030

Closed

1 task

uilianries mentioned this pull request Jun 18, 2020

[package] protobuf/any: access to protobuf_generate_cpp function #1956

Open

Croydon mentioned this pull request Jun 22, 2020

Update to protobuf 3.11.4 version bincrafters/conan-protobuf#43

Closed

madebr mentioned this pull request Jun 23, 2020

[feature] Add executables to cpp_info model conan-io/conan#7240

Open

uilianries approved these changes Jun 23, 2020

View reviewed changes

uilianries requested review from danimtb, SSE4 and jgsogo June 23, 2020 19:31

jgsogo approved these changes Jun 24, 2020

View reviewed changes

danimtb approved these changes Jun 24, 2020

View reviewed changes

SSE4 approved these changes Jun 24, 2020

View reviewed changes

SSE4 merged commit 252ad9a into conan-io:master Jun 24, 2020

madebr deleted the protobuf_protoc branch June 24, 2020 14:21

madebr mentioned this pull request Jul 5, 2020

Feature/2127 add flatcc #2129

Merged

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

protobuf: add protoc option #1179

protobuf: add protoc option #1179

madebr commented Mar 24, 2020 •

edited by uilianries

Loading

conan-center-bot commented Mar 24, 2020

madebr commented Mar 25, 2020

conan-center-bot commented Mar 25, 2020

madebr commented Mar 25, 2020 •

edited

Loading

conan-center-bot commented Mar 25, 2020

madebr commented Mar 25, 2020 •

edited

Loading

Croydon commented Mar 25, 2020 •

edited

Loading

jgsogo commented Mar 25, 2020

madebr commented Mar 25, 2020

uilianries commented Mar 25, 2020

jgsogo commented Mar 25, 2020

conan-center-bot commented Mar 28, 2020

madebr commented Apr 27, 2020 •

edited

Loading

conan-center-bot commented Apr 27, 2020

conan-center-bot commented Apr 28, 2020

jgsogo commented Apr 29, 2020

madebr commented Apr 29, 2020 •

edited

Loading

madebr commented Apr 29, 2020

conan-center-bot commented Apr 29, 2020

conan-center-bot commented Jun 18, 2020

Croydon commented Jun 23, 2020 •

edited

Loading

uilianries commented Jun 23, 2020

jgsogo left a comment

madebr commented Jun 24, 2020

jgsogo commented Jun 24, 2020

Croydon commented Jun 24, 2020

protobuf: add protoc option #1179

protobuf: add protoc option #1179

Conversation

madebr commented Mar 24, 2020 • edited by uilianries Loading

conan-center-bot commented Mar 24, 2020

madebr commented Mar 25, 2020

conan-center-bot commented Mar 25, 2020

madebr commented Mar 25, 2020 • edited Loading

conan-center-bot commented Mar 25, 2020

madebr commented Mar 25, 2020 • edited Loading

Croydon commented Mar 25, 2020 • edited Loading

jgsogo commented Mar 25, 2020

madebr commented Mar 25, 2020

uilianries commented Mar 25, 2020

jgsogo commented Mar 25, 2020

conan-center-bot commented Mar 28, 2020

madebr commented Apr 27, 2020 • edited Loading

conan-center-bot commented Apr 27, 2020

conan-center-bot commented Apr 28, 2020

jgsogo commented Apr 29, 2020

madebr commented Apr 29, 2020 • edited Loading

madebr commented Apr 29, 2020

conan-center-bot commented Apr 29, 2020

conan-center-bot commented Jun 18, 2020

Croydon commented Jun 23, 2020 • edited Loading

uilianries commented Jun 23, 2020

jgsogo left a comment

Choose a reason for hiding this comment

madebr commented Jun 24, 2020

jgsogo commented Jun 24, 2020

Croydon commented Jun 24, 2020

madebr commented Mar 24, 2020 •

edited by uilianries

Loading

madebr commented Mar 25, 2020 •

edited

Loading

madebr commented Mar 25, 2020 •

edited

Loading

Croydon commented Mar 25, 2020 •

edited

Loading

madebr commented Apr 27, 2020 •

edited

Loading

madebr commented Apr 29, 2020 •

edited

Loading

Croydon commented Jun 23, 2020 •

edited

Loading