Feature: Enable bazel >= 7.1 #16196

Neeeflix · 2024-05-03T05:31:20Z

Feature: Add support for Bazel >= 7.1. This is also requested in #15754.

This will not break older bazel versions, as just some additional files are generated.

I added some of those file based test-cases, and did some "real" integration tests locally. If you know any better better way to put this into the conan test-environment, please let me know.

It's "only" supporting versions from 7.1 on, as there is a feature being used which is not available in 7.0. This is also documented in the code.

Docs: conan-io/docs#3707
Example: conan-io/examples2#147

Refer to the issue that supports this Pull Request.
If the issue has missing info, explain the purpose/use case/pain/need that covers this Pull Request.
I've read the Contributing guide.
I've followed the PEP8 style guides for Python code.
I've opened another PR in the Conan docs repo to the develop branch, documenting this one.

Neeeflix · 2024-05-03T09:13:57Z

@memsharded thanks for adding the milestone and looking into this.
Would it also be possible to add a milestone for 1.X, as for us it will not be easy to go with 2.X in the near future.

Any work that needs to be done here, could be taken over by me. I think there are some smaller differences between the 1.X generator and the 2.X one

memsharded · 2024-05-03T09:50:06Z

Would it also be possible to add a milestone for 1.X, as for us it will not be easy to go with 2.X in the near future.

This is not something that we can keep doing much longer, for new features like this one we'd strongly favor Conan 2 only. It was released more than 14 months ago, ConanCenter will soon stop updating packages for Conan 1.X too. We strongly recommend to prioritize as much as possible the update to Conan 2.

In this case, if you'd like to get this in Conan 1, and you are willing to do the effort, then we might want to wait until we get @franramirez688 feedback about this.
One of the aspects is that we should be 100% sure that it is not breaking Conan 1.X users, if there is any minimal risk (and I am a bit concerned about the proposed changes could no work for users using older versions of Bazel). @franramirez688 is more aware of the possible issues and efforts related to this.

Neeeflix · 2024-05-03T10:07:55Z

This is not something that we can keep doing much longer, for new features like this one we'd strongly favor Conan 2 only. It was released more than 14 months ago, ConanCenter will soon stop updating packages for Conan 1.X too. We strongly recommend to prioritize as much as possible the update to Conan 2.

I understand the urgency and importance of updating to Conan 2, especially with the upcoming discontinuation of updates for Conan 1.X. However, the process presents significant challenges, especially when depending on teams in different organizations, etc.

One of the aspects is that we should be 100% sure that it is not breaking Conan 1.X users,

As mentioned in the PR description, the files which are generated for the older bazel version are not touched.
There are only two additional files being generated, there is no interference between those files at all.

franramirez688 · 2024-05-07T15:09:33Z

Hi @Neeeflix

Thanks a lot for this PR! 👏
I was trying to test it locally but I got stuck with this error:

INFO: Starting clean (this may take a while). Consider using --async if the clean takes more than several minutes.

hello/0.1: RUN: bazel --bazelrc=/Users/franchuti/.conan2/p/b/hellodfdcbdac82faa/b/conan/conan_bzl.rc build --config=conan-config //main:hello
WARNING: --enable_bzlmod is set, but no MODULE.bazel file was found at the workspace root. Bazel will create an empty MODULE.bazel file. Please consider migrating your external dependencies from WORKSPACE to MODULE.bazel. For more details, please refer to https://github.com/bazelbuild/bazel/issues/18958.
ERROR: Error computing the main repository mapping: in module dependency chain <root> -> bazel_tools@_ -> rules_python@0.22.1: Error accessing registry https://bcr.bazel.build/: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
Computing main repo mapping:

hello/0.1: ERROR:
Package '015c501fa3ba2da21e4c2e71becb7b76fa605a36' build failed
hello/0.1: WARN: Build folder /Users/franchuti/.conan2/p/b/hellodfdcbdac82faa/b

I do not know how to solve this. I just tried a couple of things that I found here:

Nothing worked 😭 Do you know how I can get it running OK locally?

Neeeflix · 2024-05-08T09:50:29Z

Hey @franramirez688,
Thanks for looking into this

Regarding the your issue:

INFO: Starting clean (this may take a while). Consider using --async if the clean takes more than several minutes.

hello/0.1: RUN: bazel --bazelrc=/Users/franchuti/.conan2/p/b/hellodfdcbdac82faa/b/conan/conan_bzl.rc build --config=conan-config //main:hello
WARNING: --enable_bzlmod is set, but no MODULE.bazel file was found at the workspace root. Bazel will create an empty MODULE.bazel file. Please consider migrating your external dependencies from WORKSPACE to MODULE.bazel. For more details, please refer to https://github.com/bazelbuild/bazel/issues/18958.
ERROR: Error computing the main repository mapping: in module dependency chain <root> -> bazel_tools@_ -> rules_python@0.22.1: Error accessing registry https://bcr.bazel.build/: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
Computing main repo mapping:

hello/0.1: ERROR:
Package '015c501fa3ba2da21e4c2e71becb7b76fa605a36' build failed
hello/0.1: WARN: Build folder /Users/franchuti/.conan2/p/b/hellodfdcbdac82faa/b

Can you may give some more details about your setup. Which example did you use?
I also updated the example in: conan-io/examples2#147, can you try that?
Also noticed that I only tested there for conan 1.X. Updated it, now also conan 2.X works for me.

Reproduction steps:

Install conan from this branch
CD to the example repo and checkout my branch.
In examples/tools/google/bazel_7.1/testsuite I run conan build . -of conan

franramirez688 · 2024-05-08T17:43:21Z

Hi @Neeeflix

Thanks a lot for asking and for your help. Finally, after several hours of trying crazy things, I found the solution to my problem. I'm using Zscaler, so the solution came up with importing through keytool the Zscaler root certificate 👏
Now, I can start reviewing everything 🥳 Sorry for the long delay in keeping my environment up-to-date.

franramirez688

@Neeeflix

Finally, I was able to try this. Thanks again for the PR to support Bazel >= 7.1 versions and the huge work behind this. It looks promising!

It's working locally in macOS and Win 👏 , but I'm having some problems with the certificates & Java again in Linux (I hope I get it ready ASAP).

Meanwhile, let me know more about this approach. Sorry for my dummy questions but I want to understand everything 😁

conan/tools/google/bazeldeps.py

https://bazel.build/external/overview#canonical-repo-name

…/integration tests. Conftest with bazel 6.x and 7.x

franramirez688

Hi @Neeeflix

Sorry for the long delay and for forcing the push (I just messed up with the latest commits in dev)! 😭

Thanks again for the contribution! We are almost there 😁

I just added some lines to test it with Bazel 7.1.2, so let's see if it passes with both versions.

franramirez688 · 2024-05-30T07:19:25Z

UPDATE: CI is complaining about Linux and Bazel 7.x (not installed yet). Working on it 😁

conan/tools/google/bazeldeps.py

test/functional/toolchains/google/test_bazel.py

Co-authored-by: James <memsharded@gmail.com>

fmeum · 2024-05-30T19:49:59Z

conan/tools/google/bazeldeps.py

+        {% endfor %}
+
+            return mctx.extension_metadata(
+                root_module_direct_deps = None,


If you set this to "all", Bazel will show a warning if the user doesn't use_repo all repos defined by this extension and bazel mod tidy updates use_repo automatically. If I understood the logic correctly, that's exactly what you want here.

@fmeum thanks for the explanation! Then, it makes total sense 👍

Not sure, if it really makes sense. Some of the dependencies listed in the conanfile, you might not want to use in the bazel-build, or?

If there is a way to detect which direct dependencies should be used, those could also be returned as a list here. Could also be a follow-up though.

For me, the suggestion of "all" would be a great usability improvement. I am sure when developers add something to conanfile they will forget to add it to the MODULE.bazel. The warning and possible automation would help a lot.

For us, we will only have things in conanfile that we want in the build; I'm not sure why you would list something there and then not use it?

Yeah, agree.
I was just thinking about generator packages or compiler-packages, for me it is still not really clear how this will work.

Yeah, I got what you meant @Neeeflix Let me check it locally and see how it behaves 😁

I was running some tests, and it's indeed useful to see something like this:

WARNING: /Users/franchuti/develop/conan/bazel/test_package/MODULE.bazel:1:40: The module extension conan_extension defined in @//conan:conan_deps_module_extension.bzl reported incorrect imports of repositories via use_repo(): Not imported, but reported as direct dependencies by the extension (may cause the build to fail): build-cmake, zlib Fix the use_repo calls by running 'bazel mod tidy'.

I did not find any side effects. IMO, let's give it a try. Thanks to everyone 😁

peakschris

Thanks for a great PR, this is a big help. Only one of my comments is a blocker, the @@rules_cc one.

The other issue I am facing is that zlib conan module has some missing items in its build file, but I suspect this is an issue with zlib recipe:

Missing:

cc_import(
    name = "zlib_precompiled",
    shared_library = "bin/zlib1.dll",
    interface_library = "lib/zdll.lib",
)
deps statement in cc_library(name = "zlib"... referencing this

conan/tools/google/bazeldeps.py

peakschris · 2024-05-31T07:58:27Z

conan/tools/google/bazeldeps.py

+        In case of bazel >= 7.1, the conan_deps_module_extension.bzl file should be loaded by your
+        Module.bazel file, e.g. like this:
+
+        .. code-block:: python


It would also be nice to show an example using an include MODULE.bazel file

I added an example in the documentation repository

Besides that, you can run conan new bazel_lib_7 -d name=hello -d version=1.0 to see how it works (using Conan from source and this git branch for sure).

I mean, there is an alternative using the include directive, that helps to separate concerns and contention on the MODULE.bazel file, and it would be nice to see this option in docs:

MODULE.bazel:

include("//deps/conan:conan.MODULE.bazel")

deps/conan/conan.MODULE.bazel:

load_conan_dependencies = use_extension("//:conan_deps_module_extension.bzl", "load_dependencies") use_repo(load_conan_dependencies, "zlib")

@peakschris could you please add the URL to the documentation? I think it could be quite interesting what you're proposing 😁

The link can be found here: https://bazel.build/rules/lib/globals/module#include

But somehow I can't get it running…

ERROR: /home/[…]MODULE.bazel:1:1: name 'include' is not defined ERROR: Error computing the main repository mapping: error executing MODULE.bazel file for <root>

It's only in 7.2.0rc1 and higher.

Then, it is better to wait for a more stable version.

peakschris · 2024-05-31T08:07:55Z

conan/tools/google/bazeldeps.py

+                # Important for remote build. Actually it's not reproducible, as local paths will
+                # be different on different machines. But we assume that conan works correctly here.
+                # IMPORTANT: Not compatible with bazel < 7.1
+                reproducible = True,


One oddity I noticed is that if I manually edit one of Conan's BUILD.bazel files after an initial build, the edited version does not get picked up, unless I delete MODULE.bazel.lock. If this isn't fixable without impacting performance, then maybe some documentation could be added to explain what to do.

That's surprising because the repo rule below symlinks in the build file, so changes should be picked up no matter what Bazel does. Can you reproduce this with --nosystem_rc --nohome_rc?

--nosystem_rc --nohome_rc don't make a difference. The issue is that by default, symlinks are disabled for windows, and I was using the defaults in my hello world workspace. I think in this case bazel's symlinks function reverts to a copy. If I add --windows_enable_symlinks then build file changes are picked up. This could be addressed with documentation.

Ah, I see. That can be fixed like this:
https://github.com/bazelbuild/bazel/blob/fb171884e10787475b2936607e8c92717f7ff474/tools/build_defs/repo/local.bzl#L99

Could the watch be done only if symlinks are not enabled?

That's difficult to detect (would require checking the file type after the operation), but unnecessarily watching a few build files shouldn't matter that much.

@Wyverald What do you think, should we watch automatically if the symlink is a copy?

The automatic watch sounds like a great idea to me

Can this be done in a follow up?

This was referenced May 3, 2024

Feature: add bazel 7.1 example with gtest conan-io/examples2#147

Open

feat: Add bazel 7.1 documentation conan-io/docs#3707

Open

memsharded added this to the 2.4.0 milestone May 3, 2024

memsharded assigned franramirez688 May 3, 2024

Neeeflix force-pushed the feature/enable-bazel-7.1 branch 2 times, most recently from 605d489 to c0d8468 Compare May 7, 2024 10:47

Neeeflix force-pushed the feature/enable-bazel-7.1 branch from c0d8468 to 11171e9 Compare May 8, 2024 11:12

franramirez688 reviewed May 14, 2024

View reviewed changes

conan/tools/google/bazeldeps.py Show resolved Hide resolved

conan/tools/google/bazeldeps.py Outdated Show resolved Hide resolved

Neeeflix and others added 8 commits May 29, 2024 16:23

feat: Rename template to repository template

d0aea5c

feat: Make rule_cc a cannonical repository

3396189

https://bazel.build/external/overview#canonical-repo-name

feat: Add conan_deps_repo_rules to generate repos

093217d

feat: Generate module file to be used by consumers

25f21c8

feat: Add appropriate documentation for consumers

802848f

feat: Add content based test for conan_deps_module_extension

bc164f4

Added Bazel 7.x templates. Minor BazelDeps changes. Adding functional…

cbefeaf

…/integration tests. Conftest with bazel 6.x and 7.x

Added both versions to CI+

38ce62f

franramirez688 force-pushed the feature/enable-bazel-7.1 branch from 90938d4 to 38ce62f Compare May 29, 2024 14:23

franramirez688 reviewed May 29, 2024

View reviewed changes

Not version as kwarg

27d9896

Added Linux CI bazel folders

5163c77

franramirez688 requested a review from czoido May 30, 2024 10:38

docstrings

42fdd12

franramirez688 assigned czoido May 30, 2024

franramirez688 approved these changes May 30, 2024

View reviewed changes

franramirez688 removed their assignment May 30, 2024

memsharded approved these changes May 30, 2024

View reviewed changes

conan/tools/google/bazeldeps.py Outdated Show resolved Hide resolved

test/functional/toolchains/google/test_bazel.py Outdated Show resolved Hide resolved

franramirez688 and others added 2 commits May 30, 2024 18:02

Update conan/tools/google/bazeldeps.py

9d65a40

Co-authored-by: James <memsharded@gmail.com>

Typos

b911371

memsharded mentioned this pull request May 30, 2024

[question] Compatibility with bazel 7.0 #15363

Open

1 task

fmeum reviewed May 30, 2024

View reviewed changes

Merge branch 'develop2' into feature/enable-bazel-7.1

6248415

peakschris suggested changes May 31, 2024

View reviewed changes

peakschris reviewed May 31, 2024

View reviewed changes

franramirez688 self-requested a review May 31, 2024 12:03

franramirez688 self-assigned this May 31, 2024

Removed rules loading

9b28239

franramirez688 force-pushed the feature/enable-bazel-7.1 branch from 39b6538 to 9b28239 Compare May 31, 2024 12:16

franramirez688 added 2 commits May 31, 2024 15:17

Using less common word

4e9e396

Using root_module_direct_deps = 'all'

e53e2bf

franramirez688 approved these changes Jun 3, 2024

View reviewed changes

Feature: Enable bazel >= 7.1 #16196

Are you sure you want to change the base?

Feature: Enable bazel >= 7.1 #16196

Conversation

Neeeflix commented May 3, 2024 • edited

Neeeflix commented May 3, 2024 • edited

memsharded commented May 3, 2024

Neeeflix commented May 3, 2024

franramirez688 commented May 7, 2024

Neeeflix commented May 8, 2024

franramirez688 commented May 8, 2024

franramirez688 left a comment

Choose a reason for hiding this comment

franramirez688 left a comment

Choose a reason for hiding this comment

franramirez688 commented May 30, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

peakschris left a comment

Choose a reason for hiding this comment

peakschris May 31, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Neeeflix commented May 3, 2024 •

edited

Neeeflix commented May 3, 2024 •

edited

peakschris May 31, 2024 •

edited