Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

First implementation of the Bazel toolchain #8991

Merged
merged 8 commits into from May 30, 2021
3 changes: 3 additions & 0 deletions conan/tools/google/__init__.py
@@ -0,0 +1,3 @@
from conan.tools.google.toolchain import BazelToolchain
from conan.tools.google.bazeldeps import BazelDeps
from conan.tools.google.bazel import Bazel
49 changes: 49 additions & 0 deletions conan/tools/google/bazel.py
@@ -0,0 +1,49 @@
import os
import json

from conan.tools import CONAN_TOOLCHAIN_ARGS_FILE
from conans.util.files import load

class Bazel(object):
def __init__(self, conanfile):
self._conanfile = conanfile
self._get_bazel_project_configuration()

def configure(self, args=None):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If there is no configure step for bazel you can nicely remove it

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As I said to @memsharded in the other PR (I had to close it due to merge complications):

As far as I've seen, the convention when writing the build() method on a project's conanfile.py is:

def build(self):
    bazel = Bazel(self)
    bazel.configure()
    bazel.build(whatever)

Wouldn't deleting the configure method break this pattern?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not necessarily, the MSBuild helper does not have configure() step. It can be removed.

pass

def build(self, args=None, label=None):
# TODO: Change the directory where bazel builds the project (by default, /var/tmp/_bazel_<username> )
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

About the TODO. Why?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's @memsharded's suggestion. He said that the project should be built in the conan's cache. Shouldn't it?

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure that the concept "the directory where bazel builds the project" makes sense, as much as 'the directory of the Bazel project'

bazel must be run from somewhere in the source tree under a WORKSPACE file. It executes each build action in a symlink tree created for the action and places results in bazel-out in trees under bazel-out and bazel-bin.

After building a particular label or labels, they will appear somewhere under bazel-bin. From there, you might want to copy them out to conan's cache. It's hard to make a particular suggestion without seeing an example of how all the intermediate files should fit together. For example, what would you expect for a configure based project that depends on libfoo where libfoo is built by Bazel.

You obviously need to have the glue that says

  • for target X
  • which produces an output file that you can't easily name (libfoo.a, libfoo.so, libfoo.lib, ...)
  • tell configure how to use that library

Somehow we need to be able to tell configure not to used the installed libfoo. It needs to use the file under bazel-bin. At least, that is what I presume. I don't know the conan workflow enough to know.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @aiuto, thanks for the feedback!

Let me briefly clarify the different flows:

For the consumer point of view: a project using bazel to build that wants to use Conan existing packages as dependencies:

  • This seems to be the most requested use case
  • The BazelDeps Conan generator can output files that Bazel could understand with cc_import, etc instructions to let Bazel locate those Conan packages in the Conan cache. Those packages could have been built with other build system, not a problem, Conan abstracts the package information and can translate it to different build systems.

For the producer point of view, a Conan package which build system is Bazel.

  • This is the case this Bazel Conan helper tries to implement.
  • Conan packages when building from source, are built in the Conan cache, typically in .conan user folder, to not pollute the user projects, and to be able to reuse existing packages among projects.
  • When building from source, Conan also contains a copy of the sources in such ".conan" cache, which might have been retrieved from SCM, for example.
  • For all the other build systems, the build is done in a path that looks like .conan/data/mypkg/1.0/user/channel/build/binary_id/. All the binaries and temporary files are there, and when Conan wants to do the final package, will copy selected artifacts to something like .conan/data/mypkg/1.0/user/channel/package/binary_id/

It would be good if Bazel could be instructed to put the temporary build files in that location, instead of somewhere in bazel-out and bazel-bin, that if I understood correctly, might live in /var/tmp/_bazel by default?

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's look at the consumer view.
I think a nicely transparant integration is one that points to the conan packages and then creates a local bazel repository that containds a BUILD file with the cc_import rules directly, so that the we can refer to them directly.
So the user experience is this:

WORKSPACE

workspace(name = 'myproject')
conan_repository(
  name = "foo", 
   artifacts=['a, 'b', .. list of top build targets from the conan package...]
)

BUILD

cc_library(
  name = 'myapp',
  deps = ['@foo//:a']
  ...
)

and the glue generated by the conan_repository rule is

WORKSPACE

workspace(name=foo)

BUILD

cc_import(
  name = 'a',
  hdrs = [exported headers of the artifact a],
  static_library = "libmylib.a",
)

libmylib.a: symlink to libmylib.a built as a result of doing the conan build.

Copy link
Contributor Author

@tapia tapia May 31, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The merged generator is pretty similar to your idea. The dependencies are declared in the conanfile.py, though, not in the WORKSPACE file. For example:

class BazelExampleConan(ConanFile):
    name = "bazel-example"
    ....
    build_requires = "boost/1.76.0"

Then, the only thing you have to add to the WORKSPACE file is this:

load("@//conandeps:dependencies.bzl", "load_conan_dependencies")
load_conan_dependencies()

After that, just update the BUILD files where you need to use the new dependency:

cc_binary(
    name = "hello-world",
    srcs = ["hello-world.cc"],
    deps = [
        "@boost//:boost",
    ],
)

Give it a try and let us know what you think :-)


bazelrc_path = '--bazelrc={}'.format(self._bazelrc_path) if self._bazelrc_path else ''
bazel_config = '--config={}'.format(self._bazel_config) if self._bazel_config else ''

# arch = self._conanfile.settings.get_safe("arch")
# cpu = {
# "armv8": "arm64",
# "x86_64": ""
# }.get(arch, arch)
#
# command = 'bazel {} build --sandbox_debug --subcommands=pretty_print --cpu={} {} {}'.format(
# bazelrc_path,
# cpu,
# bazel_config,
# label
# )

command = 'bazel {} build {} {}'.format(
bazelrc_path,
bazel_config,
label
)

self._conanfile.run(command)

def _get_bazel_project_configuration(self):
self._bazel_config = None
self._bazelrc_path = None

if os.path.exists(CONAN_TOOLCHAIN_ARGS_FILE):
conan_toolchain_args = json.loads(load(CONAN_TOOLCHAIN_ARGS_FILE))
self._bazel_config = conan_toolchain_args.get("bazel_config", None)
self._bazelrc_path = conan_toolchain_args.get("bazelrc_path", None)
122 changes: 122 additions & 0 deletions conan/tools/google/bazeldeps.py
@@ -0,0 +1,122 @@
import textwrap

from jinja2 import Template

from conans.util.files import save


class BazelDeps(object):
def __init__(self, conanfile):
self._conanfile = conanfile

def generate(self):
local_repositories = []
for dependency in self._conanfile.dependencies.transitive_host_requires:
content = self._get_dependency_buildfile_content(dependency)
filename = self._save_dependendy_buildfile(dependency, content)

local_repository = self._create_new_local_repository(dependency, filename)
local_repositories.append(local_repository)

content = self._get_main_buildfile_content(local_repositories)
self._save_main_buildfiles(content)

def _save_dependendy_buildfile(self, dependency, buildfile_content):
filename = 'conandeps/{}/BUILD'.format(dependency.ref.name)
save(filename, buildfile_content)
return filename

def _get_dependency_buildfile_content(self, dependency):
template = textwrap.dedent("""
load("@rules_cc//cc:defs.bzl", "cc_import", "cc_library")

{% for lib in libs %}
cc_import(
name = "{{ lib }}_precompiled",
static_library = "{{ libdir }}/lib{{ lib }}.a"
)
{% endfor %}

cc_library(
name = "{{ name }}",
{% if headers %}
hdrs = glob([{{ headers }}]),
{% endif %}
{% if includes %}
includes = [{{ includes }}],
{% endif %}
{% if defines %}
defines = [{{ defines }}],
{% endif %}
visibility = ["//visibility:public"]
)

""")

dependency.new_cpp_info.aggregate_components()

if not dependency.new_cpp_info.libs and not dependency.new_cpp_info.includedirs:
return None

headers = []
includes = []

for path in dependency.new_cpp_info.includedirs:
headers.append('"{}/**"'.format(path))
includes.append('"{}"'.format(path))

headers = ', '.join(headers)
includes = ', '.join(includes)

defines = ('"{}"'.format(define) for define in dependency.new_cpp_info.defines)
defines = ', '.join(defines)

context = {
"name": dependency.ref.name,
"libs": dependency.new_cpp_info.libs,
"libdir": dependency.new_cpp_info.libdirs[0],
"headers": headers,
"includes": includes,
"defines": defines
}

content = Template(template).render(**context)
return content

def _create_new_local_repository(self, dependency, dependency_buildfile_name):
snippet = textwrap.dedent("""
native.new_local_repository(
name="{}",
path="{}",
build_file="{}",
)
""").format(
dependency.ref.name,
dependency.package_folder,
dependency_buildfile_name
)

return snippet

def _get_main_buildfile_content(self, local_repositories):
template = textwrap.dedent("""
def load_conan_dependencies():
{}
""")

if local_repositories:
function_content = "\n".join(local_repositories)
function_content = ' '.join(line for line in function_content.splitlines(True))
else:
function_content = ' pass'

content = template.format(function_content)

return content

def _save_main_buildfiles(self, content):
# A BUILD file must exist, even if it's empty, in order for bazel
# to detect it as a bazel package and allow to load the .bzl files
save("conandeps/BUILD", "")

save("conandeps/dependencies.bzl", content)
19 changes: 19 additions & 0 deletions conan/tools/google/toolchain.py
@@ -0,0 +1,19 @@
import json

from conan.tools import CONAN_TOOLCHAIN_ARGS_FILE
from conans.util.files import save


class BazelToolchain(object):

def __init__(self, conanfile):
self._conanfile = conanfile

def generate(self):
bazel_config = self._conanfile.conf["tools.google.bazel:config"]
bazelrc_path = self._conanfile.conf["tools.google.bazel:bazelrc_path"]

save(CONAN_TOOLCHAIN_ARGS_FILE, json.dumps({
"bazel_config": bazel_config,
"bazelrc_path": bazelrc_path
}))
9 changes: 8 additions & 1 deletion conans/client/generators/__init__.py
Expand Up @@ -67,7 +67,8 @@ def __init__(self):
"markdown": MarkdownGenerator}
self._new_generators = ["CMakeToolchain", "CMakeDeps", "MSBuildToolchain",
"MesonToolchain", "MSBuildDeps", "QbsToolchain", "msbuild",
"VirtualEnv", "AutotoolsDeps", "AutotoolsToolchain", "AutotoolsGen"]
"VirtualEnv", "AutotoolsDeps", "AutotoolsToolchain", "AutotoolsGen",
"BazelDeps", "BazelToolchain"]

def add(self, name, generator_class, custom=False):
if name not in self._generators or custom:
Expand Down Expand Up @@ -120,6 +121,12 @@ def _new_generator(self, generator_name, output):
elif generator_name == "VirtualEnv":
from conan.tools.env.virtualenv import VirtualEnv
return VirtualEnv
elif generator_name == "BazelDeps":
from conan.tools.google import BazelDeps
return BazelDeps
elif generator_name == "BazelToolchain":
from conan.tools.google import BazelToolchain
return BazelToolchain
else:
raise ConanException("Internal Conan error: Generator '{}' "
"not commplete".format(generator_name))
Expand Down
10 changes: 8 additions & 2 deletions conans/test/conftest.py
Expand Up @@ -13,7 +13,8 @@
'cygwin': 'default',
'mingw32': 'default',
'mingw64': 'default',
'ninja': '1.10.2'
'ninja': '1.10.2',
'bazel': 'default'
}

tools_locations = {
Expand Down Expand Up @@ -41,7 +42,11 @@
'3.19': '/usr/share/cmake-3.19.7/bin'
}
},
'ninja': {'Windows': {'1.10.2': 'C:/Tools/ninja/1.10.2'}}
'ninja': {'Windows': {'1.10.2': 'C:/Tools/ninja/1.10.2'}},
'bazel': {
'Darwin': {'default': '/Users/jenkins/bin'},
'Windows': {'default': 'C:/bazel/bin'},
}
}

tools_environments = {
Expand All @@ -54,6 +59,7 @@
'gcc', 'clang', 'visual_studio', 'xcode',
'msys2', 'cygwin', 'mingw32', 'mingw64',
'autotools', 'pkg_config', 'premake', 'meson', 'ninja',
'bazel',
'file',
'git', 'svn',
'compiler',
Expand Down
Empty file.