Make it possible to share Buck rules between different projects #116

Open
davido opened this Issue May 20, 2014 · 13 comments

Projects

None yet

6 participants

@davido
Contributor
davido commented May 20, 2014

When a build process for a non trivial project (e. g. Gerrit Code Review) is implemented in Buck, a lot of custom scripts/recipes are created. It turns out that substantial part of these scripts/recipes is very useful and urgently needed for other projects.

Currently there is no built in way in Buck to share/reuse scripts/recipes between projects.
One option would be to use bucklets: number of build.defs and python_scripts files:

Project foo defines bucklet local_jar() [1] and publishes it with deploy_bucklet('local_jar')

Project bar (unrelated to project foo/or a plugin for project foo) reuses that bucklet with: import_bucklet('local_jar').

Gerrit Code Review project has created bucklets [2] project and it is currently hosted on gerrit-review. First project from Gerrit eco system was migrated to Buck (re-)using only bucklets as Git submodule [3].

Please help to upstream that in one or another way. It's urgently needed to move further projects from Gerrit eco system (e. g. Gerrit plugins, JGit, EGit, ...) to Buck without code duplication.

[1] https://gerrit.googlesource.com/bucklets/+/master/local_jar.bucklet
[2] https://gerrit.googlesource.com/bucklets
[3] https://gerrit.googlesource.com/gitiles

@shs96c
Contributor
shs96c commented May 20, 2014

I've recently started work on allowing targets to be extended to support cross-repo links. The syntax will look something like:

@repo//path/to/buck/file:target-name

If we extended the include_defs() function to understand targets, would that work for you?

@sdwilsh sdwilsh added the enhancement label May 20, 2014
@shs96c shs96c was assigned by sdwilsh May 20, 2014
@davido
Contributor
davido commented May 21, 2014

Interesting.

@repo//path/to/buck/file:target-name

So the old code [1] to consume maven_jar() buckler [2](that currently must be included as Git submodule to work) would change from

include_defs('//bucklets/maven_jar.bucklet')

maven_jar(
  name = 'guava',
  id = 'com.google.guava:guava:17.0',
)

to

include_defs('https://github.com/davido/bucklets//maven_jar.bucklet')

maven_jar(
  name = 'guava',
  id = 'com.google.guava:guava:17.0',
)

or even just:

@https://github.com/davido/maven_jar.bucklet:maven_jar(
  name = 'guava',
  id = 'com.google.guava:guava:17.0',
)

?

@https://github.com/davido/maven_jar.bucklet:maven_jar would be too verbose to write. May be provide a .bucklets file with alias/mapping of external targets to internal ones:

$cat .bucklets
maven_jar = @https://github.com/davido/maven_jar.bucklet:maven_jar
local_jar = @https://github.com/davido/local_jar.bucklet:local_jar
[...]

If @https://github.com/davido/maven_jar.bucklet:maven_jar includes only one target
maven_jar() may be maven_jar suffix can be omitted?

Another question: the backend for maven_jar() is actually download_file.py [3]. Would it work and be seamlessly available from the remote repo in the local build process as well?

If you can provide a patch for it, (as a file or branch) i would be happy to give it a try.

[1] https://gerrit.googlesource.com/gitiles/+/master/lib/BUCK
[2] https://gerrit.googlesource.com/bucklets/+/master/maven_jar.bucklet
[3] https://gerrit.googlesource.com/bucklets/+/master/tools/download_file.py

@shs96c
Contributor
shs96c commented May 21, 2014

Simpler than that. It'd be:

include_defs('@bucklets//bucklets:maven_jar')

maven_jar(
  name = 'guava',
  id = 'com.google.guava:guava:17.0',
)

Where the include becomes a target in a build file (probably using export_file)

@davido
Contributor
davido commented May 21, 2014

include_defs('@bucklets//bucklets:maven_jar')

Very promising.

Where is the link/mapping between '@bucklets//bucklets:maven_jar' and actual @repo (https://github.com/davido in example above)?
When two different files would reference the same remote target, i. e.:

$cat lib/guava/BUCK
include_defs('@bucklets//bucklets:maven_jar')

maven_jar(
  name = 'guava',
  id = 'com.google.guava:guava:17.0',
)

$cat lib/gwt/BUCK
include_defs('@bucklets//bucklets:maven_jar')

maven_jar(
  name = 'gwt-dev',
  id = 'com.google.gwt:gwt-dev:' + VERSION,
  license = 'Apache2.0',
  deps = [
    ':javax-validation',
    ':javax-validation_src',
  ],
  attach_source = False,
  exclude = ['org/eclipse/jetty/*'],
)

We would get multiple maven_jar() target definitions? One per file where it is used?

@Coneko
Contributor
Coneko commented Feb 2, 2015

You'd have to check out the remote repo locally, maybe as a submodule or something like that.

@sdwilsh
Contributor
sdwilsh commented Jun 5, 2015

We actually backed out the cross repo stuff as it wasn't working correctly anymore.

@Coneko
Contributor
Coneko commented Jun 9, 2015

We're still planning on adding it, we just decided we are better off rewriting it from scratch.

@LegNeato
Contributor
LegNeato commented Oct 1, 2015

Cross repo stuff is in now, right?

@k21
Contributor
k21 commented Oct 1, 2015

Cross repo (or cross cell, as we are going to call it) is currently under development and although preliminary support has landed in the master branch, most build rules do not support it properly yet.

@sdwilsh
Contributor
sdwilsh commented Oct 1, 2015

It should be in within a month would be my guess. You could probably start to play with it seriously as early as next week.

@davido
Contributor
davido commented Oct 2, 2015

Cross repo (or cross cell, as we are going to call it) is currently under development and although preliminary support has landed in the master branch, most build rules do not support it properly yet.

Can the cell support be extended to non locally cloned repositories? The idea behind plugins or extensions or bucklets concept is to make them first-class citizens in Buck. In similar way, Buck allows to specify what buck version to use in .buckversion file, I would expect to be able to provide some meta data for cell without actually cloning it:

  [cell name = "bucklets"]
    remote =  https://github.com/davido/bucklets
    version = d2936a48fc559e90b66e83de7ff163202e75486b
    # optional, with sensible default, e.g.
    # local = $HOME/.bucklets

in some configuration file, and without cloning this repo, or adding it to my own project as a submodule. I should be able to say include_defs('//bucklets/gerrit_plugin.bucklet') and just use gerrit_plugin rule. See also how the cross repo rule sharing is implemented in Gerrit plugin by using Git submodule concept: [1]. The problem to solve is, that buck build foo would need to take care for checking and cloning the cell(s) for me and make them accessible (I call this "mount") from within build rules, like //foo/bar/baz, where foo is a known cell and also from the shell, because Eclipse project generation is implemented as a standalone Python script: [2] and currently invoked as:

  ./bucklets/tools/eclipse.py

Another approach would be to add dedicated Buck command to handle that with say buck getcell, something similar is done with Golang's go get: [3] command:

get download and install packages and dependencies

[1] https://github.com/davido/gerrit-oauth-provider/blob/master/BUCK#L1,L24
[2] https://github.com/davido/bucklets/blob/master/tools/eclipse.py
[3] https://golang.org/cmd/go

@davido
Contributor
davido commented Dec 4, 2015

I started to implement cross cell support in gerrit, to simplify workflow for integration of WIP stuff across project boundaries. One typical example of it include changes in JGit (main Gerrit dependency and Git implementation in Java programming language), Gerrit itself and dedicated plugin, when the feature implementation is split in core and plugin part, like for example LFS support to allow plugin to provide their own specific protocol implementations: [1]. [2], [3].

I hope that cross cell support, isolating jgit in gerrit project in its own cell, can solve this. It would work as is per default and could be hijacked per CLI option and be routed to JGit development tree. I still have some issues to make it actually work and filed #544 and #545.

Another problem I'm facing is the missing support for include_def() across cell boundaries. My hope was, that when fetching of cell is solved, the cross-cell concept would replace git submodules for reusing of Buck build files, see https://github.com/davido/bucklets.

So when a plugin is implemented standalone Buck build (outside of gerrit tree), see for example: https://github.com/davido/gerrit-oauth-provider it's basically consuming pre-defined bucklet: [4]

  include_defs('//bucklets/gerrit_plugin.bucklet')

Right now I'm getting error: "include should start with '//' or some such" when i was trying to pass cell prefix: @bucklets//gerrit_plugin.buckletto include_def(). Is this something that will be supported in future or cell concept is mainly to share rules and not build files? In which case we still don't have any replacement for git submodules?

@lucamilanesio lucamilanesio pushed a commit to GerritCodeReview/gerrit that referenced this issue Jan 28, 2016
@davido davido + David Ostrovsky Update Buck to newest version
Now, that cross-cell support is implemented: [1] we can share rules
between projects. Cell is a container with its own buck-out directory.
The Buck design allows to replace a cell with alternative location.
This is a preparation change to support cross-cell dependencies.

The prerequisite for the hijacked cell to work properly, is that the
classpaths are always represented as absolute paths. That was not the
case in Buck implementation and thus classpath resolution across cell
boundaries was broken. This got fixed in: [2], [3].

[1] facebook/buck#116
[2] facebook/buck#545
[3] facebook/buck#558

Change-Id: I3b17129c0f7ca5801f03091fff5651fb63d1482f
feeb212
@davido
Contributor
davido commented Jun 6, 2016 edited

Two years later, after this issue was created, I've discovered, that Bazel team released the cross project dependency support, with the ability to fetch the deps from remote repositories, "mount" and re-use them in build toolchain: [1].

git_repository(
    name = "io_bazel_rules_scala",
    remote = "https://github.com/bazelbuild/rules_scala.git",
    tag = "0.0.1",
)
load("@io_bazel_rules_scala//scala:scala.bzl", "scala_repositories")
scala_repositories()

[...]

This will download all of the tools the rules need to build Scala programs.
Then load and use normally from your BUILD files:

load("@io_bazel_rules_scala//scala:scala.bzl", "scala_library")
scala_library(...)

We have extracted Bucklets from Gerrit Code Review build tool chain: [2]. Unfortunately, Buck still doesn't have support for fetching and using external Buck scripts. And it's just annoying to link Bucklets as git submodules to each and every Gerrit Code Review plugin: [3].

[1] http://www.bazel.io/blog/2016/02/23/0.2.0-release.html
[2] https://github.com/davido/bucklets
[3] https://github.com/davido/gerrit-oauth-provider

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment