Have a generic, easy-to-setup distributed caching #904

Open
damienmg opened this Issue Feb 15, 2016 · 18 comments

Projects

None yet

6 participants

@damienmg
Member

Build up on Alpha Lam's initial version to have a viable version for all users of Bazel.

@philwo is synchronizing all the efforts in that perspective.

@philwo philwo was assigned by damienmg Feb 15, 2016
@damienmg damienmg added this to the 0.5 milestone Feb 15, 2016
@tfarina
Contributor
tfarina commented Mar 22, 2016

@hhclam fyi

@ittaiz
Contributor
ittaiz commented May 17, 2016

Hi,
What's the status of this issue?

@philwo
Contributor
philwo commented May 17, 2016

@ittaiz We have prototype implementations of distributed caching and remote execution in Bazel now:

Distributed caching: 79adf59
Remote execution: a1a79cb

Here is some documentation: https://github.com/bazelbuild/bazel/blob/master/src/main/java/com/google/devtools/build/lib/remote/README.md

We will continue to improve this and welcome feedback :) Please file bugs and assign to @hhclam if you encounter issues with these features.

@ittaiz
Contributor
ittaiz commented May 17, 2016

got you.
Couple of questions:

  1. Are there any current thoughts on improvements or waiting for feedback? Asking since I saw some thoughts in the google group discussion
  2. Can we have the readme's contents incorporated to the bazel.io site? I looked for it there and couldn't find it. Also maybe update the roadmap that this is done. People just skimming the roadmap might not know it's there (I knew something is there by deep diving into the code).
  3. If I have thoughts they should be written here? Should they start in bazel-discuss and "graduate" here?

Thanks!

@damienmg
Member

It is still experimental so I think we need to at least discuss on the future of the Bazel interface before documenting it (if we going to change it a lot, it is better to not put too much documentation for now).

@ittaiz
Contributor
ittaiz commented May 17, 2016

ok. I think when people evaluate bazel vs pants vs buck the distributed caching and execution plays a big part.
Having a note saying it's already there in as a prototype and can be used by early adopters, willing to be broken, can encourage people. I'm such a user.

@damienmg
Member

Yeah that sounds reasonable. We should at least do a blog post with basic
documentation of both feature. But we need to sync with Alpha who did
almost all the work.

On Tue, May 17, 2016 at 12:53 PM Ittai Zeidman notifications@github.com
wrote:

ok. I think when people evaluate bazel vs pants vs buck the distributed
caching and execution plays a big part.
Having a note saying it's already there in as a prototype and can be used
by early adopters, willing to be broken, can encourage people. I'm such a
user.


You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub
#904 (comment)

@ittaiz
Contributor
ittaiz commented May 17, 2016

The discussion about where this feature should proceed will be in bazel-dev?

@damienmg
Member

We are planning to have a live discussion then we will follow up on
bazel-dev.

On Tue, May 17, 2016 at 1:21 PM Ittai Zeidman notifications@github.com
wrote:

The discussion about where this feature should proceed will be in
bazel-dev?


You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub
#904 (comment)

@ittaiz
Contributor
ittaiz commented May 17, 2016

looking forward to it

@LavaScornedOven
LavaScornedOven commented Nov 10, 2016 edited

Just as a feedback from a random C++ developer, missing distributed cache is a deal breaker for me, especially when I consider that I already got through pain of learning make, QMake, CMake, gradle, and what not, and doing it anew just to get another way to build locally... It just doesn't justify the effort.

@damienmg damienmg modified the milestone: 0.6, 0.5 Dec 19, 2016
@softprops

Any progress on this?

@damienmg
Member
damienmg commented Jan 9, 2017

Not really, documentation is still missing...

@ittaiz
Contributor
ittaiz commented Jan 9, 2017

@damienmg not to belittle documentation but does that mean the code itself is ready in your opinion?

@damienmg
Member
damienmg commented Jan 9, 2017

To use for remote caching yes, but it still has no server side. Hazelcast can be used but I am not sure about the stability.

I also prototyped a local disk cache yesterday, see https://cr.bazel.build/8133

@damienmg
Member
damienmg commented Jan 9, 2017

(side-note: the protocol is still not stable, so we might change stuff in the future)

@ittaiz
Contributor
ittaiz commented Jan 9, 2017

Thanks for the clarification. Do you expect distributed actions to be supported any time soon?

@damienmg
Member
damienmg commented Jan 9, 2017

It is in the same state that caching, no server-side, no documentation still subject to change
As said on IRC to @abergmeier, the protocol itself should work.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment