Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add --config=remote and --config=remote-cache modes for bazel #76815

Merged
merged 1 commit into from Apr 23, 2019
Merged

Add --config=remote and --config=remote-cache modes for bazel #76815

merged 1 commit into from Apr 23, 2019

Conversation

fejta
Copy link
Contributor

@fejta fejta commented Apr 19, 2019 •
edited

/assign @mikedanese

Enables remote execution on bazel (requires providing your own instance)

bazel test //pkg/... --config=remote \
  --remote_instance_name=projects/AN_RBE_PROJ/instances/default_instance

--config=remote-cache should allow us to simplify/improve our caching solution as well, potentially obviating greenhouse.

Builds on similar work happening in test-infra:

@k8s-ci-robot k8s-ci-robot added do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. needs-kind Indicates a PR lacks a `kind/foo` label and requires one. needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. release-note-none Denotes a PR that doesn't merit a release note. and removed do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. labels Apr 19, 2019
@fejta
Copy link
Contributor Author

fejta commented Apr 19, 2019

/assign @cblecker

@fejta
Copy link
Contributor Author

fejta commented Apr 19, 2019

/retest

@cblecker
Copy link
Member

@fejta this is an optional flag, right? It won't enable unless someone specifically adds the remote flag and specifies a remote build instance?

@fejta
Copy link
Contributor Author

fejta commented Apr 19, 2019 •
edited

Correct, this does nothing until someone adds a --config=remote flag to their bazel execution (which is equivalent to adding the 30 or so flags it declares)

Also the toolchains and rbe platform are unevaluated until required (which right now only happens when you add those flags).

So this is a big noop, but will make it easier to people (and our CI system) to start using remote execution.

@cblecker
Copy link
Member

Works for me! I don't mind approving this for build/ but I don't have enough bazel expertise to code review syntax. If @mikedanese wants to lgtm this, that would be great.

/approve

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Apr 19, 2019
mikedanese
mikedanese reviewed Apr 23, 2019
View reviewed changes
build/root/.bazelrc
build:remote --host_javabase=@rbe_default//java:jdk
build:remote --javabase=@rbe_default//java:jdk
build:remote --host_java_toolchain=@bazel_tools//tools/jdk:toolchain_hostjdk8
build:remote --java_toolchain=@bazel_tools//tools/jdk:toolchain_hostjdk8
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's so special about these java args?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TBH I don't understand this very deeply right now. There's some info at https://github.com/bazelbuild/bazel-toolchains/blob/master/rules/rbe_repo.bzl

I'm just copying the config the RBE team setup for test-infra.

I believe this configures the toolchain to use the tools that are present in remote containers.

build/root/.bazelrc
build:remote --spawn_strategy=remote
build:remote --strategy=Javac=remote
build:remote --strategy=Closure=remote
build:remote --strategy=Genrule=remote
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do you need these --strategy's if you already have --spawn_strategy?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Don't know right now

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

They look unnecessary given that --spawn_strategy value is the default and --strategy is used for action specific overrides.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can remove them later?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, we can play around with it.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sent a mail asking for more details. Other changes will likely be necessary as a variety of tests fail when RBE is enabled. Will address this along with making those tests pass

build/root/.bazelrc
build:remote --strategy=Javac=remote
build:remote --strategy=Closure=remote
build:remote --strategy=Genrule=remote
build:remote --define=EXECUTOR=remote
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this special magic that something else understands or is this for our usage?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure

build/root/.bazelrc
build:remote-cache --remote_cache=remotebuildexecution.googleapis.com
build:remote-cache --tls_enabled=true
build:remote-cache --remote_timeout=3600
build:remote-cache --auth_enabled=true
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we only want --auth_enabled and --tls_enabled on remote-cache?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's a composition: build:remote --config=remote-cache

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure, but the question is "why aren't --auth_enabled and --tls_enabled on --config=remote rather than the composite?"

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

They are. See L52. The remote config includes everything in remote-cache (not the reverse).

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah thanks.

@mikedanese
Copy link
Member

/lgtm
/approve

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Apr 23, 2019
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: cblecker, fejta, mikedanese

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@cblecker
Copy link
Member

/kind feature
/sig testing
/priority backlog

@k8s-ci-robot k8s-ci-robot added kind/feature Categorizes issue or PR as related to a new feature. sig/testing Categorizes an issue or PR as relevant to SIG Testing. priority/backlog Higher priority than priority/awaiting-more-evidence. and removed needs-kind Indicates a PR lacks a `kind/foo` label and requires one. needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. labels Apr 23, 2019
@fejta
Copy link
Contributor Author

fejta commented Apr 23, 2019

/hold cancel

Thanks so much!

@fejta-bot
Copy link

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel or /hold comment for consistent failures.

@k8s-ci-robot k8s-ci-robot merged commit 137aa9f into kubernetes:master Apr 23, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@mikedanese mikedanese mikedanese left review comments

@jbeda jbeda Awaiting requested review from jbeda

@zmerlynn zmerlynn Awaiting requested review from zmerlynn

Assignees

@cblecker cblecker

@mikedanese mikedanese

Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/feature Categorizes issue or PR as related to a new feature. lgtm "Looks good to me", indicates that a PR is ready to be merged. priority/backlog Higher priority than priority/awaiting-more-evidence. release-note-none Denotes a PR that doesn't merit a release note. sig/testing Categorizes an issue or PR as relevant to SIG Testing. size/M Denotes a PR that changes 30-99 lines, ignoring generated files.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
5 participants
@fejta @cblecker @mikedanese @k8s-ci-robot @fejta-bot