Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Remoting/use maven mirror #8638

Closed
wants to merge 6 commits into from
Closed

Remoting/use maven mirror #8638

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
b1e418e
Point RBE at a mirror of maven central.
jsirois Nov 15, 2019
2c82f20
Set remote-specific options inside tests.
benjyw Nov 16, 2019
c0c4187
BUILD file dep on remote ivy settings.
benjyw Nov 17, 2019
f88c1e5
Add local and toolchain binhost repos.
benjyw Nov 17, 2019
024cb1b
Really make the ivysettings identical.
benjyw Nov 17, 2019
09b5f83
Simpler change.
benjyw Nov 17, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions build-support/docker/remote_execution/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,3 +44,5 @@ RUN ${PYENV_BIN} install ${PYTHON_37_VERSION}
ENV PATH "${PYENV_ROOT}/versions/${PYTHON_27_VERSION}/bin:${PATH}"
ENV PATH "${PYENV_ROOT}/versions/${PYTHON_36_VERSION}/bin:${PATH}"
ENV PATH "${PYENV_ROOT}/versions/${PYTHON_37_VERSION}/bin:${PATH}"

RUN touch /PANTS_GCP_REMOTE && chmod 755 /PANTS_GCP_REMOTE
33 changes: 33 additions & 0 deletions build-support/ivy/remote.ivysettings.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
<?xml version="1.0"?>
<!--
Copyright 2019 Pants project contributors (see CONTRIBUTORS.md).
Licensed under the Apache License, Version 2.0 (see LICENSE).
-->

<ivysettings>
<settings defaultResolver="maven-central-mirror"/>
<resolvers>
<!--
Identical to ivysettings.xml, except we point remote execution at the RBE maven central mirror
to avoid DDOSing maven central. -->
<chain name="pants-chain-repos" returnFirst="true">
<ibiblio
name="maven-central-mirror"
m2compatible="true"
descriptor="required"
root="https://maven-central.storage-download.googleapis.com/repos/central/data"/>

<!-- The mvn standard local filesystem repo/cache -->
<filesystem name="local.m2" m2compatible="true" local="true" checkmodified="true">
<ivy pattern="${m2.repo.dir}/${m2.repo.pom}"/>
<artifact pattern="${m2.repo.dir}/${m2.repo.artifact}"/>
</filesystem>

<!-- Custom repo for Kythe jars, as Google don't currently publish them anywhere. -->
<url name="toolchainlabs/binhost">
<artifact pattern="https://github.com/toolchainlabs/binhost/raw/master/${kythe.artifact}" />
</url>
</chain>

</resolvers>
</ivysettings>
35 changes: 34 additions & 1 deletion pants.remote.ini
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ remote_instance_name: projects/pants-remoting-beta/instances/default_instance
remote_execution_extra_platform_properties: [
# This allows network requests, e.g. to resolve dependencies with Pex.
"dockerNetwork=standard",
"container-image=docker://gcr.io/pants-remoting-beta/rbe-remote-execution@sha256:5d818cd71c9180d977e16ca7a20e90ced14211621b69fe1d6c3fc4c42c537a14",
"container-image=docker://gcr.io/pants-remoting-beta/rbe-remote-execution@sha256:dbc9005efdcf30f9e22b450c2ee0aef50077ff98dee84a9922e63f905aff926e",
]

# This should correspond to the number of workers running in Google RBE. See
Expand All @@ -32,6 +32,11 @@ process_execution_speculation_strategy: none
# p95 of RBE appears to be ~ 2 seconds, but we need to factor in local queue time which can be much longer, but no metrics yet.
process_execution_speculation_delay: 15

# DRY up maven central mirror setup in variables used below.
maven_central_mirror_root_url: https://maven-central.storage-download.googleapis.com/repos/central/data
maven_central_mirror_ivy_bootstrap_jar_url: %(maven_central_mirror_root_url)s/org/apache/ivy/ivy/2.4.0/ivy-2.4.0.jar
maven_central_mirror_ivy_settings: %(pants_supportdir)s/ivy/remote.ivysettings.xml

[python-setup]
# TODO(#7735): This config is not ideal, that we must specify the PATH for both local and remote
# platforms. This should be replaced by a proper mechanism to differentiate between the two.
Expand All @@ -47,3 +52,31 @@ interpreter_search_paths: [
[python-native-code]
ld_flags: []
cpp_flags: []

# Note that the logic below is repeated in jvm_task_test_base.py. See there for details.
# If you set remote-specific ivy/coursier options here, set them there as well.

[ivy]
bootstrap_jar_url: %(maven_central_mirror_ivy_bootstrap_jar_url)s
Copy link
Sponsor Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should be able to remove the bootstrap_jar_url overrides once #8637 lands.

bootstrap_ivy_settings: %(maven_central_mirror_ivy_settings)s
ivy_settings: %(maven_central_mirror_ivy_settings)s

[ivy.outdated]
bootstrap_jar_url: %(maven_central_mirror_ivy_bootstrap_jar_url)s
bootstrap_ivy_settings: %(maven_central_mirror_ivy_settings)s
ivy_settings: %(maven_central_mirror_ivy_settings)s

[ivy.outdated.ivy]
bootstrap_jar_url: %(maven_central_mirror_ivy_bootstrap_jar_url)s
bootstrap_ivy_settings: %(maven_central_mirror_ivy_settings)s
ivy_settings: %(maven_central_mirror_ivy_settings)s

[coursier]
repos: [
# Google RBE maven central mirror that avoids DDOSing maven central.
'%(maven_central_mirror_root_url)s',

# Custom repo for Kythe jars, as Google doesn't currently publish them anywhere.
'https://raw.githubusercontent.com/toolchainlabs/binhost/master/'
]

1 change: 1 addition & 0 deletions src/python/pants/testutil/jvm/BUILD
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,7 @@ python_library(
name='jvm_task_test_base',
sources=['jvm_task_test_base.py'],
dependencies=[
'build-support/ivy',
'src/python/pants/backend/jvm/tasks:classpath_products',
'src/python/pants/testutil/subsystem',
'src/python/pants/testutil:task_test_base',
Expand Down
15 changes: 15 additions & 0 deletions src/python/pants/testutil/jvm/jvm_task_test_base.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,21 @@ def setUp(self):
super().setUp()
init_subsystem(JvmResolveSubsystem)
self.set_options_for_scope('resolver', resolver='ivy')
if os.path.isfile('/PANTS_GCP_REMOTE'):
# Use the GCP maven mirrors, so we don't get throttled for DOSing maven central.
# Note that this recapitulates the logic in pants.remote.ini. Unfortunately we
# can't access the real options inside tests, and plumbing these through everywhere
# they're needed is prohibitive. So we repeat the logic here.
maven_central_mirror_root_url = 'https://maven-central.storage-download.googleapis.com/repos/central/data'
maven_central_mirror_ivy_bootstrap_jar_url = f'{maven_central_mirror_root_url}/org/apache/ivy/ivy/2.4.0/ivy-2.4.0.jar'
Copy link
Sponsor Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ditto

maven_central_mirror_ivy_settings = 'build-support/ivy/remote.ivysettings.xml'

self.set_options_for_scope('coursier', repos=[maven_central_mirror_root_url])
for scope in ['ivy', 'ivy.outdated', 'ivy.outdated.ivy']:
self.set_options_for_scope(scope,
bootstrap_jar_url=maven_central_mirror_ivy_bootstrap_jar_url,
bootstrap_ivy_settings=maven_central_mirror_ivy_settings,
ivy_settings=maven_central_mirror_ivy_settings)

def populate_runtime_classpath(self, context, classpath=None):
"""
Expand Down