Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create dedicated Dockerfile with Python 3.6 to reduce CI time #7352

Merged
merged 3 commits into from Mar 12, 2019
Merged
Changes from all commits
Commits
File filter...
Filter file types
Jump to…
Jump to file or symbol
Failed to load files and symbols.

Always

Just for now

@@ -270,19 +270,17 @@ py27_linux_build_engine: &py27_linux_build_engine
- PREPARE_DEPLOY=1
- CACHE_NAME=linuxpexbuild.py27
- BOOTSTRAPPED_PEX_KEY_SUFFIX=py27.linux
- BOOTSTRAP_ARGS='-2b'

py36_linux_build_engine: &py36_linux_build_engine
<<: *py36_linux_config
<<: *base_linux_build_engine
name: "Build Linux native engine and pants.pex (Py3.6 PEX)"
env:
- docker_image_name=travis_ci
- docker_image_name=travis_ci_py36
# Note that we also build fs_util, to take advantage of the rust code built during bootstrapping.
- docker_run_command="./build-support/bin/ci.sh -b && ./build-support/bin/release.sh -f"
- CACHE_NAME=linuxpexbuild.py36
- BOOTSTRAPPED_PEX_KEY_SUFFIX=py36.linux
- BOOTSTRAP_ARGS='-b'

base_osx_build_engine: &base_osx_build_engine
<<: *native_engine_cache_config
@@ -4,19 +4,6 @@
# Use our custom Centos6 image for binary compatibility with old linux distros.
FROM pantsbuild/centos6:latest

# TODO(7064) These changes belong in centos6/Dockerfile, because we always want that image to
# have Py3 available and also avoid the cost of rebuilding Python 3 every time in CI. However,
# this requires publishing an updated image to Docker hub, so we copy the code here until then.
ARG PYTHON_3_VERSION=3.6.8
RUN yum install sqlite-devel -y
ENV PYENV_ROOT /pyenv-docker-build
RUN mkdir ${PYENV_ROOT}
RUN git clone https://github.com/pyenv/pyenv ${PYENV_ROOT}
RUN /usr/bin/scl enable devtoolset-7 -- bash -c '\
${PYENV_ROOT}/bin/pyenv install ${PYTHON_3_VERSION} \
&& ${PYENV_ROOT}/bin/pyenv global ${PYTHON_3_VERSION}'
ENV PATH "${PYENV_ROOT}/shims:${PATH}"

This comment has been minimized.

Copy link
@cosmicexplorer

cosmicexplorer Mar 11, 2019

Contributor

Does this end up making the first CI stage faster by 3 minutes in real time as well? Or are we still paying that cost when installing python 3 for the py 3 linux pex shard in stage 1, and the win here is just in overall time across all shards?

This comment has been minimized.

Copy link
@Eric-Arellano

Eric-Arellano Mar 11, 2019

Author Contributor

Unfortunately win is only in overall CI time, not in wall time. We still have to build Py36 for Linux Build Engine Py36, and soon the two wheel building shards for Py3.

Which is why #7064 will be so great! And your PR will also help with Py37 to save time with our cron job, once I figure out #7261.

# Setup mount points for the travis ci user & workdir.
VOLUME /travis/home
VOLUME /travis/workdir
@@ -0,0 +1,46 @@
# Copyright 2019 Pants project contributors (see CONTRIBUTORS.md).
# Licensed under the Apache License, Version 2.0 (see LICENSE).

# TODO(7064): this file duplicates travis_ci/Dockerfile, except it installs Python 3.6.
# We split it out into a separate file to avoid the cost of installing Python 3.6 for Py2 only
# shards. Ideally this file would not exist though, as these changes belong in the base
# image Centos6! Once those changes are published to Dockerhub, this file should be deleted
# and the shards should go back to using `travis_ci`.

# Use our custom Centos6 image for binary compatibility with old linux distros.
FROM pantsbuild/centos6:latest

# ----------------------------------------------------------
# TODO(7064) Add all the below changes to Centos6 image.

RUN yum install sqlite-devel -y
ARG PYTHON_36_VERSION=3.6.8
ENV PYENV_ROOT /pyenv-docker-build
RUN mkdir ${PYENV_ROOT}
RUN git clone https://github.com/pyenv/pyenv ${PYENV_ROOT}
RUN /usr/bin/scl enable devtoolset-7 -- bash -c '\
${PYENV_ROOT}/bin/pyenv install ${PYTHON_36_VERSION} \
&& ${PYENV_ROOT}/bin/pyenv global ${PYTHON_36_VERSION}'
ENV PATH "${PYENV_ROOT}/shims:${PATH}"

# End TODO(7064)
# ----------------------------------------------------------

# Setup mount points for the travis ci user & workdir.
VOLUME /travis/home
VOLUME /travis/workdir

# Setup a non-root user to execute the build under (avoids problems with npm install).
ARG TRAVIS_USER=travis_ci
ARG TRAVIS_UID=1000
ARG TRAVIS_GROUP=root
ARG TRAVIS_GID=0

RUN groupadd --gid ${TRAVIS_GID} ${TRAVIS_GROUP} || true
RUN useradd -d /travis/home -g ${TRAVIS_GROUP} --uid ${TRAVIS_UID} ${TRAVIS_USER}
USER ${TRAVIS_USER}:${TRAVIS_GROUP}

# Our newly created user is unlikely to have a sane environment: set a locale at least.
ENV LC_ALL="en_US.UTF-8"

WORKDIR /travis/workdir
@@ -234,19 +234,17 @@ py27_linux_build_engine: &py27_linux_build_engine
- PREPARE_DEPLOY=1
- CACHE_NAME=linuxpexbuild.py27
- BOOTSTRAPPED_PEX_KEY_SUFFIX=py27.linux
- BOOTSTRAP_ARGS='-2b'
py36_linux_build_engine: &py36_linux_build_engine
<<: *py36_linux_config
<<: *base_linux_build_engine
name: "Build Linux native engine and pants.pex (Py3.6 PEX)"
env:
- docker_image_name=travis_ci
- docker_image_name=travis_ci_py36
# Note that we also build fs_util, to take advantage of the rust code built during bootstrapping.
- docker_run_command="./build-support/bin/ci.sh -b && ./build-support/bin/release.sh -f"
- CACHE_NAME=linuxpexbuild.py36
- BOOTSTRAPPED_PEX_KEY_SUFFIX=py36.linux
- BOOTSTRAP_ARGS='-b'
base_osx_build_engine: &base_osx_build_engine
<<: *native_engine_cache_config
ProTip! Use n and p to navigate between commits in a pull request.
You can’t perform that action at this time.