Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ARROW-2676: [Packaging] Deploy build artifacts to github releases #2109

Closed
wants to merge 35 commits into from
Closed

ARROW-2676: [Packaging] Deploy build artifacts to github releases #2109

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
35 commits
Select commit Hold shift + click to select a range
0ab0567
don't create tree based on master
kszucs Jun 4, 2018
b65cb3f
track remote branches
kszucs Jun 5, 2018
8a5fcb0
trigger build task; explicit task names during submission
kszucs Jun 6, 2018
a0a8127
tigger builds template
kszucs Jun 6, 2018
bd5ece4
rename trigger build; add license header
kszucs Jun 6, 2018
edeee71
refactooooor
kszucs Jun 12, 2018
55e7ccb
query statuses API
kszucs Jun 13, 2018
96e101a
draft for downloading artifacts
kszucs Jun 13, 2018
0b79d13
trying to remove python dependency from parquet-cpp's recipe
kszucs Jun 13, 2018
5374866
cleanup status check and artifact download
kszucs Jun 13, 2018
8c069c6
set autoincremented job_id outside of queue put
kszucs Jun 14, 2018
5c6e154
fix conda-win deployments
kszucs Jun 14, 2018
fd6bd23
print build id
kszucs Jun 14, 2018
14da2d5
build prefix
kszucs Jun 14, 2018
ccbc150
little more status context
kszucs Jun 14, 2018
167a393
gry to remove osx wheel flag
kszucs Jun 14, 2018
af6dd48
remove flag
kszucs Jun 14, 2018
0773f3a
call rake version update
kszucs Jun 15, 2018
95bc3fa
retrieve commit's combined status
kszucs Jun 17, 2018
bfb5eae
fix osx wheel builkds
kszucs Jun 17, 2018
71b6943
postfix conda pkgs with arch
kszucs Jun 18, 2018
0561e62
force update conda win assets
kszucs Jun 18, 2018
9b22238
update tasks
kszucs Jun 18, 2018
4c64db6
Add email property and use target repo user.email by default
cpcloud Jun 18, 2018
1b3d5fb
Code formatting cleanups
cpcloud Jun 18, 2018
12efaba
Validate github token
cpcloud Jun 18, 2018
337234d
Validate submit tasks
cpcloud Jun 18, 2018
f317a06
outdated readme section
kszucs Jun 18, 2018
2e9fcf9
remove logging config
kszucs Jun 18, 2018
e4e971c
don't depend on any previous commit
kszucs Jun 18, 2018
b892471
explicit task names instead of placeholder in readme; foxme note for …
kszucs Jun 18, 2018
062da7c
conda win renames
kszucs Jun 19, 2018
0701d64
fixed conda-win deployments
kszucs Jun 20, 2018
83f333b
don't use dirty flag in version number
kszucs Jun 20, 2018
87860d0
batch correctly rename double extensions
kszucs Jun 20, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
3 changes: 3 additions & 0 deletions cpp/CMakeLists.txt
View file
Open in desktop
Expand Up @@ -343,6 +343,9 @@ set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} ${ARROW_CXXFLAGS}")
# For any C code, use the same flags.
set(CMAKE_C_FLAGS "${CMAKE_CXX_FLAGS}")

# Remove --std=c++11 to avoid errors from C compilers
string(REPLACE "-std=c++11" "" CMAKE_C_FLAGS ${CMAKE_C_FLAGS})
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FYI, this is a temporary fix until we can address ARROW-2720. This is fragile if (with CMAKE_CXX_STANDARD set to 11) -std=gnu++11 ends up in CMAKE_C_FLAGS.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is removed in master (I removed it in a patch recently). I think this is a rebase artifact and should be removed

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep, that removal broke osx builds though. That's why we added it back.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It broke osx wheel and/or conda builds, not CI builds (obviously) though.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's super weird, how did that flag end up in CMAKE_C_FLAGS? What kind of error did it cause?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't exactly what's happening, but I know that we are setting CMAKE_CXX_STANDARD to 11 and then settings CMAKE_C_FLAGS to CMAKE_CXX_FLAGS, which might explain how the std flag is ending up there. However, this doesn't show up on any NIX systems except OSX which is why I'm not 100% sure what's happening.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The error you get is error: invalid argument '-std=c++11' not allowed with 'C/ObjC' when the ae.c file is being compiled during the plasma compilation. Without a mac to figure out what's going on, it's pretty hard for me to tell what the heck is happening.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh I see. I think gcc and clang just ignore the flag when compiling C code. This suggests we should be setting up the CMAKE_C_FLAGS separate from CXX. I'm opening a JIRA

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

https://issues.apache.org/jira/browse/ARROW-2730

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great! Thanks.


# Add C++-only flags, like -std=c++11
set(CMAKE_CXX_FLAGS "${CXX_ONLY_FLAGS} ${CMAKE_CXX_FLAGS}")

Expand Down
81 changes: 26 additions & 55 deletions dev/tasks/README.md
View file
Open in desktop
Expand Up @@ -80,12 +80,12 @@ submission. The tasks are defined in `tasks.yml`
6. Install the python dependencies for the script:

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One thing that is worth nothing in this README.md is that you need at least one commit in your queue repo or else this happens:

Traceback (most recent call last):
  File "crossbow.py", line 413, in <module>
    crossbow(auto_envvar_prefix='CROSSBOW')
  File "/home/phillip/miniconda/envs/pyarrow36/lib/python3.6/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/home/phillip/miniconda/envs/pyarrow36/lib/python3.6/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/home/phillip/miniconda/envs/pyarrow36/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/phillip/miniconda/envs/pyarrow36/lib/python3.6/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/phillip/miniconda/envs/pyarrow36/lib/python3.6/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "crossbow.py", line 356, in submit
    queue.put(job)
  File "crossbow.py", line 186, in put
    branch = self._create_branch(task.branch, files=task.files())
  File "crossbow.py", line 158, in _create_branch
    parents = parents or [self.head]
  File "crossbow.py", line 96, in head
    return self.repo.head.target
_pygit2.GitError: reference 'refs/heads/master' not found

Is there any way that we can automate this away? If not, make sure to add a note about it in the README.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We don't need parent commits, it just makes the graph nicer.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Resolved

```bash
conda install -y jinja2 pygit2 click pyyaml
conda install -y jinja2 pygit2 click pyyaml setuptools_scm github3.py
```

```bash
# pygit2 requires libgit2: http://www.pygit2.org/install.html
pip install -y jinja2 pygit2 click pyyaml
pip install -y jinja2 pygit2 click pyyaml setuptools_scm github3.py
```

7. Try running it:
Expand All @@ -106,7 +106,7 @@ The script does the following:
$ git clone https://github.com/kszucs/crossbow

$ cd arrow/dev/tasks
$ python crossbow.py
$ python crossbow.py submit conda-win conda-linux conda-osx
```

2. Gets the HEAD commit of the currently checked out branch and generates
Expand All @@ -115,94 +115,65 @@ The script does the following:

```bash
git checkout ARROW-<ticket number>
python dev/tasks/crossbow.py --dry-run
python dev/tasks/crossbow.py submit --dry-run conda-linux conda-osx
```

> Note that the arrow branch must be pushed beforehand, because the script
> will clone the selected branch.

3. Reads and renders the required build configurations with the parameters
substituted.
2. Create a commit per build configuration to its own branch. For example
to build `travis-linux-conda.yml` it will place a commit to the tip of
`crossbow@travis-linux-conda` branch.
2. Create a branch per task, prefixed with the job id. For example
to build conda recipes on linux it will create a new branch:
`crossbow@build-<id>-conda-linux`.
3. Pushes the modified branches to GitHub which triggers the builds.
For authentication it uses github oauth tokens described in the install
section.


### Examples

The script accepts a pattern as a first argument to narrow the build scope:

Run all builds:
### Query the build status

```bash
$ python crossbow.py
Repository: https://github.com/kszucs/arrow@tasks
Commit SHA: 810a718836bb3a8cefc053055600bdcc440e6702
Version: 0.9.1.dev48+g810a7188.d20180414
Pushed branches:
- travis-osx-wheel
- travis-linux-packages
- travis-linux-wheel
- appveyor-win-wheel
- appveyor-win-conda
- travis-linux-conda
- travis-osx-conda
python crossbow.py status <build id / branch name>
```

Just render without applying or committing the changes:
### Download the build artifacts

```bash
$ python crossbow.py --dry-run
python crossbow.py artifacts <build id / branch name>
```

Run only `conda` package builds but on all platforms:
### Examples

The script accepts a pattern as a first argument to narrow the build scope:

Run multiple builds:

```bash
$ python crossbow.py conda
$ python crossbow.py submit linux-packages conda-linux wheel-win
Repository: https://github.com/kszucs/arrow@tasks
Commit SHA: 810a718836bb3a8cefc053055600bdcc440e6702
Version: 0.9.1.dev48+g810a7188.d20180414
Pushed branches:
- appveyor-win-conda
- travis-linux-conda
- travis-osx-conda
- linux-packages
- conda-linux
- wheel-win
```

Run `wheel` builds:
Just render without applying or committing the changes:

```bash
$ python crossbow.py wheel
Repository: https://github.com/kszucs/arrow@tasks
Commit SHA: 810a718836bb3a8cefc053055600bdcc440e6702
Version: 0.9.1.dev48+g810a7188.d20180414
Pushed branches:
- travis-osx-wheel
- travis-linux-wheel
- appveyor-win-wheel
$ python crossbow.py submit --dry-run task_name
```

Run `osx` builds:
Run only `conda` package builds but on all platforms:

```bash
$ python crossbow.py osx
Repository: https://github.com/kszucs/arrow@tasks
Commit SHA: cad1df2c7f650ad3434319bbbefed0d4abe45e4a
Version: 0.9.1.dev130+gcad1df2c.d20180414
Pushed branches:
- travis-osx-wheel
- travis-osx-conda
$ python crossbow.py submit conda-win conda-osx conda-linux
```

Run only `linux-conda` package build:
Run `wheel` builds:

```bash
$ python crossbow.py linux-conda
Repository: https://github.com/kszucs/arrow@tasks
Commit SHA: 810a718836bb3a8cefc053055600bdcc440e6702
Version: 0.9.1.dev48+g810a7188.d20180414
Pushed branches:
- travis-linux-conda
$ python crossbow.py submit wheel-osx wheel-linux wheel-win
```
26 changes: 21 additions & 5 deletions dev/tasks/conda-recipes/appveyor.yml
View file
Open in desktop
Expand Up @@ -23,7 +23,6 @@ environment:
- TARGET_ARCH: x64
CONDA_PY: 36
CONDA_INSTALL_LOCN: C:\\Miniconda36-x64
ARROW_SRC: C:\apache-arrow
ARROW_VERSION: {{ ARROW_VERSION }}

install:
Expand All @@ -45,11 +44,28 @@ install:
build: off

test_script:
- git clone -b {{ ARROW_BRANCH }} {{ ARROW_REPO }} %ARROW_SRC% || exit /B
- git checkout {{ ARROW_SHA }} || exit /B
- pushd %ARROW_SRC%\dev\tasks\conda-recipes
- conda.exe build parquet-cpp arrow-cpp pyarrow
- git clone -b {{ ARROW_BRANCH }} {{ ARROW_REPO }} arrow || exit /B
- git -C arrow checkout {{ ARROW_SHA }} || exit /B
- pushd arrow\dev\tasks\conda-recipes
- conda.exe build --output-folder . parquet-cpp arrow-cpp pyarrow
- pushd win-64
- for %%f in (*.tar.bz2) do (
set %%g=%%~nf
ren "%%f" "%%~ng-win-64.tar.bz2"
)

artifacts:
# this must be relative and child of the build C:\projects\crossbow directory
- path: arrow\dev\tasks\conda-recipes\win-64\*.tar.bz2

deploy:
release: {{ BUILD_TAG }}
provider: GitHub
auth_token: "%CROSSBOW_GITHUB_TOKEN%"
artifact: /.*\.tar\.bz2/
draft: false
prerelease: false
force_update: true

notifications:
- provider: Email
Expand Down
7 changes: 1 addition & 6 deletions dev/tasks/conda-recipes/parquet-cpp/meta.yaml
View file
Open in desktop
Expand Up @@ -36,26 +36,21 @@ source:

build:
number: 0
skip: true # [win and not (py35 and win64)]
features:
- vc14 # [win and py35]
- vc14 # [win]

requirements:
build:
- toolchain
- boost-cpp 1.66.0
- cmake
- thrift-cpp >=0.11
- python # [win]
- arrow-cpp {{ ARROW_VERSION }}

run:
- arrow-cpp {{ ARROW_VERSION }}

test:
requires:
- python {{ environ['PY_VER'] + '*' }} # [win]

commands:
- test -f $PREFIX/lib/libparquet.so # [linux]
- test -f $PREFIX/lib/libparquet.dylib # [osx]
Expand Down
18 changes: 17 additions & 1 deletion dev/tasks/conda-recipes/travis.linux.yml
View file
Open in desktop
Expand Up @@ -26,6 +26,7 @@ env:
- CONDA_PY=35
- CONDA_PY=36
global:
- TRAVIS_TAG={{ BUILD_TAG }}
- ARROW_VERSION={{ ARROW_VERSION }}

install:
Expand Down Expand Up @@ -55,7 +56,22 @@ script:
- git clone -b {{ ARROW_BRANCH }} {{ ARROW_REPO }} arrow
- git -C arrow checkout {{ ARROW_SHA }}
- pushd arrow/dev/tasks/conda-recipes
- conda build parquet-cpp arrow-cpp pyarrow
- conda build --output-folder . parquet-cpp arrow-cpp pyarrow
- |
pushd linux-64
for file in *.tar.bz2; do
mv "$file" "$(basename "$file" .tar.bz2)-linux-64.tar.bz2"
done
popd

deploy:
provider: releases
api_key: $CROSSBOW_GITHUB_TOKEN
file_glob: true
file: $TRAVIS_BUILD_DIR/arrow/dev/tasks/conda-recipes/linux-64/*.tar.bz2
skip_cleanup: true
on:
tags: true

notifications:
email:
Expand Down
18 changes: 17 additions & 1 deletion dev/tasks/conda-recipes/travis.osx.yml
View file
Open in desktop
Expand Up @@ -26,6 +26,7 @@ env:
- CONDA_PY=35
- CONDA_PY=36
global:
- TRAVIS_TAG={{ BUILD_TAG }}
- ARROW_VERSION={{ ARROW_VERSION }}

before_install:
Expand Down Expand Up @@ -64,7 +65,22 @@ script:
- git clone -b {{ ARROW_BRANCH }} {{ ARROW_REPO }} arrow
- git -C arrow checkout {{ ARROW_SHA }}
- pushd arrow/dev/tasks/conda-recipes
- conda build parquet-cpp arrow-cpp pyarrow
- conda build --output-folder . parquet-cpp arrow-cpp pyarrow
- |
pushd osx-64
for file in *.tar.bz2; do
mv "$file" "$(basename "$file" .tar.bz2)-osx-64.tar.bz2"
done
popd

deploy:
provider: releases
api_key: $CROSSBOW_GITHUB_TOKEN
file_glob: true
file: $TRAVIS_BUILD_DIR/arrow/dev/tasks/conda-recipes/osx-64/*.tar.bz2
skip_cleanup: true
on:
tags: true

notifications:
email:
Expand Down
67 changes: 67 additions & 0 deletions dev/tasks/config-nightlies/travis.linux.yml
View file
Open in desktop
@@ -0,0 +1,67 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

branches:
# don't attempt to build branches intented for windows builds
except:
- /.*win.*/

os: linux
dist: trusty
language: generic

before_install:
# Install Miniconda.
- echo `pwd`
- |
echo ""
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any particular reason why we're echoing a newline here?

echo "Installing a fresh version of Miniconda."
MINICONDA_URL="https://repo.continuum.io/miniconda"
MINICONDA_FILE="Miniconda3-latest-Linux-x86_64.sh"
curl -L -O "${MINICONDA_URL}/${MINICONDA_FILE}"
bash $MINICONDA_FILE -b

# Configure conda.
- |
echo ""
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any particular reason why we're echoing a newline here?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy paste from conda forge. This newline probably makes the output more readable.

echo "Configuring conda."
source /home/travis/miniconda3/bin/activate root
conda config --remove channels defaults
conda config --add channels defaults
conda config --add channels conda-forge
conda config --set show_channel_urls true

install:
- conda install -y -q jinja2 pygit2 click pyyaml setuptools_scm

script:
# fetch all branches of crossbow
- git config remote.origin.fetch +refs/heads/*:refs/remotes/origin/*

# clone arrow with crossbow tool
- pushd ..
- git clone -b {{ ARROW_BRANCH }} {{ ARROW_REPO }}

# submit packaging tasks
- |
python arrow/dev/tasks/crossbow.py \
conda-linux \
conda-win \
conda-osx \
wheel-linux \
wheel-win \
wheel-osx \
linux-packages
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure about this. I was thinking we would want to schedule nightlies using a cronjob of some kind. Or at least the manifest of nightly jobs would be specified in a YAML file someplace

Copy link
Member Author

@kszucs kszucs Jun 12, 2018 •
edited

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This would be just for convenience, to use travis' cron instead of a self hosted one.
BTW I'm just refactoring the branching logic to support tagging / deploying (github releases).

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems like a fine place to keep the nightly jobs list for now. There's nothing preventing us from moving this to a non-travis cron in the future if there's some reason we would need to do that.