Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ARROW-17512: [Doc] Updates to crossbow documentation for clarity #13993

Merged
merged 23 commits into from
Sep 29, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
bba3137
Modify wording for Travis CI 'auto cancellation' setting to indicate …
lafiona Aug 29, 2022
1218d16
Update link for Travis CI from travis-ci.org to travis-ci.com
kevingurney Aug 29, 2022
1736528
Update broken link to Jinja documentation.
kevingurney Aug 29, 2022
f013474
Fix typo.
kevingurney Aug 29, 2022
8dad119
Add link to ursacomputing/crossbow repository.
kevingurney Aug 29, 2022
2ae3354
Fixed broken link to Travis CI's auto cancellation feature and minor …
lafiona Aug 29, 2022
f40353a
Fix indentation of crossbow CI settings section
lafiona Sep 1, 2022
dcdebc6
Fix link to ursacomputing/crossbow repository.
kevingurney Sep 1, 2022
7cc7174
Fix indentation of crossbow CI settings section 2
lafiona Sep 1, 2022
cb0e6af
Fix nested list of environment variables for CI services.
kevingurney Sep 1, 2022
8f11851
Improve wording for Travis CI auto cancellation feature default.
lafiona Sep 1, 2022
8138e57
Modify wording, for submitting manual jobs, in overview.rst.
lafiona Sep 2, 2022
e6ff42c
Fix typo in link to Travis CI "Getting Started" page.
kevingurney Sep 27, 2022
ba5b67f
Remove Circle CI and update GitHub new repository link
lafiona Sep 27, 2022
e7c0926
Add line-block directive to add more spacing after nested list.
kevingurney Sep 27, 2022
2e3eabf
Spelling of Mac/mac -> macOS
lafiona Sep 28, 2022
325b3ff
Spelling of TravisCI -> Travis CI
lafiona Sep 28, 2022
366cdf8
Remove mentions of AppVeyor from the Crossbow documentation.
kevingurney Sep 28, 2022
9dd3b85
Add link for and reference to GitHub Actions in Queue section of doc …
lafiona Sep 28, 2022
a261cae
Add clarification about queue repository name convention in Queue sub…
lafiona Sep 28, 2022
88a9ed1
Add GitHub Actions to the list of public CI services for all platforms
lafiona Sep 28, 2022
5e49a21
Adjust the wording of step 6, as the only service that needs to be co…
lafiona Sep 28, 2022
aea90d2
Adjust wording in step 3 to clarify
lafiona Sep 28, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
64 changes: 29 additions & 35 deletions docs/source/developers/continuous_integration/crossbow.rst
Original file line number Diff line number Diff line change
Expand Up @@ -25,8 +25,8 @@ The content of ``arrow/dev/tasks`` directory aims for automating the process of
Arrow packaging and integration testing.

Packages:
- C++ and Python `conda-forge packages`_ for Linux, Mac and Windows
- Python `Wheels`_ for Linux, Mac and Windows
- C++ and Python `conda-forge packages`_ for Linux, macOS and Windows
- Python `Wheels`_ for Linux, macOS and Windows
- C++ and GLib `Linux packages`_ for multiple distributions
- Java for Gandiva

Expand All @@ -46,21 +46,21 @@ Executors

Individual jobs are executed on public CI services, currently:

- Linux: TravisCI, CircleCI, Azure Pipelines
- Mac: TravisCI, Azure Pipelines
- Windows: AppVeyor, Azure Pipelines
- Linux: GitHub Actions, Travis CI, Azure Pipelines
- macOS: GitHub Actions, Travis CI, Azure Pipelines
- Windows: GitHub Actions, Azure Pipelines

pitrou marked this conversation as resolved.
Show resolved Hide resolved
Queue
~~~~~

Because of the nature of how the CI services work, the scheduling of
jobs happens through an additional git repository, which acts like a job
queue for the tasks. Anyone can host a ``queue`` repository which is usually
called as ``crossbow``.
queue for the tasks. Anyone can host a ``queue`` repository (usually
named ``<ghuser>/crossbow``).

A job is a git commit on a particular git branch, containing only the required
configuration file to run the requested build (like ``.travis.yml``,
``appveyor.yml`` or ``azure-pipelines.yml``).
A job is a git commit on a particular git branch, containing the required
configuration files to run the requested builds (like ``.travis.yml``,
``azure-pipelines.yml``, or ``crossbow.yml`` for `GitHub Actions`_ ).

Scheduler
~~~~~~~~~
Expand All @@ -74,22 +74,20 @@ Install
The following guide depends on GitHub, but theoretically any git
server can be used.

If you are not using the `ursacomputing/crossbow <https://github.com/ursacomputing/crossbow>`_
If you are not using the `ursacomputing/crossbow`_
repository, you will need to complete the first two steps, otherwise procede
to step 3:

1. `Create the queue repository`_

2. Enable `TravisCI`_, `Appveyor`_, `Azure Pipelines`_ and `CircleCI`_
integrations on for the newly created queue repository.
2. Enable `Travis CI`_ and `Azure Pipelines`_ integrations for the newly
created queue repository.

- turn off Travis’ `auto cancellation`_ feature on branches

3. Clone either ursacomputing/crossbow if you are using that, or the newly
3. Clone either `ursacomputing/crossbow`_ if you are using that, or the newly
created repository next to the arrow repository:

By default the scripts looks for ``crossbow`` next to arrow repository, but
this can configured through command line arguments.
By default the scripts looks for a ``crossbow`` clone next to the ``arrow``
directory, but this can configured through command line arguments.

.. code:: bash

Expand All @@ -110,22 +108,18 @@ to step 3:

or pass as an argument to the CLI script ``--github-token``

6. Export the previously created GitHub token on both CI services:
6. Add the previously created GitHub token to **Travis CI**:

Use ``CROSSBOW_GITHUB_TOKEN`` encrypted environment variable. You can
set them at the following URLs, where ``ghuser`` is the GitHub
set it at the following URL, where ``ghuser`` is the GitHub
username and ``ghrepo`` is the GitHub repository name (typically
``crossbow``):

- TravisCI: ``https://travis-ci.org/<ghuser>/<ghrepo>/settings``
- Appveyor:
``https://ci.appveyor.com/project/<ghuser>/<ghrepo>/settings/environment``
- CircleCI:
``https://circleci.com/gh/<ghuser>/<ghrepo>/edit#env-vars``

On Appveyor check the ``skip branches without appveyor.yml`` checkbox
on the web UI under crossbow repository’s settings.
``https://travis-ci.com/<ghuser>/<ghrepo>/settings``

- Confirm the `auto cancellation`_ feature is turned off for branch builds. This should be the default setting.

.. line-block::
7. Install Python (minimum supported version is 3.7):

Miniconda is preferred, see installation instructions:
Expand Down Expand Up @@ -176,8 +170,8 @@ The script does the following:
3. Reads and renders the required build configurations with the
parameters substituted.

4. Create a branch per task, prefixed with the job id. For example to
build conda recipes on linux it will create a new branch:
4. Create a branch per task, prefixed with the job id. For example, to
build conda recipes on linux, it will create a new branch:
``crossbow@build-<id>-conda-linux``.

5. Pushes the modified branches to GitHub which triggers the builds. For
Expand Down Expand Up @@ -251,11 +245,11 @@ see its help page:
.. _conda-forge packages: conda-recipes
.. _Wheels: python-wheels
.. _Linux packages: linux-packages
.. _Create the queue repository: https://help.github.com/articles/creating-a-new-repository
.. _TravisCI: https://travis-ci.org/getting_started
.. _Appveyor: https://www.appveyor.com/docs/
.. _CircleCI: https://circleci.com/docs/2.0/getting-started/
.. _Create the queue repository: https://docs.github.com/en/repositories/creating-and-managing-repositories/creating-a-new-repository
.. _Github Actions: https://docs.github.com/en/actions/quickstart
.. _Travis CI: https://travis-ci.com/getting-started/
.. _Azure Pipelines: https://docs.microsoft.com/en-us/azure/devops/pipelines/get-started/pipelines-sign-up
.. _auto cancellation: https://docs.travis-ci.com/user/customizing-the-build/#Building-only-the-latest-commit
.. _auto cancellation: https://docs.travis-ci.com/user/customizing-the-build/#building-only-the-latest-commit
.. _Create a Personal Access Token: https://help.github.com/articles/creating-a-personal-access-token-for-the-command-line/
.. _setuptools_scm: https://pypi.python.org/pypi/setuptools_scm
.. _ursacomputing/crossbow: https://github.com/ursacomputing/crossbow
6 changes: 3 additions & 3 deletions docs/source/developers/continuous_integration/overview.rst
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ Some files central to Arrow CI are:

We use :ref:`Docker<docker-builds>` in order to have portable and reproducible Linux builds, as well as running Windows builds in Windows containers. We use :ref:`Archery<Archery>` and :ref:`Crossbow<Crossbow>` to help co-ordinate the various CI tasks.

One thing to note is the some of the services defined in ``docker-compose.yml`` are interdependent. When running services locally, you must either manually build its dependencies first, or build it via the use of ``archery run ...`` which automatically finds and builds dependencies.
One thing to note is that some of the services defined in ``docker-compose.yml`` are interdependent. When running services locally, you must either manually build its dependencies first, or build it via the use of ``archery run ...`` which automatically finds and builds dependencies.

There are numerous important directories in the Arrow project which relate to CI:

Expand Down Expand Up @@ -69,8 +69,8 @@ Extended builds
Crossbow is a subcomponent of Archery and can be used to manually trigger builds. The tasks which can be run on Crossbow can be found in the ``dev/tasks`` directory. This directory contains:

- the file ``dev/tasks/tasks.yml`` containing the configuration for various tasks which can be run via Crossbow
- subdirectories containing different task templates (specified using `jinja2 syntax <https://jinja2docs.readthedocs.io/en/stable/>`_), divided roughly by language or package management system.
- subdirectories containing different task templates (specified using `jinja2 syntax <https://jinja.palletsprojects.com/>`_), divided roughly by language or package management system.

Most of these tasks are run as part of the nightly builds, though also can be triggered manually by add a comment to a PR which begins with ``@github-actions crossbow submit`` followed by the name of the task to be run.
Most of these tasks are run as part of the nightly builds, though they can also be triggered manually by add a comment to a PR which begins with ``@github-actions crossbow submit`` followed by the name of the task to be run.

For convenience purpose, the tasks in ``dev/tasks/tasks.yml`` are defined in groups, which makes it simpler for multiple tasks to be submitted to Crossbow at once. The task definitions here contain information about which service defined in ``docker-compose.yml`` to run, the CI service to run the task on, and which template file to use as the basis for that task.