scikit-ci-yml.rst

Configuration file

The configuration file is read by the scikit-ci executable to find out which commands to execute for a given step.

The configuration file should named scikit-ci.yml and is usually added to the root of a project.

It is a YAML file that can be validated against scikit-ci-schema.yml.

Concept of Step

A step consist of a list of commands and optional key/value pairs describing the environment.

More specifically, a step can be described using the following structure:

before_install:
  environment:
    FOO: bar
  commands:
    - echo "Hello world"

where before_install can be replaced by any of these:

• before_install
• install
• before_build
• build
• test
• after_test

Mapping with Appveyor, Azure Pipelines, CircleCI and TravisCI steps

scikit-ci do not impose any particular mapping.

Documentation specific to each services is available here:

• Appveyor build pipeline
• Azure pipelines
• CircleCI configuration 2.0
• CircleCI configuration 1.0 (deprecated)
• TravisCI build lifecycle

Reported below are some recommended associations that are know to work.

• appveyor.yml:

../appveyor.yml

Note

Since on windows the ci executable is installed in the Scripts directory (e.g C:\Python27\Scripts\ci.exe) which is not in the PATH by default, the python -m ci syntax is used.

• azure-pipelines.yml:

../azure-pipelines.yml

• .circleci/config.yml (CircleCI 2.0):

../.circleci/config.yml

• circle.yml (CircleCI 1.0):

circle-v1-yml.txt

• .travis.yml

../.travis.yml

Order of steps

scikit-ci execute steps considering the following order:

1. before_install
2. install
3. before_build
4. build
5. test
6. after_test

This means that the mapping specified <step_mapping> in the continuous integration file has to be done accordingly.

Automatic execution of dependent steps

Considering the step ordering <step_order>, executing any step(n) ensures that step(n-1) has been executed before.

Keeping track of executed steps

scikit-ci keeps track of executed steps setting environment variables of the form SCIKIT_CI_<STEP_NAME> where <STEP_NAME> is any of the step name in upper-case.

Note

Specifying the command line option --force allows to force the execution of the steps ignoring the values of the SCIKIT_CI_<STEP_NAME> environment variables.

Environment variable persistence

Environment variable defined in any given step are always guaranteed to be set in steps executed afterward.

This is made possible by serializing the environment on the filesystem.

Note

After executing steps, a file named env.json is created in the current directory along side scikit-ci.yml. This is where the environment is cached for re-use in subsequent steps.

Specifying the command line option --clear-cached-env allows to execute steps after removing the env.json file.

Step specialization

For any given step, it is possible to specify commands and environment variables specific to each continuous integration service.

Recognized services are:

• appveyor
• azure
• circle
• travis

Commands

commands common to all services are executed first, then commands specific to each services are executed.

For example, considering this configuration used on CircleCI and TravisCI:

before_install:
  commands:
    - echo "Hello Everywhere"

  circle:
    commands:
      - echo "Hello on CircleCI"

  travis:
    linux:
      commands:
        - echo "Hello on TravisCI"

The output on the different service will be the following:

• CircleCI:

Hello Everywhere
Hello on CircleCI

• TravisCI:

Hello Everywhere
Hello on TravisCI

Note

Sections command_specification and python_command_specification describe the different types of command.

Environment

Similarly, environment can be overridden for each service.

For example, considering this configuration used on CircleCI and TravisCI:

before_install:

  circle:
    environment:
      CATEGORY_2: 42

  travis:
    linux:
      environment:
        CATEGORY_1: 99

  environment:
    CATEGORY_1: 1
    CATEGORY_2: 2

  commands:
    - echo "CATEGORY_1 is ${CATEGORY_1}"
    - echo "CATEGORY_2 is ${CATEGORY_2}"

The output on the different service will be the following:

• on CircleCI:

CATEGORY_1 is 1
CATEGORY_2 is 42

• on TravisCI:

CATEGORY_1 is 99
CATEGORY_2 is 2

Reserved Environment Variables

• CI_NAME: This variable is automatically set by scikit-ci and will contain the name of the continuous integration service currently executing the step.

Environment variable usage

To facilitate the use of environment variable across interpreters, scikit-ci uses a specific syntax.

Environment variable specified using $<NAME_OF_VARIABLE> in both commands and environment variable will be expanded.

For example, considering this configuration used on Appveyor, CircleCI and TravisCI:

before_install:

  appveyor:
    environment:
      TEXT: Windows$<TEXT>

  travis:
    linux:
      environment:
        TEXT: LinuxWorld

  environment:
    TEXT: World

  commands:
    - echo $<TEXT>

The output on the different service will be the following:

• on Appveyor:

WindowsWorld

• on CircleCI:

World

• on TravisCI:

LinuxWorld

Note

On system having a POSIX interpreter, the environment variable will NOT be expanded if included in string start with a single quote.

ci.driver.Driver

Command Specification

Specifying command composed of a program name and arguments is supported on all platforms.

For example:

test:
  commands:
    - echo "Hello"
    - python -c "print('world')"
    - git clone git://github.com/scikit-build/scikit-ci

On unix based platforms (e.g CircleCI and TravisCI), commands are interpreted using bash.

On windows based platform (e.g Appveyor), commands are interpreted using the windows command terminal cmd.exe.

Since both interpreters expand quotes differently, we recommend to avoid single quoting argument. The following table list working recipes:

	CircleCi, TravisCI	Appveyor
scikit-ci command	bash output	cmd output
echo Hello1	Hello1	Hello1
echo "Hello2"	Hello2	"Hello2"
echo 'Hello3'	Hello3	'Hello3'
python -c "print('Hello4')"	Hello4	Hello4
python -c 'print("Hello5")'	Hello5	no output
python -c "print('Hello6\'World')"	Hello6'World	Hello6'World

And here are the values associated with sys.argv for different scikit-ci commands:

python program.py --things "foo" "bar" --more-things "doo" 'dar'

Output on CircleCi, TravisCI:

arg_1 [--things]
arg_2 [foo]
arg_3 [bar]
arg_4 [--more-things]
arg_5 [doo]
arg_6 [dar]

Output on Appveyor:

arg_1 [--things]
arg_2 [foo]
arg_3 [bar]
arg_4 [--more-things]
arg_5 [doo]
arg_6 ['dar']    # <-- Note the presence of single quotes

python program.py --things "foo" "bar" --more-things "doo" 'dar'

Output on CircleCi, TravisCI:

arg_1 [--the-foo=foo]
arg_2 [-the-bar=bar]

Output on Appveyor:

arg_1 [--the-foo=foo]
arg_2 [-the-bar='bar']    # <-- Note the presence of single quotes

Note

Here are the source of program.py:

import sys
for index, arg in enumerate(sys.argv):
    if index == 0:
        continue
    print("arg_%s [%s]" % (index, sys.argv[index]))

Python Command Specification

0.10.0

The python commands are supported on all platforms.

For example:

test:
  commands:
    - python: print("single_line")
    - python: "for letter in ['a', 'b', 'c']: print(letter)"
    - python: |
              import os
              if 'FOO' in os.environ:
                  print("FOO is set")
              else:
                  print("FOO is *NOT* set")

Note

By using os.environ, they remove the need for specifying environment variable using the $<NAME_OF_VARIABLE> syntax described in environment_variable_usage.