Skip to content

A Concourse CI resource for Python PyPI packages.

License

Notifications You must be signed in to change notification settings

Pix4D/concourse-pypi-resource

 
 

Repository files navigation

PyPI Package Resource

A Concourse CI resource for Python PyPI packages.

It can be used to check/download existing packages and to manages your own builds as well. Internally it uses Pip 10.0.1 for check and in (downloads) and twine for output.

Docker image publicly available on Docker Hub: https://hub.docker.com/r/cfplatformeng/concourse-pypi-resource.

Resource Configuration

Parameter default status description
PACKAGE SELECTION
name - required name of the package
name_must_match true optional require the project name and the packge name to match (see PEP-423)
pre_release false optional check dev and pre-release versions (see PEP-440)
release true optional check release versions
filename_match -/- optional only include packages containing this string (e.g. py2.py3, .whl)
packaging any optional only include source or binary (or any) packages
platform -/- optional only include releases compatible with this platform (implicit default will be the os used for Concourse's workers)
python_abi -/- optional TODO
python_implementation -/- optional TODO
python_version -/- optional only include packages compatible with this Python interpreter version number (see pip's --python-version)
REPOSITORY
repository.test false optional set to true as shortcut to use the PyPI test server for index_url and repository_url
repository.index_url PyPI optional url to a pip compatible index for check and download
repository.repository_url PyPI optional url to a twine compatible repository for upload
repository.username -/- req. for uploads username for PyPI server authentication
repository.password -/- req. for uploads password for PyPI server authentication
repository.authenticate out optional set to in to authenticate to a private repo for check and download only, always to authenticate to a private repository for upload, check and download.

Deprecated parameters (since version 0.2.0)

  • repository: (special index-server name if it is specified in ~/.pypirc). This is no longer available to the current implementation of check and in. Also there's no way to inject a .pypirc file into this Concourse resource type.
  • repository, test, username and password: get mapped to repository.<key>. This allows to configure private repositories through a single yaml-map parameter value, thus removing redundancy from the pipeline.

Examples

resource_types:
- name: pypi
  type: registry-image
  source:
    repository: cfplatformeng/concourse-pypi-resource

resources:
- name: my-public-package
  type: pypi
  source:
    name: my_package
    packaging: any
    repository:
      username: user
      password: pass

- name: my-private-package
  type: pypi
  source:
    name: my_package
    pre_release: true
    # In a live pipeline you would probably set the while repository configuration
    # through a single yaml parameter:
    # repository: ((my-private-repository))
    repository:
      authenticate: always
      index_url: http://nexus.local:8081/repository/pypi-private/simple
      repository_url: http://nexus.local:8081/repository/pypi-private/
      username: admin
      password: admin123

get: Download the latest version

  • version.version: Optional, defaults to latest version
  • count_retries: Optional Number of maximum retry before the task fails. By default 20 times.
  • delay_between_retries: Optional Time to wait in sec between two iterations of retry. By default 3s.

Additional files populated

  • version: Python version number of the downloaded package
  • semver: Semver-formatted version number that can be processed with a Concourse SemVer Resource.

Example

plan:
- get: my-public-package
- get: my-private-package
  version: {version: '4.2'}

put: Upload a new version

  • glob: Required A glob expression matching the package file to upload.

Note

You can modify count_retries and delay_between_retries in get_params to give enough time to PyPi to make available your package.

Example

plan:
- put: my-pypi-package
  params:
    glob: 'task-out-folder/my_package-*.tar.gz'
  get_params:
    count_retries: 10
    delay_between_retries: 30

Development

To run the unit tests, go to the root of the repository and run:

# install pipenv
pip3 install --user pipenv
# setup and run unittests
make test

To build the docker image for the resource:

# package
make dist
# optionally upload
make push

Run local test runs like this (nexus):

# check
docker run -i cfplatformeng/concourse-pypi-resource:latest-rc check < test/input/check-nexus.json | jq
# in
docker run --rm -i --volume $(pwd)/output:/destdir cfplatformeng/concourse-pypi-resource:latest-rc in /destdir < test/input/in-nexus.json | jq

or using public PyPi:

# check
docker run -i cfplatformeng/concourse-pypi-resource:latest-rc check < test/input/check-public.json | jq
# in
docker run --rm -i --volume $(pwd)/output:/destdir cfplatformeng/concourse-pypi-resource:latest-rc in /destdir < test/input/in-public.json | jq

Private repository integration tests (using Sonatype Nexus 3)

  • Spin-up a docker instance of Nexus 3:
    docker run -d -p 8081:8081 --name nexus sonatype/nexus3
  • Fetch the default admin password
    docker exec -it nexus cat /nexus-data/admin.password
  • Use this password to log in for the first time as admin user
  • During initial Setup wizard:
    • Change the password to admin123, as configured in test/nexus-integration.py.
    • Disable user anonymous access.
  • Create pypi hosted repositories called pypi-private and pypi-release (enable redeploy).
# ensure test-packages are built
make test
pipenv run python -m pytest "test/nexus-integration.py"

Test using Concourse CI. Check that the provided params match your repository.

fly -t test sp -p demo -c test/ci/pipeline.yml -l test/ci/params.yml

Public Integration Tests

# provide login data to be able to upload concourse-pypi-resource
export TWINE_USERNAME=<...>
export TWINE_PASSWORD=<...>
python -m pytest -v test/pypi-integration.py 

# or skip the upload
python -m pytest -v -k 'not test_upload' test/pypi-integration.py 

Releases

No releases published

Packages

No packages published

Languages

  • Python 95.2%
  • Makefile 4.1%
  • Dockerfile 0.7%