Skip to content
A Concourse CI resource for Python PyPI packages.
Branch: master
Clone or download
Latest commit 910e2a6 Apr 13, 2019
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
.vscode provide debug configuration and entry points for vscode Jul 10, 2018
ci Rename job to reflect it does a get as well. Sep 21, 2017
pypi_resource Simplify URL-mangling logic. Apr 11, 2019
test
tmp metadata output and integration testing Jul 10, 2018
.env migration to use pip library codebase for in Jul 5, 2018
.gitignore additional ignored files Jul 8, 2018
.project cleanup Jul 9, 2018
.pydevproject cleanup Jul 9, 2018
.version Bump minor version. Apr 11, 2019
Dockerfile bump to specific version, save image layers Jul 3, 2018
LICENSE Fix copyright year. Dec 5, 2016
MANIFEST.in
NOTICE Add clause to notice. Dec 5, 2016
Pipfile Update dependencies. Apr 14, 2019
Pipfile.lock Update dependencies. Apr 14, 2019
README.md Fix PyPI URLs. Apr 11, 2019
makefile Restore main line docker prefix. Apr 11, 2019
setup.py cleanup Jul 9, 2018

README.md

PyPI Package Resource

A Concourse CI resource for Python PyPI packages. This is an enhanced fork of cf-platform-eng/concourse-pypi-resource, that adds compatibility to most PyPI repository types and extended package selection parameters.

It can be used to check/download existing packages and to manages your own builds as well. Internally it uses Pip 10.0.1 for check and in (downloads) and [twine] for output.

Docker image publicly available on Docker Hub: https://hub.docker.com/r/punkadiddle/concourse-pypi-resource.

Resource Configuration

Parameter default status description
PACKAGE SELECTION
name - required name of the package
pre_release false optional check dev and pre-release versions (see PEP-440)
release true optional check release versions
filename_match -/- optional only include packages containing this string (e.g. py2.py3, .whl)
packaging any optional only include source or binary (or any) packages
platform -/- optional only include releases compatible with this platform (implicit default will be the os used for Concourse's workers)
python_abi -/- optional TODO
python_implementation -/- optional TODO
python_version -/- optional only include packages compatible with this Python interpreter version number (see pip's --python-version)
REPOSITORY
repository.test false optional set to true as shortcut to use the PyPI test server for index_url and repository_url
repository.index_url PyPI optional url to a pip compatible index for check and download
repository.repository_url PyPI optional url to a twine compatible repository for upload
repository.username -/- req. for uploads username for PyPI server authentication
repository.password -/- req. for uploads password for PyPI server authentication
repository.authenticate out optional set to always to authenticate to a private repository for check and download also

Deprecated parameters (since version 0.2.0)

  • python_version: gets mapped to filename_match if it's not a version number. python_version is now only used for the actual interpreter version to have a transparent mapping to pip.
  • repository: (special index-server name if it is specified in ~/.pypirc). This is no longer available to the current implementation of check and in. Also there's no way to inject a .pypirc file into this Concourse resource type.
  • repository, test, username and password: get mapped to repository.<key>. This allows to configure private repositories through a single yaml-map parameter value, thus removing redundancy from the pipeline.

Examples

resource_types:
- name: pypi
  type: docker-image
  source:
    repository: punkadiddle/concourse-pypi-resource

resources:
- name: my-public-package
  type: pypi
  source:
    name: my_package
    packaging: any
    repository:
      username: user
      password: pass

- name: my-private-package
  type: pypi
  source:
    name: my_package
    pre_release: true
    # In a live pipeline you would probably set the while repository configuration
    # through a single yaml parameter:
    # repository: ((my-private-repository))
    repository:
      authenticate: always
      index_url: http://nexus.local:8081/repository/pypi-private/simple
      repository_url: http://nexus.local:8081/repository/pypi-private/
      username: admin
      password: admin123

get: Download the latest version

No parameters. TODO

  • version.version: Optional, defaults to latest version

Additional files populated

  • version: Python version number of the downloaded package
  • semver: Semver-formatted version number that can be processed with a Concourse SemVer Resource.

Example

plan:
- get: my-public-package
- get: my-private-package
  version: {version: '4.2'}

put: Upload a new version

  • glob: Required A glob expression matching the package file to upload.

Example

plan:
- put: my-pypi-package
  params:
    glob: 'task-out-folder/my_package-*.tar.gz'

Development

To run the unit tests, go to the root of the repository and run:

# install pipenv
pip3 install --user pipenv
# setup and run unittests
make test

To build the docker image for the resource:

# package
make dist
# optionally upload
make push

Run local test runs like this:

# check
docker run -i concourse-pypi-resource:latest-rc check < test/input/check-nexus.json | jq
# in
docker run --rm -i --volume destdir concourse-pypi-resource:latest-rc in destdir < test/input/in-nexus.json | jq

Private repository integration tests (using Sonatype Nexus 3)

  • Spin-up a docker instance of Nexus 3:
    docker run -d -p 8081:8081 --name nexus sonatype/nexus3
  • Create pypi hosted repositories called pypi-private and pypi-release (enable redeploy).
  • Disable user anonymous access.
# ensure test-packages are built
make test
pipenv run python -m pytest "test/nexus-integration.py"

Test using Concourse CI. Check that the provided params match your repository.

fly -t test sp -p demo -c test/ci/pipeline.yml -l test/ci/params.yml

Public Integration Tests

# provide login data to be able to upload concourse-pypi-resource
export TWINE_USERNAME=<...>
export TWINE_PASSWORD=<...>
python -m pytest -v test/pypi-integration.py 

# or skip the upload
python -m pytest -v -k 'not test_upload' test/pypi-integration.py 
You can’t perform that action at this time.