DMWM Packaging with PyPi

This document outlines the transition of DMWM software to the Python Package Index, https://pypi.org/

General Information

DMWM PyPi account: cms-oc-dmwm

Erik Gough and Andrew Melo have the password if needed.

General docs for packaging projects: https://packaging.python.org/tutorials/packaging-projects/

Test packages location: https://test.pypi.org/user/cms-oc-dmwm/

Production packages location: https://www.pypi.org/user/cms-oc-dmwm/

Create a virtualenv for building packages. Need setuptools and twine.

$ virtualenv pypibuild
$ source pypibuild/bin/activate
(pypibuild) $ pip install twine

Packaging and uploading WMAgent to pypi

Clone the WMCore repo and fetch the set of tags

$ git clone https://github.com/dmwm/WMCore.git
$ cd WMCore
$ git fetch && git fetch --tags

Checkout what tag you want

$ git checkout <tag>

Create a source tarball for distribution

(pypibuild) $ python setup.py sdist

Upload to test pypi using twine (username and password required)

(pypibuild) $ twine upload --repository-url https://test.pypi.org/legacy/ dist/wmagent-<tag>.tar.gz

Upload to production pypi using twine (username and password required)

(pypibuild) $ twine upload dist/wmagent-<tag>.tar.gz

WMAgent Dependencies

This is the list of dependencies that are required before one can install wmagent via pip. Python dependencies are automatically pulled when the wmagent package is installed.

• MariaDB
• CouchDB
• Oracle Client
• Condor
• libcurl-devel
• jemalloc
• libuuid
• yui (might not be necessary)
• pystack (might not be necessary)
• gcc

If any of these packages are installed in a non-standard location, you will need to set some environment variables as well.

• Add mariadb bin and gcc bin directory to PATH
• Add gcc lib directory to LD_LIBRARY_PATH
• Set ORACLE_HOME to Oracle client location

Download and install WMAgent

Create a wmagent virtualenv

$ virtualenv wmagent
$ source wmagent/bin/activate
(wmagent) $ pip install wmagent

Minimalistic setup.py example for DBS client.

This provides the limited necessary pieces for package distribution with an example of how to pull python dependencies.

#!/usr/bin/env python
from distutils.core import setup
setup(name = 'dbs-client',
        version = '3.3.160',
        maintainer = 'CMS DWMWM Group',
        maintainer_email = 'hn-cms-dmDevelopment@cern.ch',
        packages = ['dbs',
                    'dbs.apis',
                    'dbs.exceptions'],
        package_dir = {'' : 'Client/src/python/'},
        install_requires = ['pycurl-client'],
        url = "https://github.com/dmwm/DBS",
    )