Skip to content

DMWM Packaging with PyPi

Jump to bottom
Alan Malta Rodrigues edited this page Feb 6, 2019 · 19 revisions

This document outlines the transition of DMWM software to the Python Package Index, https://pypi.org/

General Information

DMWM PyPi account: cms-oc-dmwm

Erik Gough and Andrew Melo have the password if needed.

General docs for packaging projects: https://packaging.python.org/tutorials/packaging-projects/

Test packages location: https://test.pypi.org/user/cms-oc-dmwm/

Production packages location: https://www.pypi.org/user/cms-oc-dmwm/

Create a virtualenv for building packages. Need setuptools and twine.

$ virtualenv pypibuild
$ source pypibuild/bin/activate
(pypibuild) $ pip install twine

Packaging and uploading WMAgent to pypi

Clone the WMCore repo and fetch the set of tags

$ git clone https://github.com/dmwm/WMCore.git
$ cd WMCore
$ git fetch && git fetch --tags

Checkout what tag you want

$ git checkout <tag>

Create a source tarball for distribution

(pypibuild) $ python setup.py sdist

Upload to test pypi using twine (username and password required)

(pypibuild) $ twine upload --repository-url https://test.pypi.org/legacy/ dist/wmagent-<tag>.tar.gz

Upload to production pypi using twine (username and password required)

(pypibuild) $ twine upload dist/wmagent-<tag>.tar.gz

WMAgent Dependencies

This is the list of dependencies that are required before one can install wmagent via pip. Python dependencies are automatically pulled when the wmagent package is installed.

  • MariaDB
  • CouchDB
  • Oracle Client
  • Condor
  • libcurl-devel
  • jemalloc
  • libuuid
  • yui (might not be necessary)
  • pystack (might not be necessary)
  • gcc

If any of these packages are installed in a non-standard location, you will need to set some environment variables as well.

  • Add mariadb bin and gcc bin directory to PATH
  • Add gcc lib directory to LD_LIBRARY_PATH
  • Set ORACLE_HOME to Oracle client location

Download and install WMAgent

Create a wmagent virtualenv

$ virtualenv wmagent
$ source wmagent/bin/activate
(wmagent) $ pip install wmagent

Packaging DBS Applications

Instructions on how to build and upload DBS Client and Pycurl Client to PyPi.

One time setup for users wanting to upload new releases to PyPi

Create a .pypirc file in your home directory with the following contents:

[distutils]
index-servers =
  pypi

[pypi]
repository: https://upload.pypi.org/legacy/
username: cms-oc-dmwm

Changes to be made at release time

The following changes should be made when a new version of DBS is released. Some version numbers will have to be updated manually for now. This will (hopefully) be automated in the future.

  1. dbs-client version in setup_dbs_client.py, L10
  2. pycurl-client version in setup_pycurl_client.py, L10
  3. pycurl-client version in requirements.dbs_client.py, L1

Build and upload to PyPi

Clone the DBS repo and checkout the latest tag:

$ git clone https://github.com/dmwm/DBS.git
$ git fetch && git fetch --tags
$ git checkout <tag>

Run the build_pypi_packages.sh script build and upload the packages. The script takes the following options, but you will usually want to build "all".

  all              Build all DBS packages
  dbs-client       Build dbs-client package
  pycurl-client    Build pycurl-client package

$ sh Tools/build_pypi_packages.sh all

Enter the cms-oc-dmwm account password when prompted. This can also be set in the .pypirc file if you don't mind storing passwords in clear text.

Check the details of the new build here: https://pypi.org/user/cms-oc-dmwm/

Installing DBS client and Pycurl client via PyPI

Make sure you have pip available in your system - it also asks for libcurl development on a test CC7 box - otherwise you can install it with:

sudo yum install python-pip libcurl-devel

you'll also need virtualenv in order to isolate this python environment, which can be installed with

sudo pip install virtualenv

Make a new virtual environment, activate it, and install the dbs-client package.

$ virtualenv dbsclient
$ source dbsclient/bin/activate
(dbsclient)$ pip install dbs-client
Collecting dbs-client
<snip>
Successfully installed dbs-client-3.7.X pycurl-7.19.0.3 pycurl-client-3.7.X python-cjson-1.2.1

Create a proxy

$ voms-proxy-init -rfc -voms cms

Test access with a simple python script

$ cat dbs-test.py
from dbs.apis.dbsClient import DbsApi
dbs = DbsApi('https://cmsweb.cern.ch/dbs/prod/global/DBSReader')
ds = dbs.listDatasets(dataset = '/MinBias/Summer*/*')
print(ds)

If you receive error 35 from curl/pycurl you will need to set your shared library path to include openssl built by CMS. We will need to figure out how to overcome this in a more automated way.

$ export LD_LIBRARY_PATH=/cvmfs/cms.cern.ch/slc6_amd64_gcc481/cms/cmssw/CMSSW_7_2_2/external/slc6_amd64_gcc481/lib
$ python dbs-test.py 
[{'dataset': '/MinBias/Summer09-MC_31X_V3-v1_HLT_AlCa1E31-v1/RAW'}, {'dataset': '/MinBias/Summer09-MC_31X_V3-v1_HLT_AlCa8E29-v1/RAW'}, {'dataset': '/MinBias/Summer09-MC_31X_V3_CS_DiJetAve15U-v1/USER'}, {'dataset': '/MinBias/Summer09-MC_31X_V3_7TeV_CS_DiJetAve15U-v1/USER'}, {'dataset': '/MinBias/Summer11-DESIGN42_V11_428_SLHC1-v1/GEN-SIM'}, {'dataset': '/MinBias/Summer11-DESIGN42_V11_428_SLHChcal-v1/GEN-SIM'}]
Clone this wiki locally