Skip to content

Docker Integration Test

Jump to bottom Edit New page
Kevin Pedro edited this page Oct 17, 2022 · 2 revisions
Status Link
TreeMaker CI GitHub Actions
Tag Size Pulls

Introduction

A variety of Docker images, which contain the TreeMaker source code, have been created. These images will allow you to run a reproducible/consistent environment and the ability to use TreeMaker as a ServiceX transformer. The essential difference between all of the images is whether they are based on installing CMSSW from CVMFS or if they have CMSSW installed directly inside the image. Each have benefits and disadvantages and the choice of which one to use will depend on your use case. However, from a TreeMaker standpoint, these images are identical.

Specifications

CVMFS Based Images

A non-exhaustive list of features of this image are as follows:

  1. Based on the cms-cvmfs-docker:light images, which is itself based on Scientific Linux 7.
  2. Capable of mounting multiple CernVM File System (CVMFS) directories:
    1. cms.cern.ch
    2. cms-ib.cern.ch
    3. oasis.opensciencegrid.org
    4. cms-lpc.opensciencegrid.org
    5. sft.cern.ch
    6. cms-bril.cern.ch
    7. cms-opendata-conddb.cern.ch
  3. The ability to set the UID and GID of the cmsuser inside the container
  4. setup.sh has been run within /home/cmsuser/, setting up a CMSSW_10_2_21 release and all of the associated TreeMaker software
  5. There are three versions of the image:
    • The CVMFS cache has been cleared (tag: Run2_2017-latest).
    • The CVMFS cache has been cleared and the dependencies for ServiceX have been installed (tag: Run2_2017-servicex-cvmfs)
    • The CVMFS cache contains the pieces needed to run an integration test (tag: Run2_2017-cached). This image is quite a bit bigger than the version where the cache has been cleared.

Standalone Images

The Standalone image should behave similarly to the CVMFS base images, but obviously without the ability to pull from CVMFS. This will make modifying the CMSSW environment/software inside the image much more difficult as this is not a Git-based solution. In order to modify CMSSW the entire codebase would have to be mirrored inside the container, which a problem outside the scope of this use case.

The standalone container will have the following specifications:

  1. Based on standalone, which uses clelange/cc7-cms:latest as the base image.
  2. The home area is at /home/cmsusr/, which has CMSSW_10_2_21 and the associated TreeMaker software setup
  3. There is only one variant of this image (tag: Run2_2017-servicex), which has the ServiceX dependencies installed.

Usage

Here you will find a minimal set of instructions on how to run a container. This is not meant to be an exhaustive list of directions for every computer setup and use-case.

  1. Pull the most recent image:
docker pull treemaker/treemaker:<tag>
  1. For the CVMFS images, run a docker container using a similar command to:
docker run --rm -it -P --device /dev/fuse --cap-add SYS_ADMIN -e CVMFS_MOUNTS="cms.cern.ch oasis.opensciencegrid.org" -e MY_UID=`id -u` -e MY_GID=`id -g` -v ~/.globus/:/home/cmsuser/.globus/ --name treemaker treemaker/treemaker:<tag>

You can omit the MY_UID and MY_GID parts if you don't care about the UID and GID of the user in the container. You can omit the .globus mount if the grid proxy will be provided by some other method. If you will be providing the grid proxy without using voms-proxy-init, then it is likely you can omit the oasis.opensciencegrid.org CVMFS mount.

If you are using the standalone image, then the Docker command is simpler:

docker run --rm -it --name treemaker treemaker/treemaker:<tag>
  1. Setup the CMSSW environment:
cd CMSSW_10_2_21/src/
cmsenv

Depending upon your setup, you may need to get a grid proxy. Assuming you have setup your ~/.globus/ folder with the necessary files and permissions you can use the following command:

voms-proxy-init
  1. Run the integration test:
cd TreeMaker/Production/test
python unitTest.py test=16 run=True

A CMS Open Data version of the integration test can be run using the commands:

wget https://cernbox.cern.ch/index.php/s/5DRVyX4Z9EertGa/download -O eos.opendata.cms.MonteCarlo2016.RunIISummer16MiniAODv2.QCD_Pt_600to800_TuneCUETP8M1_13TeV_pythia8.MINIAODSIM.PUMoriond17_80X_mcRun2_asymptotic_2016_TrancheIV_v6-v1.70000.0048131D-3CB3-E611-813A-001E67DFFB31_100evt.root
python unitTest.py test=0 run=True scenario=Summer16 dataset=file:eos.opendata.cms.MonteCarlo2016.RunIISummer16MiniAODv2.QCD_Pt_600to800_TuneCUETP8M1_13TeV_pythia8.MINIAODSIM.PUMoriond17_80X_mcRun2_asymptotic_2016_TrancheIV_v6-v1.70000.0048131D-3CB3-E611-813A-001E67DFFB31_100evt.root name=Summer16.QCD_Pt_600to800_TuneCUETP8M1_13TeV_pythia8_opendata

The wget command can be omitted when using the ServiceX images because this file is already located at ${CMSSW_Base}/src/TreeMaker/Production/test/.

Add a custom footer
Clone this wiki locally