ROOT.TMVA in columnflow

[JulesVandenbroeck]

Hi everyone,

I was trying to create a calibrator that uses ROOT.TMVA to calculate lepton mva scores and store them (to use at selection stage). This was done before with xgboost but the weights for 2022 are .xml files that were produced using ROOT.TMVA. I found no way to create a virtual environment (using sandboxes) in which ROOT is installed so instead I opted for a virtual environment that uses CMSSW_12_6_2 (default). In here however I get the issue gfal2 is not installed which is required for columnflow/modules/law/law/contrib/gfal/target.py. Is there a possibility to add additional packages to the virtual environment that are orginaly not in cmssw? In case not, is there another way to include ROOT into a virtual environment sandbox?

Kind Regards,

Jules Vandenbroeck

[pkausw]

Hi @JulesVandenbroeck ,

converting this into a discussion since it's more about how to integrate a ROOT package into columnflow rather than a real problem. There was recently a discussion about how to use ROOT in the context of pybind11 here, where the user was able to successfully integrate a local ROOT installation into the setup. Can you give this a try?

In principle, switching to a CMSSW bash sandbox could also work, and it's a little strange that it says that gfal2 is not installed. Can you verify that you source the /cvmfs/cms.cern.ch/cmsset_default.sh within your cmssw sandbox and that the $CF_CONDA_BASE path is still part of your PATH within the sandbox?

[JulesVandenbroeck]

For the CMSSW bash sandbox part of your response, I have tried using columnflow/sandboxes/cmssw_default.sh and columnflow/sandboxes/cmssw_columnar.sh which both source columnflow/sandboxes/_setup_cmssw.sh (which in order source /cvmfs/cms.cern.ch/cmsset_default.sh). Within the setup (before it crashes) I checked $CF_CONDA_BASE and it is /user/juvanden/columnflowProjects/ttz/data/software/conda as expected. The only "abnormality" I found was when trying with cmssw_columnar.sh is the message /ada_mnt/ada/user/juvanden/columnflowProjects/ttz/data/software/cmssw/cmssw_columnar_9b3fd8a8/CMSSW_12_6_2/venvs/columnar/bin/python3: error while loading shared libraries: libpython3.9.so.1.0: cannot open shared object file: No such file or directory (the path does exist). I will now try to integrate a local ROOT like you suggested.

[JulesVandenbroeck]

Small update:

• when initializing conda with micromamba (setup.sh) it does include gfal2 for python3.9. This means the issue lies in CMSSW overwriting something? (in CMSSW gfal2 does exist for python2)
• For the approach to have a local copy of ROOT (available on cvmfs) I run into the issue Failed to import libcppyy3_9. Please check that ROOT has been built for Python 3.9.
• I also tried to add root to the packages installed with conda (in columnflow/setup.sh) and changed the python version to match root-config --python-version (3.9.18). This made it possible to import both ROOT and gfal2 when manually activating micromamba. This however (I don't know why) did not work in the virtual environment as no packages from micromamba were importable.

[pkausw]

Hi @JulesVandenbroeck ,

sorry for the problems you're experiencing. From the top of my head:

• you coud try to switch the CMSSW version that is used for the sandbox - I'm not sure though if this will help with the python compatibility though.
• The problem with unreachable micromamba packages from within a venv is also strange - that has happened in the past, but was fixed by including a pyenv file in the venv. Can you check that this file is present and that it points to the CF_CONDA_BASE? Can you try to update to the latest version of columnflow and re-initialize the software stack by running CF_REINSTALL_SOFTWARE=1 source setup.sh YOUR_SETUP_NAME?

To further look into this, it would also be helpful if you could provide a minimal working example to reproduce the issue, i.e.

• The link to your repo in working conditions (i.e. if you have local changes in the cmsdb, it would be great if you could provide those too)
• the content of your micromamba environment with pinned package versions (preferably as a yaml file)

[JulesVandenbroeck]

To comment on the first two points:

• I tried multiple CMSSW versions with similar results. I also tried a CMSSW version which did not have gfal2 for python2 (CMSSW_11_3_4) but the python version here is outdated and had conflicts with other packages. Since I have requested to move the files I need to our cluster gfal2 is no longer required for me (only needed to access remote files). All CMSSW versions still miss some other basic packages (coffea for example) so columnfllow/sandboxes/cmssw_default.sh can not be used.
• I tried to use columnfllow/sandboxes/cmssw_columnar.sh as this has ROOT as well as all columnflow requirements (luigi, coffea, which are not in CMSSW). This however runs into the issue I mentioned earlier /ada_mnt/ada/user/juvanden/columnflowProjects/ttz/data/software/cmssw/cmssw_columnar_9b3fd8a8/CMSSW_12_6_2/venvs/columnar/bin/python3: error while loading shared libraries: libpython3.9.so.1.0: cannot open shared object file: No such file or directory after which it can not import luigi.

My main question is thus how to make columnflow/sandboxes/cmssw_columnar.sh work and if it should normally work out of the box. I believe this is a general issue of columnflow/sandboxes/cmssw_columnar.sh not working as intended (not related to an analysis) but I will try to confirm this by running it on a template analysis on lxplus rather than the belgian cluster.

EDIT:

The conclusion is that in columnflow/sandboxes/cmssw_columnar.sh including pip install -r "${CF_BASE}/sandboxes/cf.txt && fixes the issue of having an environment in which ROOT is importable (this does not work when going over files accessed using root:// I believe). Why all the micromaba packages are not there even though pyvenv.cfg clearly points to ...data/software/conda/bin is still a mystery to me that would need to be solved if someone wants to use this without transferring files. In addition the libpython3.9.so.1.0 error/warning did not go away.