Bug Report: Problems generating full list of scan dates for plotTimeSeries.py

[ElizabethRoseStarling]

Types of issue

• Bug report (report an issue with the code)
• Feature request (request for change which adds functionality)

Expected Behavior

When I type the following command:

python plotTimeSeries.py --listOfScanDatesOnly --startDate=2017.01.01

for each detector defined in chamber_config.values() the listOfScanDAtes.txt file should be found at $DATA_PATH//scurve/

Current Behavior

Instead, it halts the process and outputs the following error:

Traceback (most recent call last):
File "plotTimeSeries.py", line 26, in
from gempython.utils.wrappers import envCheck
ImportError: No module named gempython.utils.wrappers

Steps to Reproduce (for bugs)

1. python plotTimeSeries.py --listOfScanDatesOnly --startDate=2017.01.01

Possible Solution (for bugs)

Context (for feature requests)

I am trying to generate the full list of scan dates since 2017 installation for the various detectors to study the presence of burned VFAT inputs in time-series SCurve plots.

Your Environment

• Version used:
• Shell used: Mac terminal

[mexanick]

@QuantumKittydynamics,
please report:

1. PC where you tried to run it
2. Account from which you've tried to run
3. How the package was installed (system-wide install of RPM or local github clone?)
4. Version of the package if installed as RPM or github branch/commit

[bdorney]

Additionally please include the commands, from the very beginning, and their outputs.

[bdorney]

Possibly related to:

#94
#95

[ElizabethRoseStarling]

1. My MacBook Pro
2. estarlin on LXPlus
3. Local github clone
4. commit dd3c00f (I think)

[mexanick]

If you connect to a certain PC over ssh in your macbook terminal and then run the code there - it is that PC of our interest. So as far as I can understand, a correct answer to the first question should be ''lxplus"
Now if you are using the local github clone of latest, please cross check the commit by running git show
Then if you have indicated the commit correctly, you must build a package and install it under the virtual environment (this is valid for lxplus, on other PCs, like gem904* or P5 ones there could be other options)

[bdorney]

Then if you have indicated the commit correctly, you must build a package and install it under the virtual environment (this is valid for lxplus, on other PCs, like gem904* or P5 ones there could be other options)

Indeed I think this is why it's important to have the exact commands executed, and their output. The info above does not show this.

Also I suspect the issue is stemming from the fact that after the rebase & merge of the packaging PR's the installation and setup statements of the README.md still reflect the pre-packaging PR instructions:

https://github.com/cms-gem-daq-project/gem-plotting-tools/#setup

@QuantumKittydynamics please provide exact commands, and their output.

@BenjaminRS, @jsturdy and @mexanick is there an updated installation recipe's documented somewhere?

[ElizabethRoseStarling]

I see, I didn't understand you wanted the commands from genesis.

Input:
ssh estarlin@lxplus.cern.ch

Output:
Warning: Permanently added the RSA host key for IP address '188.185.78.215' to the list of known hosts. Password:

Input:
hunter2 (No, not really)

Output:
Warning: No xauth data; using fake authentication data for X11 forwarding.

---

• Welcome to lxplus086.cern.ch, SLC, 6.9
• Archive of news is available in /etc/motd-archive
• Reminder: you have agreed to the CERN
• computing rules, in particular OC5. CERN implements
• the measures necessary to ensure compliance.
• https://cern.ch/ComputingRules
• Puppet environment: qa, Roger state: production
• Foreman hostgroup: lxplus/nodes/login
• Availability zone: cern-geneva-a
• LXPLUS Public Login Service - https://cern.ch/lxplusdoc

---

-bash: warning: setlocale: LC_CTYPE: cannot change locale (UTF-8): No such file or directory

Input: (***)
cd $BUILD_HOME/gem-plotting-tools/macros

(Because this is the only place where I can even run the script, rather than getting the following error: python: can't open file 'plotTimeSeries.py': [Errno 2] No such file or directory)

Output:
It does the thing.

Input:
python plotTimeSeries.py --listOfScanDatesOnly --startDate=2017.01.01

Output:

Traceback (most recent call last):
File "plotTimeSeries.py", line 27, in
from gempython.gemplotting.utils.anautilities import getDirByAnaType, makeListOfScanDatesFile
ImportError: No module named gemplotting.utils.anautilities

*** I rebuilt the three shell variables and resourced the paths as described here: https://github.com/cms-gem-daq-project/gem-plotting-tools/#setup Hence the new error.

And, indeed, with cmsgemos on commit eda69b0c2f4f261958dc4d204943a649fc45b665 , I do not see anything called anautilities in $BUILD_HOME/cmsgemos/gempython/gemplotting/utils/anautilities

[BenjaminRS]

ok I will leave it to @jsturdy to give the instructions as I will likely get it wrong. 2.5 weeks ago I had to install some updates with

cp ~sturdy/public/cmsgemos_gempython-0.3.1.tar.gz .
pip install --no-deps cmsgemos_gempython-0.3.1.tar.gz

But as I said, it is best to let Jared give the correct and up-to-date instructions,

From my area on lxplus I get the following:

$ python plotTimeSeries.py --listOfScanDatesOnly --startDate=2017.01.01
Options: vt1bump=vt1bump0, dataPath=/afs/cern.ch/user/b/benjamin/GEMDev, anaType=scurve
Traceback (most recent call last):
  File "plotTimeSeries.py", line 63, in <module>
    makeListOfScanDatesFile(chamber, anaType, options.startDate, options.endDate)
  File "/afs/cern.ch/user/b/benjamin/GEMDev/venv/slc6/py27/lib/python2.7/site-packages/gempython/gemplotting/utils/anautilities.py", line 352, in makeListOfScanDatesFile
    dirPath = getDirByAnaType(anaType, chamberName, ztrim)
  File "/afs/cern.ch/user/b/benjamin/GEMDev/venv/slc6/py27/lib/python2.7/site-packages/gempython/gemplotting/utils/anautilities.py", line 57, in getDirByAnaType
    from gempython.gemplotting.anaInfo import ana_config
ImportError: No module named anaInfo

[jsturdy]

I will try to update with a minimal get-it-working set of instructions for a user (@QuantumKittydynamics, your current use case is as a user, rather than developing a feature, correct?)

[ElizabethRoseStarling]

@jsturdy , yes, I'm not trying to develop anything right now, just a user. :) Thank you!

[jsturdy]

Hi @QuantumKittydynamics, as a first go, can you try this
Missing is the cmsgemos tarball, which you can currently find on the 904 DAQ PCs in the location specified by @BenjaminRS above.

I think this will be enough get you going, but I may have missed a step or two, in which case, report here the errors.

Once this document is updated to a level that works, I'll work it into the READMEs for the different repositories

[ElizabethRoseStarling]

Hi @jsturdy , sorry for the late reply, I got held up by another task. So, I was able to successfully get all three tarballs, but I fail when trying to set up the virtual environment.

I checked my Python version using:

Input: python -V

Output: Python 2.7.11

So I inputted that into the instructions here: https://gist.github.com/jsturdy/a82a6b7b7d497ae8ef98ee736119c6b5

Input: [estarlin@lxplus027 CMS_GEM_DAQ]$ virtualenv -p 2.7.11 --system-site-packages /afs/cern.ch/user/e/estarlin/CMS_GEM/CMS_GEM_DAQ/venv /afs/cern.ch/user/e/estarlin/CMS_GEM/CMS_GEM_DAQ/venv/bin/activate

Output: The executable 2.7.11 (from --python=2.7.11) does not exist

I'm not sure I understand what's happening? Thanks for your help.

[lmoureaux]

Hi @QuantumKittydynamics,

virtualenv expects you to provide a callable python command. If you have Python 2.7 in your PATH, you can use: virtualenv -p python ... or virtualenv -p python2.7 ...

I on my side haven't been able to find the third tarball. Is it in a "public" location somewhere?

[ElizabethRoseStarling]

Benjamin posted the command for the third tarball, @lmoureaux : cp ~sturdy/public/cmsgemos_gempython-0.3.1.tar.gz .

[ElizabethRoseStarling]

Thanks, Louis, that seemed to work.

Input:

virtualenv -p python --system-site-packages /afs/cern.ch/user/e/estarlin/CMS_GEM/CMS_GEM_DAQ/venv

Output:

Running virtualenv with interpreter /cvmfs/cms.cern.ch/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_0_10/external/slc6_amd64_gcc530/bin/python
New python executable in /afs/cern.ch/user/e/estarlin/CMS_GEM/CMS_GEM_DAQ/venv/bin/python
Installing setuptools, pip...done.

This makes my venv with folders: bin include lib lib64 slc6

But the second line of the instructions doesn't make sense: /path/to/your/venv/bin/activate

I can't add it to the initial command, because I get the error "There must be only one argument: DEST_DIR"

And I can't do anything with it on its own.

Just entering it alone does nothing, as expected:

Input:

[estarlin@lxplus027 CMS_GEM_DAQ]$ /afs/cern.ch/user/e/estarlin/CMS_GEM/CMS_GEM_DAQ/venv/bin/activate

Output:

-bash: /afs/cern.ch/user/e/estarlin/CMS_GEM/CMS_GEM_DAQ/venv/bin/activate: Permission denied

I can't cd to it, because it doesn't exist.

And replacing the DEST_DIR with /path/to/your/venv/bin/activate results in a giant mess:

Running virtualenv with interpreter /cvmfs/cms.cern.ch/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_0_10/external/slc6_amd64_gcc530/bin/python
Traceback (most recent call last):
File "/usr/lib/python2.6/site-packages/virtualenv.py", line 2352, in
main()
File "/usr/lib/python2.6/site-packages/virtualenv.py", line 825, in main
symlink=options.symlink)
File "/usr/lib/python2.6/site-packages/virtualenv.py", line 985, in create_environment
site_packages=site_packages, clear=clear, symlink=symlink))
File "/usr/lib/python2.6/site-packages/virtualenv.py", line 1159, in install_python
mkdir(lib_dir)
File "/usr/lib/python2.6/site-packages/virtualenv.py", line 448, in mkdir
os.makedirs(path)
File "/cvmfs/cms.cern.ch/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_0_10/external/slc6_amd64_gcc530/bin/../../../../../../external/python/2.7.11-ikhhed/lib/python2.7/os.py", line 150, in makedirs
makedirs(head, mode)
File "/cvmfs/cms.cern.ch/slc6_amd64_gcc530/cms/cmssw/CMSSW_8_0_10/external/slc6_amd64_gcc530/bin/../../../../../../external/python/2.7.11-ikhhed/lib/python2.7/os.py", line 157, in makedirs
mkdir(name, mode)
OSError: [Errno 20] Not a directory: '/afs/cern.ch/user/e/estarlin/CMS_GEM/CMS_GEM_DAQ/venv/bin/activate/lib'

[lmoureaux]

Just entering it alone does nothing, as expected:
Input:

[estarlin@lxplus027 CMS_GEM_DAQ]$ /afs/cern.ch/user/e/estarlin/CMS_GEM/CMS_GEM_DAQ/venv/bin/activate

Output:

-bash: /afs/cern.ch/user/e/estarlin/CMS_GEM/CMS_GEM_DAQ/venv/bin/activate: Permission denied

I can't cd to it, because it doesn't exist.

Source it:

source /path/to/your/venv/bin/activate

[ElizabethRoseStarling]

I can't source it...because it doesn't exist. Am I supposed to make it first?

Input:

[estarlin@lxplus027 CMS_GEM_DAQ]$ source /afs/cern.ch/user/e/estarlin/CMS_GEM/CMS_GEM_DAQ/venv/bin/activate/

Output:

-bash: /afs/cern.ch/user/e/estarlin/CMS_GEM/CMS_GEM_DAQ/venv/bin/activate/: Not a directory

[lmoureaux]

The error message doesn't say "No such file or directory" but "Not a directory". It believes that you're trying to access a directory. That's because of the trailing slash after activate. Remove it and it should work.

[ElizabethRoseStarling]

Thank you so much for all the help, Louis. I am not super command line literate yet..

So the only one I can't successfully install now is the cmsgemos tarball. Using the option --nodeps gives "no such option: --nodeps". And not adding that option gives:

Processing ./cmsgemos_gempython-0.3.1.tar.gz
Collecting cactusboards-amc13-python>=1.2.5 (from cmsgemos-gempython==0.3.1)
Could not find a version that satisfies the requirement cactusboards-amc13-python>=1.2.5 (from cmsgemos-gempython==0.3.1) (from versions: )
No matching distribution found for cactusboards-amc13-python>=1.2.5 (from cmsgemos-gempython==0.3.1)

[lmoureaux]

The correct option is --no-deps.

[ElizabethRoseStarling]

Great, that worked. I went through everything else, it all worked...and then finally back to the original problem...

Input:

(venv)[estarlin@lxplus027 macros]$ python plotTimeSeries.py --listOfScanDatesOnly --startDate=2017.01.01

Output:

You must source the environment properly, DATA_PATH is not set
Traceback (most recent call last):
File "plotTimeSeries.py", line 54, in
envCheck('DATA_PATH')
File "/afs/cern.ch/user/e/estarlin/CMS_GEM/CMS_GEM_DAQ/venv/lib/python2.7/site-packages/gempython/utils/wrappers.py", line 38, in envCheck
raise EnvironmentError('%s is not set'%envVar)
EnvironmentError: DATA_PATH is not set

[bdorney]

EnvironmentError: DATA_PATH is not set

This means that shell variable $DATA_PATH is not set. However just to cross-check there's no misunderstanding what are you trying to do right now with this command? Have you already transferred the data from P5? If so this would have already generated a listOfScanDates for you.

[jsturdy]

@lmoureaux did this fail at the virtualenv creation? Or when updating setuptools/pip/importlib?

I just today was trying to work around the slc6 python2.6 issues installing the updates
I just tried on lxplus with the modifications I made to the gist, i.e., pip install -I importlib 'setuptools<39.2.0' 'pip<10.1.0' and the virtualenv created fine (I didn't test beyond this)

Also, on lxplus can you try a python version provided in one of the LCG package bundles? This will provide an environment with python2.7 and some version of ROOT

• /cvmfs/sft.cern.ch/lcg/views/
  • e.g., source /cvmfs/sft.cern.ch/lcg/views/LCG_93/x86_64-slc6-gcc7-opt/setup.sh

On the same page? Else can you please provide a link to it?

A separate gist, but it right now is mostly just a clone of the user setup, and will mostly only diverge once the user setup is stable.

[lmoureaux]

@jsturdy Thank you 👍. I'll take a look tomorrow. I hadn't refreshed your gist...

[lmoureaux]

2.6

To update pip, importlib and setuptools in my venv at P5, I downloaded the packages on lxplus:

pip download importlib 'setuptools<39.2.0'  'pip<10.1.0'

I uploaded the tarballs using scp. When I try to install them at P5, it breaks my pip:

[lmoureau@srv-s2g18-33-01 ~]$ rm -rf venv26                                     
[lmoureau@srv-s2g18-33-01 ~]$ virtualenv -p python --system-site-packages venv26
Already using interpreter /usr/bin/python
New python executable in venv26/bin/python
Installing Setuptools...................................................................................................................................................................................................................................done.
Installing Pip..........................................................................................................................................................................................................................................................................................................................................done.
[lmoureau@srv-s2g18-33-01 ~]$ source venv26/bin/activate
(venv26)[lmoureau@srv-s2g18-33-01 ~]$ pip install -U importlib-1.0.4.zip pip-9.0.3-py2.py3-none-any.whl setuptools-36.8.0-py2.py3-none-any.whl 
Unpacking ./importlib-1.0.4.zip
  Running setup.py egg_info for package from file:///cmsnfshome0/nfshome0/lmoureau/importlib-1.0.4.zip
Unpacking ./pip-9.0.3-py2.py3-none-any.whl
Unpacking ./setuptools-36.8.0-py2.py3-none-any.whl
Installing collected packages: importlib, pip, setuptools
  Running setup.py install for importlib
  Found existing installation: pip 1.4.1
    Uninstalling pip:
      Successfully uninstalled pip
  Found existing installation: distribute 0.6.10
    Not uninstalling distribute at /usr/lib/python2.6/site-packages, outside environment /cmsnfshome0/nfshome0/lmoureau/venv26
Successfully installed importlib pip setuptools
Cleaning up...
(venv26)[lmoureau@srv-s2g18-33-01 ~]$ pip
-bash: /cmsnfshome0/nfshome0/lmoureau/venv26/bin/pip: No such file or directory

Note: The system pip is also broken:

$ /usr/bin/pip --version
Traceback (most recent call last):
  File "/usr/bin/pip", line 9, in <module>
    load_entry_point('pip==1.3.1', 'console_scripts', 'pip')()
[snipped]
  File "/usr/lib64/python2.6/io.py", line 631, in __init__
    _fileio._FileIO.__init__(self, name, mode, closefd)
IOError: [Errno 13] Permission denied: '/usr/lib/python2.6/site-packages/cactusboards_amc13_python-1.2.5-py2.6.egg-info'

2.7

Also, on lxplus can you try a python version provided in one of the LCG package bundles? This will provide an environment with python2.7 and some version of ROOT

$ source /cvmfs/sft.cern.ch/lcg/views/LCG_93/x86_64-slc6-gcc7-opt/setup.sh
$ python -V
Python 2.7.13
$ virtualenv -p python --system-site-packages venv27
'import site' failed; use -v for traceback
Traceback (most recent call last):
  File "/usr/bin/virtualenv", line 2, in <module>
    import virtualenv
ImportError: No module named virtualenv

I get the same results even after pip install --user virtualenv.

[jsturdy]

OK, the system pip issue is not pip per se, just some bad permissions on a package I built and installed (I had fixed it on gem-dqm01, but forgot to fix it on gem-daq01, so that is now fixed)

OK, on the LCG package, this is because virtualenv must not be included in the set..., might be worth asking on the computing tools hypernews.
Actually, i just tried the following:

source /cvmfs/sft.cern.ch/lcg/views/LCG_93/x86_64-slc6-gcc7-opt/setup.sh
python -m virtualenv -p python /tmp/${USER}/lcgvenv --system-site-packages
python -m pip list
python -m pip install -U importlib setuptools pip

This worked, probably due to the way that the LCG bundle is packaged, and in general, it should always work, even if the pip and virtualenv binary method works.

[bdorney]

For setting up a virtualenv on lxplus i think the example illustrated here:

https://github.com/cms-gem-daq-project/gem-plotting-tools/blob/develop/setup/createVirtualEnvPy27.sh

Could be adapted, note there is one for py26 as well.

Again I would suggest against trying to reinvent the wheel and try to adapt existing tools.

[jsturdy]

That will only work if the rh-python27 package is installed on the machine (I have no idea what the CERN policy is on that), the LCG views are a "standard" option available to any lxplus user

One can also get a cc7 machine by using lxplus7 and get python2.7 as the default, which can also be more stable than relying on an LCG view, or a possibly installed extra.

In addition, it's not clear that that method would work with the updates to the three packages (I'm pretty sure the same issues already encountered would be present), as a "standard" virtualenv setup is not a problem, it's a problem if there are some user installed local python packages (pip list --user) or some compatibility issues between the versions of these updates, which I previously had not placed upper limits on, but probably should, simply because python2.6 is likely not tested, and python2.7 is EOL in 2020.

[bdorney]

That will only work if the rh-python27 package is installed on the machine (I have no idea what the CERN policy is on that), the LCG views are a "standard" option available to any lxplus user

Indeed, but it seems lxplus nodes have rh-python27 installed. However the point is taken. The script could be updated to use the LCG if that was desired.

One can also get a cc7 machine by using lxplus7 and get python2.7 as the default, which can also be more stable than relying on an LCG view, or a possibly installed extra.

Does the subversion of python2.7 matter much here? I assume not, on cc7 the default is 2.7.5 whereas rh-python27 is 2.7.11 iirc.

In addition, it's not clear that that method would work with the updates to the three packages (I'm pretty sure the same issues already encountered would be present), as a "standard" virtualenv setup is not a problem, it's a problem if there are some user installed local python packages (pip list --user) or some compatibility issues between the versions of these updates, which I previously had not placed upper limits on, but probably should, simply because python2.6 is likely not tested, and python2.7 is EOL in 2020.

I'll give it a try as well and report back. I'm trying the developer instructions so @lmoureaux can continue with the user instructions.

[lmoureaux]

Python 2.7

On my lxplus account:

$ source /cvmfs/sft.cern.ch/lcg/views/LCG_93/x86_64-slc6-gcc7-opt/setup.sh
$ python -m virtualenv -p python /tmp/${USER}/lcgvenv --system-site-packages
/cvmfs/sft.cern.ch/lcg/views/LCG_93/x86_64-slc6-gcc7-opt/bin/python: No module named virtualenv

However,

$ source /cvmfs/sft.cern.ch/lcg/views/LCG_93/x86_64-slc6-gcc7-opt/setup.sh
$ pip install --user virtualenv
$ python -m virtualenv -p python /tmp/${USER}/lcgvenv --system-site-packages
Already using interpreter /cvmfs/sft.cern.ch/lcg/views/LCG_93/x86_64-slc6-gcc7-opt/bin/python
PYTHONHOME is set.  You *must* activate the virtualenv before using it
New python executable in /tmp/lmoureau/lcgvenv/bin/python
Installing setuptools, pip, wheel...done.

...and the rest of the instructions work fine, though:exclamation:the gemplotting macros are not executable.

We may want to use $VIRTUAL_ENV in the instructions.

What's a good Python 2.7 distribution available at P5?

[bdorney]

...and the rest of the instructions work fine, though❗️the gemplotting macros are not executable.

What is the output of echo $PATH?

We may want to use $VIRTUAL_ENV in the instructions.

What do you mean?

[lmoureaux]

Python 2.6

Still on my lxplus account, using the default Python :

• python -m pip throws an exception, one has to use pip
• Else (on lxplus) everything is as for Python 2.7

[lmoureaux]

the gemplotting macros are not executable.

What is the output of echo $PATH?

The PATH is set correctly, but one has to chmod +x the macros. This is a problem in the package.

We may want to use $VIRTUAL_ENV in the instructions.

What do you mean?

Instead of:

export PATH=<path/to/venv/lib/../site-packages/gempython/scripts>:$PATH

write:

export "PATH=$VIRTUAL_ENV/lib/python*/site-packages/gempython/scripts:$PATH"

which can be copy-pasted as-is, provided there's only one Python version installed in the virtualenv. It can be changed in two other locations.

I also think that the .. part of the path should be replaced, as it has a meaning in this context.

[bdorney]

I also think that the .. part of the path should be replaced, as it has a meaning in this context.

I think @jsturdy was using it simply to imply additional sub folders exist, and includes a **do not blindly copy this!** warning. However if instead of using VIRTUAL_ENV I would suggest to use instead VENV_ROOT following the same schema we have used in the past for our other repo's (e.g. shell variables defined XHAL_ROOT and CTP7_ROOT.

Additionally in:

export "PATH=$VENV_ROOT/lib/python*/site-packages/gempython/scripts:$PATH"

The number of python versions installed should not matter since you should be setting up a separate virtualenv with a different VENV_ROOT for a different python directory, the lib/python* is already inside the venv.

[lmoureaux]

I also think that the .. part of the path should be replaced, as it has a meaning in this context.

I think @jsturdy was using it simply to imply additional sub folders exist

That's obviously the meaning of it, but I didn't understand it at first (there are two parts to replace). Just trying to avoid complaints about the instructions not working

instead of using VIRTUAL_ENV I would suggest to use instead VENV_ROOT

My point is that source foo/bar/bin/activate already sets VIRTUAL_ENV to the correct value for virtualenv currently in use.

provided there's only one Python version installed in the virtualenv

The number of python versions installed should not matter since you should be setting up a separate virtualenv with a different VENV_ROOT for a different python directory, the lib/python* is already inside the venv.

Even though I couldn't find one, maybe @jsturdy had a good reason not to use a wildcard here.

[bdorney]

My point is that source foo/bar/bin/activate already sets VIRTUAL_ENV to the correct value for virtualenv currently in use.

I see, I envisioned a different use case in which $VENV_ROOT is an argument provided by the user which is used in a script to create the venv in the correct location.

Even though I couldn't find one, maybe @jsturdy had a good reason not to use a wildcard here.

When using wildcards the following fails for me:

find <path/to/venv/lib/../site-packages/gempython> -type f -name buildMapFiles.py -exec python {} \;

In general I would suggest PATH to not include wildcards. You can do something like:

PYTHON_VERSION=$(python -c "import sys; sys.stdout.write(sys.version[:3])") 
export "PATH=$VENV_ROOT/lib/python$PYTHON_VERSION/site-packages/gempython/scripts:$PATH"

With some help from @jsturdy got the syntax working nicely.

[lmoureaux]

When using wildcards the following fails for me:

find <path/to/venv/lib/../site-packages/gempython> -type f -name buildMapFiles.py -exec python {} \;

How does it fail? I just checked and for me it finds the file (not sure about executing it).

In general I would suggest PATH to not include wildcards.

Wildcards are expanded when the variable is set, and AFAIK not when looking it up. So my command above $PATH will only contain the wildcard when it didn't match anything, and even in that case it will be harmless. We can construct it from the Python version, although a cleaner (in the long term) option would be to install the tools in venv/bin, so that they are in the PATH to begin with :-)

[bdorney]

source /cvmfs/sft.cern.ch/lcg/views/LCG_93/x86_64-slc6-gcc7-opt/setup.sh

@jsturdy this probably also covers devtoolset-6/7 I assume as well?

[bdorney]

I cannot repeat @lmoureaux's success story:

$ source /cvmfs/sft.cern.ch/lcg/views/LCG_93/x86_64-slc6-gcc7-opt/setup.sh
$ pip install --user virtualenv
$ python -m virtualenv -p python /tmp/${USER}/lcgvenv --system-site-packages
Already using interpreter /cvmfs/sft.cern.ch/lcg/views/LCG_93/x86_64-slc6-gcc7-opt/bin/python
PYTHONHOME is set. You must activate the virtualenv before using it
New python executable in /tmp/lmoureau/lcgvenv/bin/python
Installing setuptools, pip, wheel...done.

specifically:

...and the rest of the instructions work fine

When attempting to run pip install -U importlib setuptools pip:

Running virtualenv with interpreter /cvmfs/sft.cern.ch/lcg/views/LCG_93/x86_64-slc6-gcc7-opt/bin/python
PYTHONHOME is set.  You *must* activate the virtualenv before using it
New python executable in /afs/cern.ch/user/d/dorney/scratch0/CMS_GEM/CMS_GEM_DAQ/venv/slc6/v2b/py27/bin/python
Installing setuptools, pip, wheel...done.
Traceback (most recent call last):
  File "/afs/cern.ch/user/d/dorney/scratch0/CMS_GEM/CMS_GEM_DAQ/venv/slc6/v2b/py27/bin/pip", line 7, in <module>
    from pip._internal import main
ImportError: No module named _internal

With the LCG version of pip and python there seems to be some anomaly in pip after creating the user installed virtualenv. The internet suggests this is because there are multiple versions of pip in $PATH and the attempt to upgrade is ambiguous; a bit surprised by this actually. Seems like LCG is leaving some residual things in $PATH that are ambiguous (or PYTHONPATH is not set correctly...?).

Going to give up on the LCG view since this doesn't seem to be functional. Diverting back to the relying on the original strategy shown in createVirtualEnvPy27.sh

[lmoureaux]

When attempting to run pip install -U importlib setuptools pip:

At the time, the instructions stated:

python -m pip install -U importlib setuptools pip

[jsturdy]

@lmoureaux under which conditions was python -m pip working and when did pip alone work?
I had modified the instructions (with a note that if pip/virtualenv themselves fail, to try with python -m pip/virtualenv, based on your reported semi-success)

[jsturdy]

@bdorney

source /cvmfs/sft.cern.ch/lcg/views/LCG_93/x86_64-slc6-gcc7-opt/setup.sh

@jsturdy this probably also covers devtoolset-6/7 I assume as well?

I definitely don't know what's being asked here... if a person is developing a feature that depends on some special version of some tool, I fully expect them to be able to set this up themselves (or follow a README/CONTRIBUTING that tells them what to do). The current attempt at providing a set of instructions is specific to the python side of things, and makes no attempt at wider support (which could be added to the aforementioned README/CONTRIBUTING in the repository of interest)

[lmoureaux]

@lmoureaux under which conditions was python -m pip working and when did pip alone work?

The ones documented here. It's maybe due to me having installed virtualenv locally.

[ElizabethRoseStarling]

Fantastic, I will try everything when I get back home from my trip. Thank you so much to everyone.

[ElizabethRoseStarling]

Following the new instructions in 29fb355 , I get the exact same initial error I reported in this bug report when attempting to generate the list of scan dates...

[lmoureaux]

Following the new instructions in 29fb355, I get the exact same initial error I reported in this bug report when attempting to generate the list of scan dates...

Did you start from a clean directory? I can't reproduce.