python-modules: use venv to install environment (O2-1089) #2034

ktf · 2020-01-30T00:18:20Z

Now that we can rely on python3, we can simply setup our modules in a
virtualenv environment, with the added advantage that we can pick up whatever
is already available from the system or any previously setup virtualenv,
allowing people to easily customize their Python-modules environment.

ktf · 2020-01-30T00:22:11Z

@alisw/o2-wp3 please let's test this carefully, since it is quite a change. Hopefully it will however solve most / all our python woes.

might solve #1807 #1826

ktf · 2020-01-30T00:23:22Z

@aphecetche @miranov25 we discussed this at some point, IIRC.

ktf · 2020-01-30T00:33:16Z

@ironMann @mkrzewic could you try as well?

aphecetche · 2020-01-30T09:29:41Z

@ktf will try to give it a try next week, but a question right now : if there's already a virtualenv defined, why not using it simply and installing the modules in that one instead of defining a custom PYTHON_MODULES_INSTALLROOT ?
That's of course assuming people that have already defined a virtualenv kind of reserve it for Alice stuff, but I would guess that's not an unreasonable assumption to make ? Or should even become a recommendation ?

ktf · 2020-01-30T09:36:43Z

@aphecetche I thought about it, the reason why I set up another virtual env with a copy of the packages of the previous one is that this way if people forget to do "bin/activate", they still get a self contained python-modules.

ktf · 2020-01-30T09:39:18Z

Also this follows the mantra "Do not break user installation, do not assume user has a special python environment (apart from python3)". If I were to use a prexisting virtualenv, it would install extra packages in it which might have counter indications.

miranov25 · 2020-01-30T17:26:31Z

Hello Giulio.

Thanks for modification. I had day of full meeting - I did not spot the pull request.

For the 2 options discussed above I'm also in favour of user defining virtuaalenv.

user defined environment
copy of the packages of the previous one

I would like to be able to switch between the virtual env to test new functionatly (install experimental packages). Will it be supported in your schema? I will check how it it done, but I assume in copy version it will be more tricky to achieve that ...
I will pass by your office tomorrow

Marian.

mkrzewic · 2020-01-30T21:10:21Z

seems to work in my centos7 system with my favourite flag --always-prefer-system, but on second thought maybe i should not have used it?

ktf · 2020-01-30T21:12:51Z

@mkrzewic I think that's fine, because python-modules is never picked up from the system. Did you aliens enter Python-modules/latest?

mkrzewic · 2020-01-30T21:29:48Z

@ktf now I did, then tried to run aliDoctor:
[Python-modules/latest] ~/alisw $> aliDoctor
ImportError: No module named site

ktf · 2020-01-30T21:59:54Z

What about e.g. "import tensorflow"?

mkrzewic · 2020-01-30T22:13:33Z

@ktf: same:
ModuleNotFoundError: No module named 'tensorflow'

ktf · 2020-01-30T23:41:55Z

@miranov25 in order to switch between virtualenvs you will have to rebuild.

Now that we can rely on python3, we can simply setup our modules in a virtualenv environment, with the added advantage that we can pick up whatever is already available from the system or any previously setup virtualenv, allowing people to easily customize their Python-modules environment.

ktf · 2020-01-31T12:24:26Z

@marcovanleeuwen can you try now?

ktf · 2020-02-02T20:36:02Z

@marcovanleeuwen can you try again? I think this might fix your certifi runtime issues.

mkrzewic · 2020-02-05T13:10:45Z

@ktf with latest alidist + venv merged:

DEBUG:O2:Python-modules:0: /home/mkrzewic/alisw/sw/SPECS/slc7_x86-64/Python-modules/1.0-5/Python-modules.sh: /home/mkrzewic/alisw/sw/INSTALLROOT/d780b9e7e2712302a8a0e3695e6d30805d3491c9/slc7_x86-64/Python-modules/1.0-5/share/python-modules/bin/pip3: /home/mkrzewic/alisw/sw/INSTALLROOT/d780b9e7e2712302a8a0e3695e6d30805d3491c9/s: bad interpreter: No such file or directory
DEBUG:O2:Python-modules:0:
ERROR:O2:Python-modules:0: Error while executing /home/mkrzewic/alisw/sw/SPECS/slc7_x86-64/Python-modules/1.0-5/build.sh on `centos7.localdomain'.
ERROR:O2:Python-modules:0: Log can be found in /home/mkrzewic/alisw/sw/BUILD/Python-modules-latest/log.
ERROR:O2:Python-modules:0: Please upload it to CERNBox/Dropbox if you intend to request support.
ERROR:O2:Python-modules:0: Build directory is /home/mkrzewic/alisw/sw/BUILD/Python-modules-latest/Python-modules.

maireiphc · 2020-02-05T16:04:53Z

Hello Giulio (all),

I come here with the AliGenerators perspective, i.e. adding some extra entropy.
I am wondering if it would be safer/worse to have direct dependencies to the Python recipe whenever possible or rather transit systematically via Python-modules.

I ran the command : aliBuild deps AliGenerators --outgraph Deps-AliGenerators.pdf
to get an overview of the various dependencies.
For AliGenerators, there are several packages that do depend directly on Python modules :

Yoda
lhapdf

In turn, this means that some other packages do depend indirectly to Python rather than Python-modules, that are :

pythia (= pythia8)
sacrifice (so Pythia8)
lhapdf-pdfsets
Rivet
Rivet-hi

Is there any preferred way to create the dependencies to Python, direct or indirect via Python-modules ?

(I am still facing Python issues, at least to have Rivet running under my Kubuntu 18.04 with python3.6 available on the system but python2.7 being default.)

ktf · 2020-02-05T18:40:54Z

I would depend directly on Python (or Python-system, on mac) as much as possible. Depending on Python-modules if you do not really need it should really be considered as a mistake.

miranov25 · 2020-02-21T08:44:56Z

Dear all.

We need urgently some fox for the virtual environment. Can we help with testing?
In case pull request is problematic to merge this pull request, could we detect virtualenv and do not try --user installation in python recipes

Marian

miranov25 · 2020-03-13T07:48:44Z

Hallo @ktf and @teo
Is there some progress or do you have some other solution?

aphecetche · 2020-04-17T15:42:35Z

I'd also like to see some progress on this one. The setup I was using (define a virtual env where I install alibuild and activate when -and only when- I'm working on Alice stuff) and that was working fine with the existing Python-modules-* recipes has recently hit a snag. Seems recent pip (>=20 ?) now refuses by default the --user option, in which case the Python-modules build fails with :

ERROR: Can not perform a '--user' install. User site-packages are not visible in this virtualenv.

For the time being I "solved" it using a quick and dirty hack of two recipes : python-modules.sh xjalienfs.sh (see diffs here), and pre-installing Cython 0.29.16 in my dev virtual env.

miranov25 · 2020-04-18T06:40:11Z

I'm also stuck here. I'm not able to rerun aliBuild anymore.
My working schema stopped to work.
We URGENTLY need solution

miranov25 · 2020-04-18T15:00:35Z

In meantime I merged @aphecetche "way around sollution" to my branch, but on the next failure I will declare it blocker production tests

ktf · 2020-07-27T13:27:37Z

I am back at this. If people could try it out and let me know their milage, I think we can merge this now.

aphecetche · 2020-08-19T11:22:55Z

@ktf just tried a aliBuild --defaults o2 build O2 on a fresh directory (except for this PR for alidist) under my usual setup (basically a custom python virtualenv, see below) (on my Mac) and it seems that all went fine for me.

[ pr ]~/alice/pr$ ls -al
total 8
drwxr-xr-x    7 laurent  staff   224 19 aoû 11:11 .
drwxr-xr-x   12 laurent  staff   384  9 déc  2019 ..
-rw-r--r--    1 laurent  staff    26 19 aoû 11:11 .tool-versions
drwxr-xr-x    3 laurent  staff    96 19 aoû 11:08 .venv
drwxr-xr-x   40 laurent  staff  1280 19 aoû 13:12 O2
drwxr-xr-x  229 laurent  staff  7328 19 aoû 11:10 alidist
drwxr-xr-x   10 laurent  staff   320 19 aoû 12:54 sw
[ pr ]~/alice/pr$ tree .venv -L 3
.venv
└── pr
    ├── bin
    │   ├── activate
    │   ├── activate.csh
    │   ├── activate.fish
    │   ├── aliBuild
    │   ├── aliDeps
    │   ├── aliDoctor
    │   ├── alienv
    │   ├── chardetect
    │   ├── distro
    │   ├── easy_install
    │   ├── easy_install-3.7
    │   ├── pb
    │   ├── pip
    │   ├── pip3
    │   ├── pip3.7
    │   ├── python -> /Users/laurent/.asdf/installs/python/3.7.6/Python.framework/Versions/3.7/bin/python
    │   └── python3 -> python
    ├── include
    ├── lib
    │   └── python3.7
    └── pyvenv.cfg

ktf · 2020-08-19T12:09:15Z

Ok, great. Anyone else has troubles? Otherwise I guess we can merge this and fix the bugs later on. 9 tests are green, after all...

aphecetche · 2020-08-19T12:26:07Z

Might not be related to this PR (it's been a while I did not test to run a full ctest), but I note that some tests are failing in my setup :

[ pr ]~/alice/pr/sw/BUILD/O2-latest/O2$ ctest -j 8
...snip...
The following tests did not run:
         80 - test_Framework_test_BoostSerializedProcessing (Disabled)
         83 - test_Framework_test_DanglingInputs (Disabled)
        100 - test_Framework_test_CustomGUIGL (Disabled)
        101 - test_Framework_test_CustomGUISokol (Disabled)
        102 - test_Framework_test_SimpleTracksED (Disabled)

The following tests FAILED:
        172 - Detectors/Base/test/testMatBudLUT.cxx (Failed)
        178 - Detectors/CTF/test/test_ctf_io_itsmft.cxx (Failed)
        179 - Detectors/CTF/test/test_ctf_io_tpc.cxx (Failed)
        180 - Detectors/CTF/test/test_ctf_io_ft0.cxx (Failed)
        181 - Detectors/CTF/test/test_ctf_io_fv0.cxx (Failed)
        363 - o2sim_G4 (Failed)
        364 - o2sim_G4_mt (Failed)
        365 - o2sim_checksimkinematics_G4 (Not Run)
        366 - o2sim_G3 (Not Run)
        367 - o2sim_checksimkinematics_G3 (Not Run)
        368 - o2sim_hepmc (Failed)
Errors while running CTest
[ pr ]~/alice/pr/sw/BUILD/O2-latest/O2$

Now that we can rely on python3, we can simply setup our modules in a virtualenv environment, with the added advantage that we can pick up whatever is already available from the system or any previously setup virtualenv, allowing people to easily customize their Python-modules environment.

ktf requested a review from a team as a code owner January 30, 2020 00:18

ghost requested a review from teo January 30, 2020 00:18

ktf changed the title ~~python-modules: use venv to install environment~~ python-modules: use venv to install environment (O2-1089) Jan 30, 2020

ktf mentioned this pull request Jan 30, 2020

Update Prerequisites for Different Platforms alice-doc/alice-analysis-tutorial#82

Merged

ktf force-pushed the venv branch from 8fa6d40 to 0b5d16c Compare January 30, 2020 22:00

This was referenced Jan 30, 2020

Python-modules site-package interferes with python virtualenv #1807

Open

Error compiling Python 3.6 on SL6 #1653

Open

ktf force-pushed the venv branch from 33047ba to 00e4252 Compare January 31, 2020 12:23

Make sure our python3 is used when looking for certifi

2dd4e24

Merge branch 'master' into venv

221783f

Merge branch 'master' into venv

9862f44

ktf merged commit 38a199d into master Aug 19, 2020

ktf deleted the venv branch August 19, 2020 14:47

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

python-modules: use venv to install environment (O2-1089) #2034

python-modules: use venv to install environment (O2-1089) #2034

ktf commented Jan 30, 2020

ktf commented Jan 30, 2020

ktf commented Jan 30, 2020

ktf commented Jan 30, 2020

aphecetche commented Jan 30, 2020

ktf commented Jan 30, 2020

ktf commented Jan 30, 2020

miranov25 commented Jan 30, 2020

mkrzewic commented Jan 30, 2020

ktf commented Jan 30, 2020

mkrzewic commented Jan 30, 2020

ktf commented Jan 30, 2020

mkrzewic commented Jan 30, 2020

ktf commented Jan 30, 2020

ktf commented Jan 31, 2020

ktf commented Feb 2, 2020

mkrzewic commented Feb 5, 2020

maireiphc commented Feb 5, 2020

ktf commented Feb 5, 2020

miranov25 commented Feb 21, 2020 •

edited

Loading

miranov25 commented Mar 13, 2020

aphecetche commented Apr 17, 2020

miranov25 commented Apr 18, 2020

miranov25 commented Apr 18, 2020

ktf commented Jul 27, 2020

aphecetche commented Aug 19, 2020

ktf commented Aug 19, 2020

aphecetche commented Aug 19, 2020

python-modules: use venv to install environment (O2-1089) #2034

python-modules: use venv to install environment (O2-1089) #2034

Conversation

ktf commented Jan 30, 2020

ktf commented Jan 30, 2020

ktf commented Jan 30, 2020

ktf commented Jan 30, 2020

aphecetche commented Jan 30, 2020

ktf commented Jan 30, 2020

ktf commented Jan 30, 2020

miranov25 commented Jan 30, 2020

mkrzewic commented Jan 30, 2020

ktf commented Jan 30, 2020

mkrzewic commented Jan 30, 2020

ktf commented Jan 30, 2020

mkrzewic commented Jan 30, 2020

ktf commented Jan 30, 2020

ktf commented Jan 31, 2020

ktf commented Feb 2, 2020

mkrzewic commented Feb 5, 2020

maireiphc commented Feb 5, 2020

ktf commented Feb 5, 2020

miranov25 commented Feb 21, 2020 • edited Loading

miranov25 commented Mar 13, 2020

aphecetche commented Apr 17, 2020

miranov25 commented Apr 18, 2020

miranov25 commented Apr 18, 2020

ktf commented Jul 27, 2020

aphecetche commented Aug 19, 2020

ktf commented Aug 19, 2020

aphecetche commented Aug 19, 2020

miranov25 commented Feb 21, 2020 •

edited

Loading