[feat] Add bootstrap script to install dependencies of ReFrame

[vkarak]

This really facilitates the processes of getting ready to run when you clone ReFrame from Github. You need only to do

./bootstrap.sh

or

./bootstrap.sh +docs

This will pip install all the dependencies inside the ReFrame source directory under external/. ReFrame's main "binaries" are adapted to look into this directory first for their dependencies. This solution does not preclude any of the other installation methods.

Fixes #1365.

Still to do:

• Update README
• Update documentation
• Update deployment script
• Update CI script

[codecov-commenter]

Codecov Report

Merging #1371 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master    #1371   +/-   ##
=======================================
  Coverage   91.66%   91.66%           
=======================================
  Files          83       83           
  Lines       12675    12675           
=======================================
  Hits        11618    11618           
  Misses       1057     1057           

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1f97a38...4c0d1dd. Read the comment docs.

[vkarak]

To test this PR, start from an empty virtual environment, so that ReFrame cannot find any of its dependencies, in case you have already installed them.

[teojgo]

Is it possible to have an option to pass the python interpreter to use? For example on my system I have both python3.6, python3.8.

[vkarak]

@teojgo That should be possible, but I will need to separate where I install the external dependencies, e.g., external/py36 and external/py38. Do you have any other suggestion?

[teojgo]

@teojgo That should be possible, but I will need to separate where I install the external dependencies, e.g., external/py36 and external/py38. Do you have any other suggestion?

You can have the default python command as python3 and the user can specify another if needed. Then since you get pyver and use it for external dependencies it will get the correct one.

[vkarak]

@teojgo The pyver is used exclusively for retrieving the right pip that was installed by ensurepip. All ReFrame's dependencies are then placed unconditionally under external/. Notice that I'm using --target and not --prefix. Prefix differentiates over the python version through the site-packages. I could do that, but that would require me to update all the files that update the sys.path. Why do you need this distinction? With your solution, you would have to re-bootstrap ReFrame for every Python version, whereas with the current one, you don't need to do so. The only problem I see is that when one of the packages installed does not run with a different Python version than the one used to bootstrap ReFrame, which in this case we should take it into account when writing the requirements.txt.

[teojgo]

@teojgo The pyver is used exclusively for retrieving the right pip that was installed by ensurepip. All ReFrame's dependencies are then placed unconditionally under external/. Notice that I'm using --target and not --prefix. Prefix differentiates over the python version through the site-packages. I could do that, but that would require me to update all the files that update the sys.path. Why do you need this distinction? With your solution, you would have to re-bootstrap ReFrame for every Python version, whereas with the current one, you don't need to do so. The only problem I see is that when one of the packages installed does not run with a different Python version than the one used to bootstrap ReFrame, which in this case we should take it into account when writing the requirements.txt.

It's ok. The problem is when you have an alternative installation of python on your system that the standard python3. But I believe this is specific to my local installation where I have to make the differentiation. Otherwise it's fine.

[vkarak]

@teojgo From our offline discussion I understood what you actually want and I'm gonna do it.

[teojgo]

Thanks @vkarak it works for all my python installations now.

[victorusu]

lgtm