update software installation script for EESSI 2021.02 pilot

[boegel]

Overview of changes:

• updates for EasyBuild v4.3.3 (fixes update build script for EasyBuild v4.3.3 #70)
• also use {linux,macos} subdirectory in 2021.02/software/ (fixes also use {linux,macos} subdirectory in software layer #71)
• change check for Prefix environment (relying on start of $PATH value is problematic, sometimes additional paths are included at front)
• fix path to Lmod init script (changes due to Remove lmod from our overlay gentoo-overlay#9?)
• filter additional dependencies (to use what's already provided in compat layer)
  • not filtering Bison, due to problems with Qt5 using yacc from compat layer (see Qt5 compilation fails with Bison from compat layer #49)
• stop using EasyBuild bootstrap script, use pip install + eb --install-latest-eb-release

(WIP, still testing it...)

[ocaisa]

We need to add an explicit build of UCX with the the option

configopts = '--enable-optimizations --enable-cma --enable-mt --with-verbs --with-rdmacm=$EPREFIX/usr --with-sysroot=$EPREFIX'

[boegel]

@ocaisa Is that to fix a particular problem? Can we fix it in a generic way on the EasyBuild side?

[ocaisa]

See #63 , I'm not sure it is a fit on the EB side since it is relying on you having certain system packages installed. For the record, the JSC configuration for UCX has way more options:

configopts = '--with-verbs '  # Build OpenFabrics support
configopts += '--without-java '
configopts += '--disable-doxygen-doc '

configopts += '--enable-optimizations '  # Enable machine-specific optimizations, default: NO
# configopts += '--enable-tuning '       # Enable parameter tuning in run-time, default: NO
configopts += '--enable-mt '           # Enable thread support in UCP and UCT, default: NO
configopts += '--disable-debug '
configopts += '--disable-logging '
configopts += '--disable-assertions '
configopts += '--disable-params-check '
configopts += '--disable-dependency-tracking '
configopts += '--with-cuda=$EBROOTCUDA '
configopts += '--enable-cma '           # Enable Cross Memory Attach
configopts += '--with-rc '              # Compile with IB Reliable Connection support
configopts += '--with-ud '              # Compile with IB Unreliable Datagram support
configopts += '--with-dc '              # Compile with IB Dynamic Connection support
configopts += '--with-mlx5-dv '         # Compile with mlx5 Direct Verbs support
configopts += '--with-ib-hw-tm '        # Compile with IB Tag Matching support
configopts += '--with-dm '              # Compile with Device Memory support
configopts += '--without-cm '           # Disable IB CM
configopts += '--with-avx '             # Compile with AVX
configopts += '--with-gdrcopy '         # Compile with GDRCopy

[ocaisa]

And the ComputeCanada customisation is at https://github.com/ComputeCanada/easybuild-computecanada-config/blob/b40236c4509d02ed7066ee91f6d3f298d500eba0/cc_hooks_gentoo.py#L533

(also their customisation for OpenMPI is pretty big, see https://github.com/ComputeCanada/easybuild-computecanada-config/blob/b40236c4509d02ed7066ee91f6d3f298d500eba0/cc_hooks_gentoo.py#L490)

[boegel]

@bedroge @ocaisa This should be good to go in now imho, so others can just run ./EESSI-pilot-install-software.sh from a clone of this repo to install the 2021.02 pilot software layer...

[bedroge]

I was just going through the project dashboard to check if everything that we wanted to do for this version has been included, and saw this issue about ReFrame under software layer:
#67
I don't see this in the installation script though, should we still add this?

[ocaisa]

I wanted to make a PR to add RStudio Server and VS Code to the list as well, I can include ReFrame

[ocaisa]

PR for additional packages opened in #76