Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ifort module includes intel mpirun etc. #1032

Closed
bartoldeman opened this issue Nov 11, 2016 · 9 comments
Closed

ifort module includes intel mpirun etc. #1032

bartoldeman opened this issue Nov 11, 2016 · 9 comments
Milestone
v3.0

Comments

@bartoldeman
Copy link
Contributor

If I do:

module show ifort

I see this

prepend_path("PATH","/cvmfs/soft.cc/nix/1/easybuild/generic/software/Core/ifort/2017.1.132/bin")
prepend_path("PATH","/cvmfs/soft.cc/nix/1/easybuild/generic/software/Core/ifort/2017.1.132/compilers_and_libraries_2017.1.132/linux/bin")
prepend_path("PATH","/cvmfs/soft.cc/nix/1/easybuild/generic/software/Core/ifort/2017.1.132/compilers_and_libraries_2017.1.132/linux/bin/intel64")
prepend_path("PATH","/cvmfs/soft.cc/nix/1/easybuild/generic/software/Core/ifort/2017.1.132/compilers_and_libraries_2017.1.132/linux/mpi/intel64/bin")

The last one is surprising. I know it happens because the icc.py easyblock simply tries the mpi path and happens to find it in the ifort installation.

There are also some duplicate paths:

prepend_path("LIBRARY_PATH","/cvmfs/soft.cc/nix/1/easybuild/generic/software/Core/ifort/2017.1.132/lib")
prepend_path("LIBRARY_PATH","/cvmfs/soft.cc/nix/1/easybuild/generic/software/Core/ifort/2017.1.132/lib/intel64")
prepend_path("LIBRARY_PATH","/cvmfs/soft.cc/nix/1/easybuild/generic/software/Core/ifort/2017.1.132/compilers_and_libraries_2017.1.132/linux/compiler/lib/intel64")
prepend_path("LIBRARY_PATH","/cvmfs/soft.cc/nix/1/easybuild/generic/software/Core/ifort/2017.1.132/compilers_and_libraries_2017.1.132/linux/mpi/intel64")
prepend_path("LIBRARY_PATH","/cvmfs/soft.cc/nix/1/easybuild/generic/software/Core/ifort/2017.1.132/compilers_and_libraries_2017.1.132/linux/compiler/lib/intel64")
prepend_path("LIBRARY_PATH","/cvmfs/soft.cc/nix/1/easybuild/generic/software/Core/ifort/2017.1.132/compilers_and_libraries_2017.1.132/linux/mpi/intel64")

of these:

  • .../lib is useless
  • .../lib/intel64 is identical to .../compilers_and_libraries_2017.1.132/linux/compiler/lib/intel64 via symbolic links
  • the mpi library path should not be there as in PATH.
  • then there is repetition (harmless but adds a little overhead)

I think for mpi we can simply do this:

--- a/easybuild/easyblocks/i/icc.py
+++ b/easybuild/easyblocks/i/icc.py
@@ -159,7 +159,6 @@ class EB_icc(IntelBase):
                 'bin/intel64',
                 'debugger/gdb/intel64/bin',
                 'ipp/bin/intel64',
-                'mpi/intel64/bin',
                 'tbb/bin/emt64',
                 'tbb/bin/intel64',
             ])
@@ -170,7 +169,6 @@ class EB_icc(IntelBase):
                 'debugger/ipt/intel64/lib',
                 'ipp/lib/intel64',
                 'mkl/lib/intel64',
-                'mpi/intel64',
                 'tbb/lib/intel64/%s' % get_tbb_gccprefix(),
             ])

but I do not understand why mpi is there? impi.py does not depend on icc.py.
I'll submit a PR.

Bart
@boegel
Copy link
Member

boegel commented Nov 11, 2016

@bartoldeman It looks lik these were added in #756.

Also some of the other paths don't make sense for icc/ifort specifically, i.e. the mkl, tbb, ipp ones...

It would also be good to filter out the duplicate paths, I guess this could be done by resolving symlinks and filtering out duplicate paths before generating the prepend_path commands.

Maybe @wpoely86 or @DirkdeDraak can shed some light on this (#756 was based on his PR #691)?

@boegel boegel added this to the v3.1 milestone Nov 11, 2016
@bartoldeman
Copy link
Contributor Author

I think the above patch may not be the right solution; I'll figure out why the ifort installer installs the mpi component.

@bartoldeman
Copy link
Contributor Author

The strange thing is that all "mpi" components are not selected but somehow it is still installed:

== 2016-11-11 17:03:31,777 intelbase.py:136 DEBUG Intel components found: ['intel-comp-l-all-vars__noarch', 'intel-comp-l-all-common__noarch', 'intel-comp-l-ps-ss-bec-wrapper__x86_64', 'intel-comp-l-all-wrapper__x86_64', 'intel-comp-l-all__x86_64', 'intel-comp-l-ps-ss-bec__x86_64', 'intel-comp-l-ps__x86_64', 'intel-comp-l-ps-ss__x86_64', 'intel-openmp-l-all__i486', 'intel-openmp-l-ps-ss-bec__i486', 'intel-openmp-l-ps__i486', 'intel-openmp-l-ps-libs-jp__i486', 'intel-openmp-l-ps-jp__i486', 'intel-openmp-l-all__x86_64', 'intel-openmp-l-ps-ss-bec__x86_64', 'intel-openmp-l-ps-ss__x86_64', 'intel-openmp-l-ps-bec__x86_64', 'intel-openmp-l-ps-libs__x86_64', 'intel-openmp-l-ps__x86_64', 'intel-openmp-l-ps-libs-jp__x86_64', 'intel-openmp-l-ps-jp__x86_64', 'intel-tbb-libs__noarch', 'intel-comp-all-doc__noarch', 'intel-comp-ps-doc__noarch', 'intel-comp-ps-doc-jp__noarch', 'intel-icc-doc__noarch', 'intel-icc-ps-doc__noarch', 'intel-icc-ps-doc-jp__noarch', 'intel-icc-ps-ss-doc__noarch', 'intel-ifort-ps-doc__noarch', 'intel-ifort-ps-doc-jp__noarch', 'intel-icc-l-all-common__noarch', 'intel-icc-l-all-common-jp__noarch', 'intel-icc-l-ps-ss-bec-common__noarch', 'intel-icc-l-ps-common__noarch', 'intel-icc-l-all-wrapper__x86_64', 'intel-icc-l-ps-ss-bec-wrapper__x86_64', 'intel-icc-l-all__x86_64', 'intel-icc-l-ps-ss__x86_64', 'intel-icc-l-ps__x86_64', 'intel-icc-l-ps-ss-bec__x86_64', 'intel-ifort-l-ps-common__noarch', 'intel-ifort-l-ps-common-jp__noarch', 'intel-ifort-l-ps-wrapper__x86_64', 'intel-ifort-l-ps-wrapper-jp__x86_64', 'intel-mpirt-l-ps-wrapper__x86_64', 'intel-ifort-l-ps-jp__x86_64', 'intel-ifort-l-ps__x86_64', 'intel-mkl-common__noarch', 'intel-mkl__i486', 'intel-mkl__x86_64', 'intel-mkl-rt__i486', 'intel-mkl-ps-rt-jp__i486', 'intel-mkl-rt__x86_64', 'intel-mkl-ps-rt-jp__x86_64', 'intel-mkl-doc__noarch', 'intel-mkl-ps-doc__noarch', 'intel-mkl-ps-doc-jp__noarch', 'intel-mkl-gnu__i486', 'intel-mkl-gnu__x86_64', 'intel-mkl-gnu-rt__i486', 'intel-mkl-gnu-rt__x86_64', 'intel-mkl-ps-cluster-64bit__x86_64', 'intel-mkl-ps-cluster__noarch', 'intel-mkl-ps-cluster-rt__x86_64', 'intel-mkl-ps-common__noarch', 'intel-mkl-ps-common-jp__noarch', 'intel-mkl-ps-common__i486', 'intel-mkl-ps-common-64bit__x86_64', 'intel-mkl-ps-mic__x86_64', 'intel-mkl-ps-mic-rt__x86_64', 'intel-mkl-ps-mic-rt-jp__x86_64', 'intel-mkl-ps-mic-cluster__x86_64', 'intel-mkl-ps-mic-cluster-rt__x86_64', 'intel-mkl-ps-pgi__x86_64', 'intel-mkl-ps-pgi-rt__x86_64', 'intel-mkl-common-c__noarch', 'intel-mkl-common-c__i486', 'intel-mkl-common-c-64bit__x86_64', 'intel-mkl-ps-common-c__noarch', 'intel-mkl-doc-c__noarch', 'intel-mkl-ps-doc-c-jp__noarch', 'intel-mkl-ps-mic-c__x86_64', 'intel-mkl-ps-cluster-c__noarch', 'intel-mkl-ps-ss-tbb__i486', 'intel-mkl-ps-ss-tbb__x86_64', 'intel-mkl-ps-ss-tbb-rt__i486', 'intel-mkl-ps-ss-tbb-rt__x86_64', 'intel-mkl-ps-tbb-mic__x86_64', 'intel-mkl-ps-tbb-mic-rt__x86_64', 'intel-mkl-ps-pgi-c__x86_64', 'intel-mkl-gnu-c__i486', 'intel-mkl-gnu-c__x86_64', 'intel-mkl-ps-common-f__noarch', 'intel-mkl-ps-common-f__i486', 'intel-mkl-ps-common-f-64bit__x86_64', 'intel-mkl-ps-mic-f__x86_64', 'intel-mkl-ps-cluster-f__noarch', 'intel-mkl-ps-pgi-f__x86_64', 'intel-mkl-ps-doc-f__noarch', 'intel-mkl-ps-doc-f-jp__noarch', 'intel-mkl-ps-gnu-f-rt__i486', 'intel-mkl-ps-gnu-f-rt__x86_64', 'intel-mkl-ps-gnu-f__x86_64', 'intel-mkl-ps-gnu-f__i486', 'intel-mkl-ps-f95-common__noarch', 'intel-mkl-ps-f__i486', 'intel-mkl-ps-f__x86_64', 'intel-mkl-ps-f95-mic__x86_64', 'intel-ipp-l-common__noarch', 'intel-ipp-l-ps-common__noarch', 'intel-ipp-sta-ss-common__noarch', 'intel-ipp-l-st-devel__i486', 'intel-ipp-sta-ss-st-devel__i486', 'intel-ipp-l-ps-st-devel__i486', 'intel-ipp-l-st__i486', 'intel-ipp-l-st__x86_64', 'intel-ipp-l-mt__i486', 'intel-ipp-l-mt__x86_64', 'intel-ipp-l-st-devel__x86_64', 'intel-ipp-l-ps-st-devel__x86_64', 'intel-ipp-sta-ss-st-devel__x86_64', 'intel-ipp-l-mt-devel__i486', 'intel-ipp-l-mt-devel__x86_64', 'intel-ipp-l-doc__noarch', 'intel-ipp-l-ps-doc-jp__noarch', 'intel-crypto-ipp-st-devel__i486', 'intel-crypto-ipp-st-devel__x86_64', 'intel-crypto-ipp-ps-st-devel__x86_64', 'intel-crypto-ipp-st__x86_64', 'intel-crypto-ipp-st__i486', 'intel-crypto-ipp-mt-devel__i486', 'intel-crypto-ipp-mt-devel__x86_64', 'intel-crypto-ipp-mt__x86_64', 'intel-crypto-ipp-mt__i486', 'intel-crypto-ipp-ss-st-devel__i486', 'intel-crypto-ipp-ss-st-devel__x86_64', 'intel-crypto-ipp-common__noarch', 'intel-crypto-ipp-doc__noarch', 'intel-tbb-source__noarch', 'intel-tbb-devel__noarch', 'intel-tbb-common__noarch', 'intel-tbb-ps-common__noarch', 'intel-tbb-ss-common__noarch', 'intel-tbb-common-jp__noarch', 'intel-tbb-doc__noarch', 'intel-tbb-doc-jp__noarch', 'intel-daal__i486', 'intel-daal__x86_64', 'intel-daal-common__noarch', 'intel-daal-ps-common-jp__noarch', 'intel-daal-doc__noarch', 'intel-daal-ps-doc-jp__noarch', 'intel-imb__x86_64', 'intel-mpi-rt-core__x86_64', 'intel-mpi-rt-mic__x86_64', 'intel-mpi-sdk-core__x86_64', 'intel-mpi-sdk-mic__x86_64', 'intel-mpi-doc__x86_64', 'intel-mpi-samples__x86_64', 'intel-gdb-gt__x86_64', 'intel-gdb-gt-src__noarch', 'intel-gdb-gt-doc__noarch', 'intel-gdb-gt-doc-jp__noarch', 'intel-gdb__x86_64', 'intel-gdb-source__noarch', 'intel-gdb-python-source__noarch', 'intel-gdb-common__noarch', 'intel-gdb-doc__noarch', 'intel-ism__noarch', 'intel-icsxe__noarch', 'intel-ips__noarch', 'intel-ipsc__noarch', 'intel-ipsf__noarch', 'intel-compxe__noarch', 'intel-ccompxe__noarch', 'intel-fcompxe__noarch', 'intel-ipp-psxe__noarch', 'intel-mkl-psxe__noarch', 'intel-tbb-psxe__noarch', 'intel-daal-psxe__noarch', 'intel-mpi-psxe__x86_64', 'intel-mpi-rt-psxe__x86_64', 'intel-psf-intel__x86_64', 'intel-gdb-ps-doc__noarch', 'intel-gdb-ps-doc-jp__noarch', 'intel-gdb-ps-mic__x86_64', 'intel-gdb-ps-mic-source__noarch', 'intel-gdb-ps-mic-cdt__x86_64', 'intel-gdb-ps-mic-cdt-source__noarch', 'intel-gdb-ps-mic-mpm__x86_64', 'intel-gdb-ps-mic-doc__noarch', 'intel-gdb-ps-mic-doc-jp__noarch', 'intel-psxe-common__noarch', 'intel-psxe-doc__noarch', 'intel-icsxe-doc__noarch', 'intel-ips-doc__noarch', 'intel-ipsc-doc__noarch', 'intel-ipsf-doc__noarch', 'intel-compxe-doc__noarch', 'intel-ccompxe-doc__noarch', 'intel-fcompxe-doc__noarch', 'intel-icsxe-pset']
== 2016-11-11 17:03:31,777 intelbase.py:137 DEBUG Using regex list: ['intel-comp', 'intel-fcomp', 'intel-ifort', 'intel-openmp', 'intel-ipsf?_']
== 2016-11-11 17:03:31,778 intelbase.py:151 DEBUG Components to install: ['intel-comp-l-all-vars__noarch', 'intel-comp-l-all-common__noarch', 'intel-comp-l-ps-ss-bec-wrapper__x86_64', 'intel-comp-l-all-wrapper__x86_64', 'intel-comp-l-all__x86_64', 'intel-comp-l-ps-ss-bec__x86_64', 'intel-comp-l-ps__x86_64', 'intel-comp-l-ps-ss__x86_64', 'intel-comp-all-doc__noarch', 'intel-comp-ps-doc__noarch', 'intel-comp-ps-doc-jp__noarch', 'intel-compxe__noarch', 'intel-compxe-doc__noarch', 'intel-fcompxe__noarch', 'intel-fcompxe-doc__noarch', 'intel-ifort-ps-doc__noarch', 'intel-ifort-ps-doc-jp__noarch', 'intel-ifort-l-ps-common__noarch', 'intel-ifort-l-ps-common-jp__noarch', 'intel-ifort-l-ps-wrapper__x86_64', 'intel-ifort-l-ps-wrapper-jp__x86_64', 'intel-ifort-l-ps-jp__x86_64', 'intel-ifort-l-ps__x86_64', 'intel-openmp-l-all__i486', 'intel-openmp-l-ps-ss-bec__i486', 'intel-openmp-l-ps__i486', 'intel-openmp-l-ps-libs-jp__i486', 'intel-openmp-l-ps-jp__i486', 'intel-openmp-l-all__x86_64', 'intel-openmp-l-ps-ss-bec__x86_64', 'intel-openmp-l-ps-ss__x86_64', 'intel-openmp-l-ps-bec__x86_64', 'intel-openmp-l-ps-libs__x86_64', 'intel-openmp-l-ps__x86_64', 'intel-openmp-l-ps-libs-jp__x86_64', 'intel-openmp-l-ps-jp__x86_64', 'intel-ips__noarch', 'intel-ipsf__noarch']

@bartoldeman
Copy link
Contributor Author

There is a funny dependency on intel-mpi-rt-mic in the core ifort rpm

$ rpm -qRp intel-ifort-l-ps-132-17.0.1-132.x86_64.rpm 
warning: intel-ifort-l-ps-132-17.0.1-132.x86_64.rpm: Header V3 RSA/SHA1 Signature, key ID 7a5a985f: NOKEY
intel-ifort-l-ps-common-132 = 17.0.1
intel-mpi-rt-mic-132 = 2017.1
intel-openmp-l-ps-132 = 17.0.1
intel-comp-l-ps-132 = 17.0.1
/bin/sh
rpmlib(PayloadFilesHavePrefix) <= 4.0-1
rpmlib(CompressedFileNames) <= 3.0.4-1
$ rpm -qRp intel-mpi-rt-mic-132-2017.1-132.x86_64.rpm 
warning: intel-mpi-rt-mic-132-2017.1-132.x86_64.rpm: Header V3 RSA/SHA1 Signature, key ID 7a5a985f: NOKEY
coreutils
intel-mpi-rt-core-132 = 2017.1
/bin/sh
rpmlib(PayloadFilesHavePrefix) <= 4.0-1
rpmlib(CompressedFileNames) <= 3.0.4-1

@bartoldeman
Copy link
Contributor Author

in the mediaconfig.xml this is encoded as follows:

<Product id="33" platform="IA32;INTEL64">
...
<Component depend="1.3;33.111;33.112;1.432;118.3" id="114" invisible="0" mandatory="0" platform="INTEL64">
      <Name>Intel Fortran Compiler for Intel(R) 64</Name>

then we have this for 118.3:

<Product id="118" platform="INTEL64">
    <Name>Intel(R) MPI Library 2017 Update 1</Name>
..
    <Component depend="118.2" id="3" invisible="1" mandatory="0" platform="INTEL64">
      <Name>Intel MPI Library, Runtime Environment for applications running on Intel(R) Many Integrated Core Architecture</Name>
...

Must be some kind of Intel bug :(

@bartoldeman
Copy link
Contributor Author

This patch referred in the easyconfig does the trick for mpi:

--- pset/mediaconfig.xml.orig      2016-10-20 09:02:57.000000000 +0000
+++ pset/mediaconfig.xml   2016-11-11 20:09:08.084892189 +0000
@@ -1551,7 +1551,7 @@
       <MapFileDir>${BUNDLE_MAPFILE_DIR}</MapFileDir>
       <ChklicIA32>${BUNDLE_CHKLIC_IA32}</ChklicIA32>
     </Component>
-    <Component depend="1.3;33.111;33.112;1.432;118.3" id="114" invisible="0" mandatory="0" platform="INTEL64">
+    <Component depend="1.3;33.111;33.112;1.432" id="114" invisible="0" mandatory="0" platform="INTEL64">
       <Name>Intel Fortran Compiler for Intel(R) 64</Name>
       <Description>Intel(R) Fortran Compiler 17.0 Update 1</Description>
       <Name locale="ja_JP.UTF-8">インテル(R) Fortran コンパイラー (インテル(R) 64)</Name>

@wpoely86
Copy link
Member

Seems like an intel issue indeed. Not the first nor the last strange thing their installer does.

If we have double paths, we should just clean them out. The whole intel install is a mess with all those symlinks.

@ocaisa ocaisa mentioned this issue Nov 14, 2016
Open
bartoldeman added a commit to bartoldeman/easybuild-easyconfigs that referenced this issue Nov 14, 2016
@bartoldeman
Eliminate dependency on mpi-mic-rt in ifort.
b9ad02d 
This dependency causes Intel mpirun/mpiexec to be added to $PATH
when the ifort module is loaded, which is not always desirable,
for instance when OpenMPI is already loaded.

Fixes easybuilders/easybuild-easyblocks#1032.
@bartoldeman bartoldeman mentioned this issue Nov 14, 2016
Merged
@boegel
Copy link
Member

boegel commented Nov 14, 2016

@bartoldeman this can be closed since easybuilders/easybuild-easyconfigs#3793 fixes this?

@boegel boegel modified the milestones: v3.0, v3.1 Nov 14, 2016
@boegel
Copy link
Member

boegel commented Nov 15, 2016

fixed in easybuilders/easybuild-easyconfigs#3793

@boegel boegel closed this as completed Nov 15, 2016
@bartoldeman bartoldeman mentioned this issue Mar 29, 2017
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
v3.0
Development

No branches or pull requests
3 participants
@wpoely86 @boegel @bartoldeman