Add initial support for a toolchain based module naming scheme #1168

ocaisa · 2015-02-11T14:48:50Z

where software is installed in a directory specific to each toolchain, and each toolchain module extends the module path variable.

NB This won't work unless toolchains properly inherit (iccifort -> iimpi ->intel) and/or dependencies are listed appropriately in toolchains.

The other required change in easyblock.py changed the order of statements written in the module file so that the 'module use' statement appears after loading the dependencies. This allows the module path to be updated in the correct order in this naming scheme.

software is installed in a directory specific to each toolchain, and each toolchain module extends the path. This won't work unless toolchains properly inherit (iccifort -> iimpi -> intel) and dependencies are listed appropriately in toolchains. The other required change in easyblock.py changed the order of statements written in the module file so that the 'module use' statement appears after loading the dependencies. This allows the module path to be updated in the correct order in this naming scheme.

hpcugentbot · 2015-02-11T14:48:51Z

Automatic reply from Jenkins: Can I test this?

ocaisa · 2015-02-11T15:04:26Z

Part of the reason for doing this is that, in a hierarchy, module spider returns pretty complicated ways to get to a particular module. Since we (choose to) expose users to our toolchains, this conflicts a little with how we explain access to installed software.

boegel · 2015-02-13T09:03:51Z

Jenkins: ok to test

boegel · 2015-02-13T09:05:33Z

easybuild/tools/module_naming_scheme/toolchain_mns.py

+        This is useful when toolchains are not exposed to users.
+        """
+#        return True
+# In our case we have to load the toolchains because they are explicitly exposed when extending the module path


indent comment

boegel · 2015-02-13T09:20:03Z

we would also need to have this tested by the unit tests, to ensure it doesn't get broken by changes in the framework in the future

Fixed the names...added some humour

hpcugentbot · 2015-02-13T09:46:38Z

Refer to this link for build results (access rights to CI server needed):
https://jenkins1.ugent.be/job/easybuild-framework-pr-builder/1337/
Test PASSed.

@boegel

Fixed comments from @boegel

Final deletions

ocaisa · 2015-02-13T10:05:23Z

I'll take a look at the tests you mentioned in #1176 and see how this might be done. I'm a little surprised it passed the tests because without modifying intel or goolf to include a dependency on iimpi/gompi (and then iteratively), you almost certainly can't install software within this MNS. It will work for gpsolf and intel-para though because I've already defined them hierarchically.

I'll be able to resolve this more completely once we implement the method for subtoolchains, since I can enforce the dependency then (given this MNS is in use).

boegel · 2015-02-13T10:06:26Z

The tests pass because they're not using this new MNS.. :-)

hpcugentbot · 2015-02-13T10:15:06Z

Refer to this link for build results (access rights to CI server needed):
https://jenkins1.ugent.be/job/easybuild-framework-pr-builder/1338/
Test PASSed.

hpcugentbot · 2015-02-13T10:43:00Z

Refer to this link for build results (access rights to CI server needed):
https://jenkins1.ugent.be/job/easybuild-framework-pr-builder/1339/
Test PASSed.

hpcugentbot · 2015-02-13T11:11:08Z

Refer to this link for build results (access rights to CI server needed):
https://jenkins1.ugent.be/job/easybuild-framework-pr-builder/1340/
Test PASSed.

hpcugentbot · 2015-02-13T11:38:48Z

Refer to this link for build results (access rights to CI server needed):
https://jenkins1.ugent.be/job/easybuild-framework-pr-builder/1341/
Test PASSed.

sync with develop

hpcugentbot · 2015-03-19T21:31:24Z

Refer to this link for build results (access rights to CI server needed):
https://jenkins1.ugent.be/job/easybuild-framework-pr-builder/1532/
Test FAILed.

boegel · 2015-03-19T21:41:11Z

@ocaisa: ignore the failing test, Jenkins must be drunk

boegel · 2015-03-19T21:41:16Z

Jenkins: test this please

hpcugentbot · 2015-03-19T22:34:00Z

Refer to this link for build results (access rights to CI server needed):
https://jenkins1.ugent.be/job/easybuild-framework-pr-builder/1533/
Test PASSed.

We currently still need to expand the toolchain load because of the current lack of support for subtoolchains within the framework

hpcugentbot · 2015-03-20T07:35:45Z

Refer to this link for build results (access rights to CI server needed):
https://jenkins1.ugent.be/job/easybuild-framework-pr-builder/1535/
Test PASSed.

…develop Conflicts: easybuild/framework/easyblock.py

hpcugentbot · 2015-09-22T11:26:04Z

EasyBuild framework unit test suite PASSed (see https://jenkins1.ugent.be/job/easybuild-framework-pr-builder/2082/console for more details).

This pull request is now ready for review/testing.

Please try and find someone who can tackle this; contact @boegel if you're not sure what to do.

boegel · 2015-09-22T12:11:28Z

@ocaisa: what's going on with your last commit? I'm not sure how useful an empty pull request is... ;)

hpcugentbot · 2015-09-22T13:18:26Z

EasyBuild framework unit test suite FAILed.

See https://jenkins1.ugent.be/job/easybuild-framework-pr-builder/2085/console for more details.

Please fix the reported issues by pushing additional commits to the branch corresponding with this pull request; contact @boegel if you're not sure what to do.

boegel · 2015-09-22T13:24:37Z

@ocaisa: last tests have failed due to an issue on Jenkins, I'm looking into it

boegel · 2015-09-22T13:27:27Z

Jenkins: test this please

ocaisa · 2015-09-22T14:11:00Z

Argh, I was tidying up everything for the subtoolchain stuff and I deleted the MNS from the branch history to keep things clean...but I never forked the original devel branch so things got complicated.

boegel · 2015-09-22T14:16:43Z

@ocaisa: git revert is your friend

hpcugentbot · 2015-09-22T14:18:33Z

EasyBuild framework unit test suite PASSed (see https://jenkins1.ugent.be/job/easybuild-framework-pr-builder/2090/console for more details).

This pull request is now ready for review/testing.

Please try and find someone who can tackle this; contact @boegel if you're not sure what to do.

…o develop

hpcugentbot · 2015-10-13T15:17:47Z

EasyBuild framework unit test suite PASSed (see https://jenkins1.ugent.be/job/easybuild-framework-pr-builder/2185/console for more details).

This pull request is now ready for review/testing.

Please try and find someone who can tackle this; contact @boegel if you're not sure what to do.

boegel · 2016-05-09T18:05:22Z

@ocaisa any idea why we ended up not including toolchain_mns.py? It was removed in 8e18877 .

@bartoldeman was hitting a problem with easybuilders/easybuild-easyconfigs#2899, basically because ToolchainMNS as it was is unaware of PGI (I think...)

Do you have an updated version of ToolchainMNS somewhere?

ocaisa · 2016-05-11T07:36:58Z

It couldn't work properly because it relied on knowing which toolchain it was inheriting from (you need to have iimpi loaded to correctly load intel since things built with iimpi are only available by the path extension provided by the iimpi module). Because eb requires components of a toolchain to be expanded (you can switch this resolution off but the framework complains then), this meant you had to explicitly include the subtoolchain as a dependency for it to work.
With the features introduced by minimal-toolchains this should no longer be necessary except that the toolchain hierarchy is only available to the easyconfig class and not the mns class (at present).

ocaisa · 2016-05-11T07:41:27Z

Annoyingly I pulled directly from my develop branch so I need to open a new PR from a fresh branch...though the scheme is still effectively broken and will only work with some manual editing of toolchain easyconfigs to add the appropriate subtoolchain dependencies

boegel · 2016-05-11T07:52:45Z

@ocaisa I was mainly asking because I was figuring out the bug that got fixed with #1754; if you know it doesn't work (well), there's probably little point in contributing it back?

Although, I have to admit, this is a scheme we would consider using, so maybe it is useful to have a PR for it that makes it clear what the known problems are.

ocaisa mentioned this pull request Feb 13, 2015

Hierarchical module naming scheme categorizing modulefiles by 'moduleclass' #1176

Merged

boegel reviewed Feb 13, 2015
View reviewed changes

Update toolchain_mns.py

4c60d07

Fixed the names...added some humour

ocaisa added 3 commits February 13, 2015 10:47

Update toolchain_mns.py

0a66721

Fixed comments from @boegel

Update toolchain_mns.py

a8f50b8

Update toolchain_mns.py

f842f54

Final deletions

boegel mentioned this pull request Mar 18, 2015

Move module path extension in module file #1232

Merged

Merge branch 'develop' into ocaisa_develop

436b5ec

This was referenced Mar 19, 2015

sync with develop ocaisa/easybuild-framework#1

Merged

sync with develop ocaisa/easybuild-framework#2

Merged

Merge pull request #2 from boegel/ocaisa_develop

bd65bfc

sync with develop

expand_toolchain_load still required

0fe7e1b

We currently still need to expand the toolchain load because of the current lack of support for subtoolchains within the framework

ocaisa mentioned this pull request Aug 19, 2015

include-module-naming-schemes doesn't seem to work #1367

Closed

Alan added 2 commits September 22, 2015 11:10

Merge branch 'develop' of github.com:ocaisa/easybuild-framework into …

4eae49d

…develop Conflicts: easybuild/framework/easyblock.py

removing files from version control

8e18877

Alan added 2 commits October 1, 2015 09:46

Merge branch 'develop' of github.com:hpcugent/easybuild-framework int…

f488cf4

…o develop

Merge branch 'develop' of github.com:hpcugent/easybuild-framework int…

2e96ad9

…o develop

boegel merged commit 2e96ad9 into easybuilders:develop Oct 14, 2015

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add initial support for a toolchain based module naming scheme #1168

Add initial support for a toolchain based module naming scheme #1168

ocaisa commented Feb 11, 2015

hpcugentbot commented Feb 11, 2015

ocaisa commented Feb 11, 2015

boegel commented Feb 13, 2015

boegel Feb 13, 2015

boegel commented Feb 13, 2015

hpcugentbot commented Feb 13, 2015

ocaisa commented Feb 13, 2015

boegel commented Feb 13, 2015

hpcugentbot commented Feb 13, 2015

hpcugentbot commented Feb 13, 2015

hpcugentbot commented Feb 13, 2015

hpcugentbot commented Feb 13, 2015

hpcugentbot commented Mar 19, 2015

boegel commented Mar 19, 2015

boegel commented Mar 19, 2015

hpcugentbot commented Mar 19, 2015

hpcugentbot commented Mar 20, 2015

hpcugentbot commented Sep 22, 2015

boegel commented Sep 22, 2015

hpcugentbot commented Sep 22, 2015

boegel commented Sep 22, 2015

boegel commented Sep 22, 2015

ocaisa commented Sep 22, 2015

boegel commented Sep 22, 2015

hpcugentbot commented Sep 22, 2015

hpcugentbot commented Oct 13, 2015

boegel commented May 9, 2016

ocaisa commented May 11, 2016

ocaisa commented May 11, 2016 •

edited

boegel commented May 11, 2016

Add initial support for a toolchain based module naming scheme #1168

Add initial support for a toolchain based module naming scheme #1168

Conversation

ocaisa commented Feb 11, 2015

hpcugentbot commented Feb 11, 2015

ocaisa commented Feb 11, 2015

boegel commented Feb 13, 2015

boegel Feb 13, 2015

Choose a reason for hiding this comment

boegel commented Feb 13, 2015

hpcugentbot commented Feb 13, 2015

ocaisa commented Feb 13, 2015

boegel commented Feb 13, 2015

hpcugentbot commented Feb 13, 2015

hpcugentbot commented Feb 13, 2015

hpcugentbot commented Feb 13, 2015

hpcugentbot commented Feb 13, 2015

hpcugentbot commented Mar 19, 2015

boegel commented Mar 19, 2015

boegel commented Mar 19, 2015

hpcugentbot commented Mar 19, 2015

hpcugentbot commented Mar 20, 2015

hpcugentbot commented Sep 22, 2015

boegel commented Sep 22, 2015

hpcugentbot commented Sep 22, 2015

boegel commented Sep 22, 2015

boegel commented Sep 22, 2015

ocaisa commented Sep 22, 2015

boegel commented Sep 22, 2015

hpcugentbot commented Sep 22, 2015

hpcugentbot commented Oct 13, 2015

boegel commented May 9, 2016

ocaisa commented May 11, 2016

ocaisa commented May 11, 2016 • edited

boegel commented May 11, 2016

ocaisa commented May 11, 2016 •

edited