one GAMESS-US easyblock to rule them all (combines #470 and #544) (WIP) #558

boegel · 2015-02-14T09:43:24Z

This easyblock combines the ones in PR #470 by @pescobar and #544 by @scintimandrion, picking the best of both and combining it in one GAMESS-US easyblock to rule them all.

Need a bit more test before I consider this final.

…ation.

…_us_to_rule_them_all

…ne_gamess_us_to_rule_them_all

hpcugentbot · 2015-02-14T09:44:32Z

Refer to this link for build results (access rights to CI server needed):
https://jenkins1.ugent.be/job/easybuild-easyblocks-pr-builder/780/
Test PASSed.

boegel · 2015-02-14T09:50:36Z

easybuild/easyblocks/g/gamess_us.py

+        try:
+            for line in fileinput.input(rungms, inplace=1, backup='.orig'):
+                line = re.sub(r"^(\s*set\s*TARGET)=.*", r"\1=%s" % self.cfg['ddi_comm'], line)
+                line = re.sub(r"^(\s*set\s*GMSPATH)=.*", r"\1=%s" % self.installdir, line)


TODO: also define ERICFMT and MCPPATH, see https://github.com/hpcugent/easybuild-easyconfigs/pull/1352/files#diff-e899eb0decac3e9b1edfaf407ba28764R29

…so patch in definiing ERICFMT and MCPPATH in rungms

hpcugentbot · 2015-02-14T18:13:07Z

Refer to this link for build results (access rights to CI server needed):
https://jenkins1.ugent.be/job/easybuild-easyblocks-pr-builder/781/
Test PASSed.

hpcugentbot · 2015-02-14T19:16:35Z

Refer to this link for build results (access rights to CI server needed):
https://jenkins1.ugent.be/job/easybuild-easyblocks-pr-builder/782/
Test PASSed.

hpcugentbot · 2015-02-14T19:23:36Z

Refer to this link for build results (access rights to CI server needed):
https://jenkins1.ugent.be/job/easybuild-easyblocks-pr-builder/783/
Test PASSed.

boegel · 2015-03-01T20:17:55Z

@scintimandrion: ping w.r.t. to this PR and how it relates to yours?

valtandor · 2015-03-02T04:06:13Z

I'll have a look at it, now that I'm back in the office.

valtandor · 2015-03-02T04:14:04Z

Is there any chance of me being granted push access to this PR?

boegel · 2015-03-02T08:51:03Z

@scintimandrion: if you have any suggestions, just make an inline remarks (via https://github.com/hpcugent/easybuild-easyblocks/pull/558/files), or issue a PR to my branch boegel:one_gamess_us_to_rule_them_all

hpcugentbot · 2015-03-03T10:06:32Z

Refer to this link for build results (access rights to CI server needed):
https://jenkins1.ugent.be/job/easybuild-easyblocks-pr-builder/800/
Test PASSed.

hpcugentbot · 2015-03-03T10:07:02Z

Refer to this link for build results (access rights to CI server needed):
https://jenkins1.ugent.be/job/easybuild-easyblocks-pr-builder/801/
Test PASSed.

boegel · 2015-03-03T10:09:17Z

@wpoely86: care to give this a review, maybe even a test? I really want to get this in for EasyBuild v2.0

easyconfigs available at easybuilders/easybuild-easyconfigs#1153 and easybuilders/easybuild-easyconfigs#1406

hpcugentbot · 2015-03-03T11:10:54Z

Refer to this link for build results (access rights to CI server needed):
https://jenkins1.ugent.be/job/easybuild-easyblocks-pr-builder/802/
Test PASSed.

wpoely86 · 2015-03-03T11:26:41Z

easybuild/easyblocks/g/gamess_us.py

@@ -0,0 +1,275 @@
+##
+# Copyright 2009-2014 Ghent University


boegel · 2015-03-04T17:38:11Z

works like a charm for old/new GAMESS-US versions, when built with intel/2015a (see easybuilders/easybuild-easyconfigs#1406)

some care has been taken to also make it work on top of GCC/OpenMPI, but a patch to rungms is required to make rungms (used in the test_step) work on top of OpenMPI... (see also easybuilders/easybuild-easyconfigs#1153)

@pescobar: can you test easybuilders/easybuild-easyconfigs#1406?

hpcugentbot · 2015-03-04T17:43:56Z

Refer to this link for build results (access rights to CI server needed):
https://jenkins1.ugent.be/job/easybuild-easyblocks-pr-builder/817/
Test PASSed.

… rungms

hpcugentbot · 2015-03-05T10:59:45Z

Refer to this link for build results (access rights to CI server needed):
https://jenkins1.ugent.be/job/easybuild-easyblocks-pr-builder/820/
Test PASSed.

hpcugentbot · 2015-03-05T11:14:13Z

Refer to this link for build results (access rights to CI server needed):
https://jenkins1.ugent.be/job/easybuild-easyblocks-pr-builder/821/
Test PASSed.

pescobar · 2015-03-05T12:48:48Z

when trying to compile this with toolchain intel/2015.02 still uses gcc (this is the debug log)

Compiling DDI object: ddi_scatter_acc.o
gcc -DLINUX -m64 -O3 -fstrict-aliasing -I./ -I/opt/soft/apps/impi/5.0.3.048-iccifort-2015.2.164-GCC-4.9.2/include64 -DDDI_MPI -DUSE_SYSV -DMAX_SMP_PROCS=16 -DMAX_NODES=256 -o ddi_scatter_acc.o -c ddi_scatter_acc.c
endif
unset echo
Finished compiling: ddi_scatter_acc.o

Compiling FORTRAN wrappers for DDI
gcc -DLINUX -m64 -O3 -fstrict-aliasing -I./ -I/opt/soft/apps/impi/5.0.3.048-iccifort-2015.2.164-GCC-4.9.2/include64 -DDDI_MPI -DUSE_SYSV -DMAX_SMP_PROCS=16 -DMAX_NODES=256 -DINT_SIZE=long -D_UNDERSCORES=1 -o ddi_fortran.o -c ddi_fortran.c
unset echo
Finished compiling: ddi_fortran.o

Also during the testing phase it asks for my ssh key????? This is what I get in the log file

== 2015-03-05 13:44:41,422 main.run DEBUG run_cmd: running cmd TMPDIR=/tmp/easybuild-JVe_h_/tmpQkJo7T I_MPI_FALLBACK=enable /home/urz/pub/.local/easybuild/software/GAMESS-US/20130501-R1-intel-2015.02/rungms exam01 20130501-R1 1 2 (in /tmp/easybuild-JVe_h_/tmpQkJo7T)
pub@maia password:

This is the content of script rungms which tries to run some mpi code and asks for the ssh key
https://gist.github.com/78be40a7620fce7ebaf0

boegel · 2015-03-05T13:03:35Z

@pescobar: it's normal that's it's using gcc for some things, e.g. compiling the DDI part:

$ sed -n "685,686p" ddi/compddi 
#       Note that in all cases, we use the gcc compiler from GNU for
#       the compilation of DDI, even if a non-GNU FORTRAN is used.

I'm not sure how we can get around the issue that rungms is assuming that your SSH setup is such that you can SSH to the host you're on... You can always skip the tests using runtest = False in the easyconfig file...

…_HYDRA_BOOTSTRAP to 'fork' to avoid use of SSH

hpcugentbot · 2015-03-05T13:36:28Z

Refer to this link for build results (access rights to CI server needed):
https://jenkins1.ugent.be/job/easybuild-easyblocks-pr-builder/822/
Test PASSed.

boegel · 2015-03-05T14:08:31Z

@pescobar: the issue you reported with the SSH password being prompted for should be fixed, I'm now setting $I_MPI_HYDRA_BOOTSTRAP=fork for the tests (the default is ssh)

hpcugentbot · 2015-03-05T15:01:40Z

Refer to this link for build results (access rights to CI server needed):
https://jenkins1.ugent.be/job/easybuild-easyblocks-pr-builder/823/
Test PASSed.

boegel · 2015-03-05T20:07:48Z

Tests with two versions of GAMESS-US and intel/2015a work on two different platforms (Debian and SL6), see easybuilders/easybuild-easyconfigs#1406, so good to go.

Building with goolf toolchain also works, but tests need to be skipped since rungms doesn't support OpenMPI (yet), see easybuilders/easybuild-easyconfigs#1153.

Going in, thanks for all the effort @scintimandrion and @pescobar, thanks for reviewing/testing @wpoely86!

one GAMESS-US easyblock to rule them all (combines #470 and #544) (WIP)

valtandor · 2015-03-30T03:30:02Z

Trying to build GAMESS-US 20130501-R1 with the ictce-5.4.0 toolchain gives this error:

== 2015-03-30 14:56:12,927 main.easyblock WARNING build failed (first 300 chars): EasyBuild crashed with an error (at easybuild/tools/run.py:332 in run_cmd_qa): run_cmd_qa: cmd  ./config  : Max nohits 50 reached: end of output tains one or more version number,
enter the specific version you prefer, giving all decimals.
b) If the above output contains paths like 'bin' and 'lib',
e

I guess this is because the output of GAMESS-US' config script is very verbose between questions?

boegel · 2015-03-30T08:16:28Z

That may indeed be the problem (e.g., things taking longer than on our end due to a slow filesystem), or for some reason a question is being generated that eb can't figure out how to solve...

Can you provide a full debug log for this? The error message you've pasted doesn't provide all information to really tell what's going on.

valtandor · 2015-03-30T22:48:26Z

I was able to resolve it by changing maxHitCount:

--- a/easybuild/tools/run.py
+++ b/easybuild/tools/run.py
@@ -251,7 +251,7 @@ def run_cmd_qa(cmd, qa, no_qa=None, log_ok=True, log_all=False, simple=False, re
     else:
         runLog = None

-    maxHitCount = 50
+    maxHitCount = 500

     try:
         p = Popen(cmd, shell=True, stdout=PIPE, stderr=STDOUT, stdin=PIPE, close_fds=True, executable="/bin/bash")

The other error I'm grappling with is a whole pile of segmentation faults. Still trying to chase that one down.

valtandor · 2015-03-31T00:15:28Z

Further to my previous comment, it seems that if I run a test through Easybuild, I get a segmentation fault, while if I run the same command on the command line on the same host, no segfault. I'm still trying to get to the bottom of the problem, but if you have any experience and tips to share where such things have happened before, I'd greatly appreciate some advice.

valtandor · 2015-03-31T01:25:37Z

Apparently, this is related to the interplay between EasyBuild and our scheduler (SLURM). There may be workarounds at this end.

pescobar and others added 12 commits September 4, 2014 13:53

mostly working with hardcoded paths

6f25aae

Initial commit of GAMESS-US EasyBlock, which is very much in progress

7584481

Updates to GAMESS-US easyblock

3c7abe4

Changes to GAMESS-US easyblock

93c7ced

Merge branch 'develop' into GAMESS-US

d47108f

Renamed gamess-us.py to gamessus.py, etc.

103fef2

Updated extract command

46794a1

Custom test and setting build version number

a692ad8

It's necessary to patch the rungms script after, not before, configur…

2a01219

…ation.

Merge remote-tracking branch 'github_pescobar/gamess' into one_gamess…

a9f2134

…_us_to_rule_them_all

Merge remote-tracking branch 'github_scintimandrion/GAMESS-US' into o…

125303a

…ne_gamess_us_to_rule_them_all

add cleaned up/enhanced easyblock for GAMESS-US

8fce4cb

boegel mentioned this pull request Feb 14, 2015

GAMESS-US: Patch rungms to support Slurm scheduler easybuilders/easybuild-easyconfigs#1352

Closed

boegel reviewed Feb 14, 2015
View reviewed changes

run tests one-by-one and verify correctness using provided script; al…

e4a2bbf

…so patch in definiing ERICFMT and MCPPATH in rungms

This was referenced Feb 14, 2015

add easyconfig file for GAMESS-US 2014.12.05-R1, including patch to make rungms work with SLURM easybuilders/easybuild-easyconfigs#1406

Merged

Gamess #470

Merged

GAMESS-US #544

Merged

make Q&A less strict to make it work with older GAMESS-US versions

164635a

fix escaping of ?

9decb56

boegel mentioned this pull request Feb 14, 2015

add easyconfig for GAMESS-US easybuilders/easybuild-easyconfigs#1153

Merged

correct MPI lib spec in GAMESS-US easyblock

c09dc42

wpoely86 reviewed Mar 3, 2015
View reviewed changes

easybuild/easyblocks/g/gamess_us.py

@@ -0,0 +1,275 @@

##

# Copyright 2009-2014 Ghent University

Copy link

Member

wpoely86 Mar 3, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

2015

copyright fix

f304630

define $TMPDIR and $I_MPI_FALLBACK when running GAMESS-US tests using…

71d7c2f

… rungms

Merge branch 'develop' into one_gamess_us_to_rule_them_all

51d09c7

avoid '['/']' characters in $TMPDIR used for GAMESS-US, define $I_MPI…

1939aee

…_HYDRA_BOOTSTRAP to 'fork' to avoid use of SSH

don't try to change obtained tmpdir for tests, just complain (early)

86c9063

boegel added a commit that referenced this pull request Mar 5, 2015

Merge pull request #558 from boegel/one_gamess_us_to_rule_them_all

0433ce4

one GAMESS-US easyblock to rule them all (combines #470 and #544) (WIP)

boegel merged commit 0433ce4 into easybuilders:develop Mar 5, 2015

boegel deleted the one_gamess_us_to_rule_them_all branch March 5, 2015 20:08

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

one GAMESS-US easyblock to rule them all (combines #470 and #544) (WIP) #558

one GAMESS-US easyblock to rule them all (combines #470 and #544) (WIP) #558

boegel commented Feb 14, 2015

hpcugentbot commented Feb 14, 2015

boegel Feb 14, 2015

hpcugentbot commented Feb 14, 2015

hpcugentbot commented Feb 14, 2015

hpcugentbot commented Feb 14, 2015

boegel commented Mar 1, 2015

valtandor commented Mar 2, 2015

valtandor commented Mar 2, 2015

boegel commented Mar 2, 2015

hpcugentbot commented Mar 3, 2015

hpcugentbot commented Mar 3, 2015

boegel commented Mar 3, 2015

hpcugentbot commented Mar 3, 2015

wpoely86 Mar 3, 2015

boegel commented Mar 4, 2015

hpcugentbot commented Mar 4, 2015

hpcugentbot commented Mar 5, 2015

hpcugentbot commented Mar 5, 2015

pescobar commented Mar 5, 2015

boegel commented Mar 5, 2015

hpcugentbot commented Mar 5, 2015

boegel commented Mar 5, 2015

hpcugentbot commented Mar 5, 2015

boegel commented Mar 5, 2015

valtandor commented Mar 30, 2015

boegel commented Mar 30, 2015

valtandor commented Mar 30, 2015

valtandor commented Mar 31, 2015

valtandor commented Mar 31, 2015

one GAMESS-US easyblock to rule them all (combines #470 and #544) (WIP) #558

one GAMESS-US easyblock to rule them all (combines #470 and #544) (WIP) #558

Conversation

boegel commented Feb 14, 2015

hpcugentbot commented Feb 14, 2015

boegel Feb 14, 2015

Choose a reason for hiding this comment

hpcugentbot commented Feb 14, 2015

hpcugentbot commented Feb 14, 2015

hpcugentbot commented Feb 14, 2015

boegel commented Mar 1, 2015

valtandor commented Mar 2, 2015

valtandor commented Mar 2, 2015

boegel commented Mar 2, 2015

hpcugentbot commented Mar 3, 2015

hpcugentbot commented Mar 3, 2015

boegel commented Mar 3, 2015

hpcugentbot commented Mar 3, 2015

wpoely86 Mar 3, 2015

Choose a reason for hiding this comment

boegel commented Mar 4, 2015

hpcugentbot commented Mar 4, 2015

hpcugentbot commented Mar 5, 2015

hpcugentbot commented Mar 5, 2015

pescobar commented Mar 5, 2015

boegel commented Mar 5, 2015

hpcugentbot commented Mar 5, 2015

boegel commented Mar 5, 2015

hpcugentbot commented Mar 5, 2015

boegel commented Mar 5, 2015

valtandor commented Mar 30, 2015

boegel commented Mar 30, 2015

valtandor commented Mar 30, 2015

valtandor commented Mar 31, 2015

valtandor commented Mar 31, 2015