add nwchem #14725

ltalirz · 2021-04-25T12:06:09Z

Checklist

Title of this PR is meaningful: e.g. "Adding my_nifty_package", not "updated meta.yaml".
License file is packaged (see here for an example).
Source is from official source.
Package does not vendor other packages. (If a package uses the source of another package, they should be separate packages or the licenses of all packages need to be packaged).
If static libraries are linked in, the license of the static library is packaged.
Build number is 0.
A tarball (url) rather than a repo (e.g. git_url) is used in your recipe (see here for more details).
GitHub users listed in the maintainer section have posted a comment confirming they are willing to be listed there.
When in trouble, please check our knowledge base documentation before pinging a team.

conda-forge-linter · 2021-04-25T12:06:13Z

Hi! This is the friendly automated conda-forge-linting service.

I wanted to let you know that I linted all conda-recipes in your PR (recipes/nwchem) and found some lint.

Here's what I've got...

For recipes/nwchem:

The recipe must have some tests.
about/license_family 'ECL' not allowed. Allowed families are AGPL,
APACHE, BSD, CC, GPL, GPL2, GPL3, LGPL, MIT, MOZILLA, NONE, OTHER,
PROPRIETARY, PSF, and PUBLIC-DOMAIN.

conda-forge-linter · 2021-04-25T12:07:50Z

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipes/nwchem) and found it was in an excellent condition.

ltalirz · 2021-05-05T07:50:52Z

@edoapra

Your patch did change something - this was the error before:

export PREFIX=/home/conda/staged-recipes/build_artifacts/nwchem_1620151640657/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold
export SRC_DIR=/home/conda/staged-recipes/build_artifacts/nwchem_1620151640657/test_tmp

 =======================================================
 QM: Running a very small subset of the available tests 
 =======================================================


 Running tests/h2o_opt/h2o_opt 
 
     cleaning scratch
     copying input and verified output files
     running nwchem ($PREFIX/bin/nwchem)  with 2 processors 
 
     NWChem execution failed
            Nuclear Dipole moment (a.u.) 
            ----------------------------
        X                 Y               Z
 ---------------- ---------------- ----------------
    -0.0000000000     0.0000000000    -0.0000000000

 Attempting to read a basis set from a  non-existing file:
 /home/conda/staged-recipes/build_artifacts/nwchem_1620151640657/work/src/basis/libraries/6-31gs                                                                                                                                                                
 ------------------------------------------------------------------------
 bas_tag_lib: failed opening basis file                   0
 ------------------------------------------------------------------------
 ------------------------------------------------------------------------
  current input line : 
    66: task scf optimize
 ------------------------------------------------------------------------
 Attempting to read a basis set from a  non-existing file:
 ------------------------------------------------------------------------
 There is an error in the specified basis set
 ------------------------------------------------------------------------
 For more information see the NWChem manual at https://github.com/nwchemgit/nwchem/wiki


 For further details see manual section:                                                                                                                                                                                                                                                                
0:bas_tag_lib: failed opening basis file:Received an Error in Communication
 /home/conda/staged-recipes/build_artifacts/nwchem_1620151640657/work/src/basis/libraries/6-31gs                                                                                                                                                                
 bas_tag_lib: failed opening basis file                   0
 ------------------------------------------------------------------------
 ------------------------------------------------------------------------
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 0 in communicator MPI COMMUNICATOR 3 DUP FROM 0
with errorcode -1.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------
  current input line : 
     0: 
 ------------------------------------------------------------------------

error after (no longer tries to read from non-existing file):

 =======================================================
 QM: Running a very small subset of the available tests 
 =======================================================


 
 Running tests/h2o_opt/h2o_opt 
 
     cleaning scratch
     copying input and verified output files
     running nwchem ($PREFIX/bin/nwchem)  with 2 processors 
 
     NWChem execution failed
 Effective nuclear repulsion energy (a.u.)       7.3282379250

            Nuclear Dipole moment (a.u.) 
            ----------------------------
        X                 Y               Z
 ---------------- ---------------- ----------------
    -0.0000000000     0.0000000000    -0.0000000000

  bas_tag_lib: looking for O               6-31G*                                                                                                                                                                                                                                                         $PREFIX/s
 ------------------------------------------------------------------------
 bas_tag_lib: failed to open basis library                   0
 ------------------------------------------------------------------------
 ------------------------------------------------------------------------
  current input line : 
    66: task scf optimize
 ------------------------------------------------------------------------
 ------------------------------------------------------------------------
 There is an error in the specified basis set
 ------------------------------------------------------------------------
 For more information see the NWChem manual at https://github.com/nwchemgit/nwchem/wiki


 For further details see manual section:

What puzzles me is the original error message

 Attempting to read a basis set from a  non-existing file:
 /home/conda/staged-recipes/build_artifacts/nwchem_1620151640657/work/src/basis/libraries/6-31gs

This reads like nwchem is trying to look for the basis sets in the work directory, while it should be looking in $PREFIX/share/nwchem.
Is it possible that the line

export NWCHEM_BASIS_LIBRARY=$PREFIX/share/nwchem/libraries/

is not picked up by the tests?

However, I can't quite square that with the fact that your patch seems to have made the error message go away (the path to the work directory was actually not that long...)

ltalirz · 2021-05-05T21:18:05Z

thanks @edoapra , I've added your second patch but the error message seems to stay (almost) the same (not sure whether the repetition of bas_tag_lib: looking for O 6-31G* means anything here):

=======================================================
 QM: Running a very small subset of the available tests 
 =======================================================


 
 Running tests/h2o_opt/h2o_opt 
 
     cleaning scratch
     copying input and verified output files
     running nwchem ($PREFIX/bin/nwchem)  with 2 processors 
 
     NWChem execution failed
 Effective nuclear repulsion energy (a.u.)       7.3282379250

            Nuclear Dipole moment (a.u.) 
            ----------------------------
        X                 Y               Z
 ---------------- ---------------- ----------------
    -0.0000000000     0.0000000000    -0.0000000000

  bas_tag_lib: looking for O               6-31G*                                                                                                                                                                                                                                                         $PREFIX/s
 bas_tag_lib: failed to open basis library                   0
  bas_tag_lib: looking for O               6-31G*                                                                                                                                                                                                                                                         $PREFIX/s
 ------------------------------------------------------------------------
 bas_tag_lib: failed to open basis library                   0
 ------------------------------------------------------------------------
 ------------------------------------------------------------------------
  current input line : 
    66: task scf optimize
 ------------------------------------------------------------------------
 ------------------------------------------------------------------------
 There is an error in the specified basis set
 ------------------------------------------------------------------------
 For more information see the NWChem manual at https://github.com/nwchemgit/nwchem/wiki

edoapra · 2021-05-05T21:21:07Z

@ltalirz Could you cat the full output file?

cat $NWCHEM_TOP/QA/testoutputs/h2o_opt.out

for unknown reasons, nwchem thinks the C compiler binary is `gcc`, despite a different value of the CC environment variable. Let's see whether exporting the value makes any difference...

first without passing it to make explicitly. if that doesn't help, we'll need to do that as well

https://conda-forge.org/docs/maintainer/knowledge_base.html

probably created somewhere else - blas problem should anyhow be solved

if that doesn't help, we'll need to figure out how to pass the `--oversubscribe` option to `mpirun`

ltalirz · 2021-05-05T23:42:02Z

sure!

by the way, I've also invited you as a collaborator to my fork in case you'd like to try any changes directly

ltalirz · 2021-05-06T12:04:36Z

I think the relevant bits are

  warning:::::::::::::: from_environment
  NWCHEM_BASIS_LIBRARY set to: <$PREFIX/sh>
  but file does not exist !
  using .nwchemrc or compiled library
  warning:::::::::::::: from_nwchemrc
  NWCHEM_BASIS_LIBRARY set to: </home/conda/staged-recipes/build_artifacts/nwchem_1620258323068/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_>
  but file does not exist !
  using compiled in library
  warning:::::::::::::: from_compile
  NWCHEM_BASIS_LIBRARY is: </home/conda/staged-recipes/build_artifacts/nwchem_1620258323068/work/src/basis/libraries/>
  but file does not exist or you do not have access to it !
 Attempting to read a basis set from a  non-existing file:
 /home/conda/staged-recipes/build_artifacts/nwchem_1620258323068/work/src/basis/libraries/aug-cc-pvdz

I guess in the first case the output is somehow cut short - this looks like the correct path (but nwchem fails to read it because the path is too long).
In the second case I'm not quite sure where the path comes from but it probably does not exist.
In the third case it looks inside the test's working directory, where the library also does not exist.

I will now add it to the $SRC directory and point nwchem there.
This might solve the issue

ltalirz · 2021-05-06T14:21:30Z

Looks to me like we're getting close...

now I get

 =======================================================
 QM: Running a very small subset of the available tests 
 =======================================================


 
 Running tests/h2o_opt/h2o_opt 
 
     cleaning scratch
     copying input and verified output files
     running nwchem ($PREFIX/bin/nwchem)  with 2 processors 
 
     verifying output ... nwparse.pl failed on test output h2o_opt.out
 
Failed

Is there a way to see a more verbose output of the parsing script?
In general, does the script make assumptions about any files being present other than the contents of the QA folder?

edoapra · 2021-05-06T16:11:53Z

Do you have perl installed? If perl is installed, it might well be some required perl components are missing.
You can see what the nwparse.pl script does by executing

cd $NWCHEM_TOP/QA
perl nwparse.pl testoutputs/h2o_opt.out

PS:I have managed to reproduce this error message by unistalling perl

ltalirz · 2021-05-06T17:51:27Z

Not sure if changing the stacksize is going to help

You would suggest ulimit -s unlimited?

ltalirz · 2021-05-06T17:54:29Z

I guess I'll wait until the tests pass on edoapra/nwchem@c8e634a before including the patch

ltalirz · 2021-05-06T17:56:12Z

better change
make 64_to_32
to
make 64_to_32 DEPEND_CC=${CC}
since this stage is already trying to use DEPEND_CC

but no _CC=${_CC} for make 64_to_32 or also that?

edoapra · 2021-05-06T17:57:48Z

I guess I'll wait until the tests pass on edoapra/nwchem@c8e634a before including the patch

Right. The first logs are promising since they show no errors when using depend.x
https://github.com/edoapra/nwchem/runs/2520622096?check_suite_focus=true
I would add the patch and see if it fixes the seg fault.
https://github.com/edoapra/nwchem/commit/c8e634a5378cb30802938b362941f97aae144b21.patch

edoapra · 2021-05-06T18:01:16Z

better change
make 64_to_32
to
make 64_to_32 DEPEND_CC=${CC}
since this stage is already trying to use DEPEND_CC

but no _CC=${_CC} for make 64_to_32 or also that?

It should not be needed, but it would not hurt to add it. In other words

make 64_to_32 DEPEND_CC=${CC} CC=${CC}  _CC=${CC}

You might want to add DEPEND_CC=${CC} to the make nwchem_config line, too, just to be extra cautious

make CC=${CC} _CC=${_CC} FC=${FC} _FC=${_FC}  DEPEND_CC=${CC} nwchem_config

ltalirz · 2021-05-06T18:24:19Z

On the previous commit it looks to me like the tests are passing, i.e. I think there is hope that we've arrived!

As mentioned previously, if you'd like to be added as a co-maintainer of this conda-forge feedstock, just let me know.

ltalirz · 2021-05-06T23:07:53Z

@conda-forge/staged-recipes Sorry for asking but I've done a couple of feedstocks by now and this has never happened to me: We just managed to make the test stage pass here, and it looks to me like we are caught in an infinite "build-test-loop"(?)

If you look at the raw logs (output is already a bit long for the interactive one) and search for BUILD START I have already 4 hits (after the second one, history just repeats itself).

Any ideas what might be causing this?

edoapra · 2021-05-07T00:32:41Z

The runs are marked at successful now (after more than 5 hours).
While trying to follow the log when the job(s) were running, I was under the impression that the log of the penultimate job (https://github.com/conda-forge/staged-recipes/runs/2522601261) seemed to be containing part of the last log (https://github.com/conda-forge/staged-recipes/runs/2522963263) and the other way around, too ... In other words, two running jobs seems to stepping into each other.
Could you try to run the checks again from scratch to see if my theory has any validity?

ltalirz · 2021-05-07T08:27:36Z

Hmm I can no longer access the logs from https://github.com/conda-forge/staged-recipes/runs/2522601261 (build not found ).

Let's maybe wait for someone from @conda-forge/staged-recipes to have a look before I start doing something that ends up removing the latest logs as well.
I think it is likely that we are not the first ones to observe this.

edoapra · 2021-05-07T16:23:49Z

According to
https://github.com/conda-forge/staged-recipes#11-some-checks-failed-but-it-wasnt-my-recipe-how-do-i-trigger-a-rebuild
you can trigger a rebuild by typing

git commit --amend
git push -f

ltalirz · 2021-05-08T06:55:15Z

Ok, for the record I'm attaching the raw log output of https://github.com/conda-forge/staged-recipes/runs/2522963263 :
log.zip.

Will now restart the build to see what happens.

isuruf · 2021-05-08T07:08:01Z

search for BUILD START I have already 4 hits

That's for each python version. (3.6, 3.7, 3.8, 3.9)

ltalirz · 2021-05-08T07:21:09Z

Thanks a lot for the hint @isuruf !

@edoapra the python dependency under run came from the recipe I started with - is it actually needed for running nwchem (e.g. do you ship some python scripts)?

if not, I can remove it from run.

edoapra · 2021-05-10T02:21:00Z

@edoapra the python dependency under run came from the recipe I started with - is it actually needed for running nwchem (e.g. do you ship some python scripts)?

Yes, NWChem does have python interface.
It is not strictly necessary to run NWChem. However, the Python interface adds more flexibility for running NWChem.

ltalirz · 2021-05-10T10:40:01Z

Thanks! That leads me to some follow-up questions:

which python versions are supported?
(in practice, it only matters whether python 3.6-3.9 are supported)
is python used during the nwchem build stage?
if yes, does the python version used during the build stage need to be the same version as the one in the environment where nwchem is installed (e.g. are you building any C-extensions, ...)

edoapra · 2021-05-10T17:39:27Z

Thanks! That leads me to some follow-up questions:

* which python versions are supported?
  (in practice, it only matters whether python 3.6-3.9 are supported)

* is python used during the nwchem build stage?
  if yes, does the python version used during the build stage need to be the same version as the one in the environment where nwchem is installed (e.g. are you building any C-extensions, ...)

Python versions 3.6-3.9 are supported
Python is used in the NWChem build stage.
I am guessing the Python version used during the build stage needs to be the same version as the one in the environment where nwchem is installed, but I cannot claim to be fluent in Python.
All the source code for the NWChem/Python is in the directory https://github.com/nwchemgit/nwchem/tree/master/src/python and it is used to create a static library linked to create the NWChem binary

ltalirz · 2021-05-10T21:27:48Z

Thanks, in that case the individual builds for the different python versions may be necessary.
It may make sense to eventually split off the python interface into a separate package (say, nwchempy) that depends on nwchem in order to save build time.

Perhaps the conda-forge maintainers have another/better suggestion for this.

ltalirz · 2021-05-10T21:28:35Z

@conda-forge/staged-recipes, ready for review

This was referenced Apr 26, 2021

consider transitioning code installation to conda marvel-nccr/quantum-mobile#161

Open

nwchem conda package on conda-forge? nwchemgit/nwchem#359

Closed

ltalirz added 23 commits May 5, 2021 23:54

add nwchem

d53787b

fix linter

720d541

fix download link

b3c9a4e

fix directory name

973f162

add lapack_lib

8acf90d

add openssh

46f8478

try to change compiler

256884b

for unknown reasons, nwchem thinks the C compiler binary is `gcc`, despite a different value of the CC environment variable. Let's see whether exporting the value makes any difference...

pass CC directly to make

ad18f88

fix nwchem_config path

8f923a4

try fixing nwchem_config path

b73bc9b

add var explicitly to make

e70b92c

try exporting CC_ FC_

2f53313

first without passing it to make explicitly. if that doesn't help, we'll need to do that as well

pass vars to make line explicitly

59a692f

typo

47a1e52

incorporate suggestions by developers

a4ae729

fix missing dependencies

06d7191

fix obvious issues with tests

99f1dc4

change blas deps according to

5958ec7

https://conda-forge.org/docs/maintainer/knowledge_base.html

fix libdir

ec8dee1

remove cat config.log

943ddf3

probably created somewhere else - blas problem should anyhow be solved

add libopenblas as explicit dependency

7a68507

fix mpi tests with openmpi

340091f

try reducing number of MPI tasks

e603b9f

if that doesn't help, we'll need to figure out how to pass the `--oversubscribe` option to `mpirun`

copy libraries to shorter path

2076185

add perl as test requirement

dba25ee

edoapra added a commit to edoapra/nwchem that referenced this pull request May 6, 2021

increased buffer size to 1024 conda-forge/staged-recipes#14725 (comment)

c8e634a

edoapra added a commit to edoapra/nwchem that referenced this pull request May 6, 2021

increased buffer size to 1024 conda-forge/staged-recipes#14725 (comment)

a21982c

add patch for depend.x

7e53550

ltalirz force-pushed the add-nwchem branch from 0e58956 to 7e53550 Compare May 8, 2021 06:55

chrisburr merged commit d22b00f into conda-forge:master May 15, 2021

ltalirz mentioned this pull request May 15, 2021

speed up build conda-forge/nwchem-feedstock#2

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add nwchem #14725

add nwchem #14725

ltalirz commented Apr 25, 2021 •

edited

Loading

conda-forge-linter commented Apr 25, 2021

conda-forge-linter commented Apr 25, 2021

ltalirz commented May 5, 2021 •

edited

Loading

ltalirz commented May 5, 2021

edoapra commented May 5, 2021

ltalirz commented May 5, 2021

ltalirz commented May 6, 2021 •

edited

Loading

ltalirz commented May 6, 2021

edoapra commented May 6, 2021 •

edited

Loading

ltalirz commented May 6, 2021

ltalirz commented May 6, 2021

ltalirz commented May 6, 2021

edoapra commented May 6, 2021

edoapra commented May 6, 2021

ltalirz commented May 6, 2021 •

edited

Loading

ltalirz commented May 6, 2021 •

edited

Loading

edoapra commented May 7, 2021

ltalirz commented May 7, 2021

edoapra commented May 7, 2021

ltalirz commented May 8, 2021 •

edited

Loading

isuruf commented May 8, 2021

ltalirz commented May 8, 2021 •

edited

Loading

edoapra commented May 10, 2021

ltalirz commented May 10, 2021 •

edited

Loading

edoapra commented May 10, 2021

ltalirz commented May 10, 2021 •

edited

Loading

ltalirz commented May 10, 2021

add nwchem #14725

add nwchem #14725

Conversation

ltalirz commented Apr 25, 2021 • edited Loading

conda-forge-linter commented Apr 25, 2021

conda-forge-linter commented Apr 25, 2021

ltalirz commented May 5, 2021 • edited Loading

ltalirz commented May 5, 2021

edoapra commented May 5, 2021

ltalirz commented May 5, 2021

ltalirz commented May 6, 2021 • edited Loading

ltalirz commented May 6, 2021

edoapra commented May 6, 2021 • edited Loading

ltalirz commented May 6, 2021

ltalirz commented May 6, 2021

ltalirz commented May 6, 2021

edoapra commented May 6, 2021

edoapra commented May 6, 2021

ltalirz commented May 6, 2021 • edited Loading

ltalirz commented May 6, 2021 • edited Loading

edoapra commented May 7, 2021

ltalirz commented May 7, 2021

edoapra commented May 7, 2021

ltalirz commented May 8, 2021 • edited Loading

isuruf commented May 8, 2021

ltalirz commented May 8, 2021 • edited Loading

edoapra commented May 10, 2021

ltalirz commented May 10, 2021 • edited Loading

edoapra commented May 10, 2021

ltalirz commented May 10, 2021 • edited Loading

ltalirz commented May 10, 2021

ltalirz commented Apr 25, 2021 •

edited

Loading

ltalirz commented May 5, 2021 •

edited

Loading

ltalirz commented May 6, 2021 •

edited

Loading

edoapra commented May 6, 2021 •

edited

Loading

ltalirz commented May 6, 2021 •

edited

Loading

ltalirz commented May 6, 2021 •

edited

Loading

ltalirz commented May 8, 2021 •

edited

Loading

ltalirz commented May 8, 2021 •

edited

Loading

ltalirz commented May 10, 2021 •

edited

Loading

ltalirz commented May 10, 2021 •

edited

Loading