Make NEST installation independent #318

Merged
merged 16 commits into from May 11, 2016

Conversation

Projects
None yet
3 participants
@tammoippen
Contributor

tammoippen commented Apr 25, 2016

This PR makes a NEST installation independent from the source and build directory, as well as from the actual installation path. Further, it respects the standard GNU sub-installation directories and allows to change them on the command-line.

  • #define NEST_SOURCEDIR "@PKGSRCDIR@" and #define NEST_BUILDDIR "@PKGBUILDDIR@" are removed, so NEST sources have no idea where to look for them.
  • As #314 points out, there are at leaste two tests failing, if the sources are not at the expected place.
    • regressiontests/ticket-659-copyright.py can now be deactivated by setting the environmental variable NEST_SOURCE to "SKIP"
    • the validate function does not regenerate the help files anymore for the test unittests/test_round_validate.sli
  • #317 and #303 expect NEST to use standard GNU install directories, that are changeable during configuring. This is made possible by using the GNUInstallDirs and replacing all nest specific install variables by the ones defined in there. During configuring, one can change the doc directory with cmake .... -DCMAKE_INSTALL_DOCDIR=documentation/of/nest
  • If NEST is build dynamically, it uses relative RPATH settings (@loader_path on OS X and $ORIGIN on Linux). This allows a NEST installation to be moved anywhere else. For NEST to find the sli init files, the help files and user modules (only PyNEST), some environmental variables have to be set, e.g. by sourcing nest_vars.sh.

This PR should fix #116, #303, #314 and #317. I propose @Hanke, @sanjayankur31, @jougs and @heplesser as reviewers.

@tammoippen tammoippen changed the title from Make NEST installation independant to Make NEST installation independent Apr 25, 2016

+# Install all stuff to NEST's install directories.
+set( CMAKE_INSTALL_LIBDIR ${NEST_LIBDIR}/nest CACHE STRING "object code libraries (lib/nest or lib64/nest or lib/<multiarch-tuple>/nest on Debian)" FORCE )
+set( CMAKE_INSTALL_DOCDIR ${NEST_DOCDIR} CACHE STRING "documentation root (DATAROOTDIR/doc/nest)" FORCE )
+set( CMAKE_INSTALL_DATADIR ${NEST_DATADIR} CACHE STRING "read-only architecture-independent data (DATAROOTDIR/nest)" FORCE )

This comment has been minimized.

@heplesser

heplesser Apr 25, 2016

Contributor

Is it intentional that these directories are relative paths? I suppose it is part of making nest relocatable?

@heplesser

heplesser Apr 25, 2016

Contributor

Is it intentional that these directories are relative paths? I suppose it is part of making nest relocatable?

This comment has been minimized.

@tammoippen

tammoippen Apr 25, 2016

Contributor

Yes. They should be relative paths. The full paths are created with include( GNUInstallDirs ) and have the form CMAKE_INSTALL_FULL_.... The relative paths are used in the install() function as a destination within the CMAKE_INSTALL_PREFIX.

@tammoippen

tammoippen Apr 25, 2016

Contributor

Yes. They should be relative paths. The full paths are created with include( GNUInstallDirs ) and have the form CMAKE_INSTALL_FULL_.... The relative paths are used in the install() function as a destination within the CMAKE_INSTALL_PREFIX.

+ echo "@static-libraries@"
+ ;;
+ --docdir)
+ echo "@CMAKE_INSTALL_DOCDIR@"

This comment has been minimized.

@heplesser

heplesser Apr 25, 2016

Contributor

--docdir will print the relative path of the location of the documentation, not the absolute path. Is that sensible? If a user wants to know where the documentation is, the absolute path would be required, wouldn't it? The same applies to --datadir and --libdir.

@heplesser

heplesser Apr 25, 2016

Contributor

--docdir will print the relative path of the location of the documentation, not the absolute path. Is that sensible? If a user wants to know where the documentation is, the absolute path would be required, wouldn't it? The same applies to --datadir and --libdir.

This comment has been minimized.

@tammoippen

tammoippen Apr 26, 2016

Contributor

The arguments nest-config --docdir are used in MyModule to get the correct install paths (see your comment here). I think for most cases, these options are not relevant for users, and if they actually need them, they can prepend the result from nest-config --prefix. I will adjust the documentation.

@tammoippen

tammoippen Apr 26, 2016

Contributor

The arguments nest-config --docdir are used in MyModule to get the correct install paths (see your comment here). I think for most cases, these options are not relevant for users, and if they actually need them, they can prepend the result from nest-config --prefix. I will adjust the documentation.

@heplesser

This comment has been minimized.

Show comment
Hide comment
@heplesser

heplesser Apr 25, 2016

Contributor

@tammoippen NEST built and tests passed on my system without problems. I experimented with relocation and ran into some trouble, but that may be due to my stupidity:

  • After moving the directory into which NEST was installed, I had to adjust bin/nest_vars.sh in that directory and then source that file into the shell to make the change effective. Is that right? Why did NEST work properly before relocating, even though I did not source nest_vars.sh then?
  • I tried to build MyModule after relocating the install dir and after sourcing nest_vars.sh. Cmake for MyModule still reported that it would install the module to lib/nest in the original install location, not in the new location. $NEST_MODULE_PATH contained the new location.
  • make installcheck no longer works after relocation as it doesn't find do_tests.sh.
Contributor

heplesser commented Apr 25, 2016

@tammoippen NEST built and tests passed on my system without problems. I experimented with relocation and ran into some trouble, but that may be due to my stupidity:

  • After moving the directory into which NEST was installed, I had to adjust bin/nest_vars.sh in that directory and then source that file into the shell to make the change effective. Is that right? Why did NEST work properly before relocating, even though I did not source nest_vars.sh then?
  • I tried to build MyModule after relocating the install dir and after sourcing nest_vars.sh. Cmake for MyModule still reported that it would install the module to lib/nest in the original install location, not in the new location. $NEST_MODULE_PATH contained the new location.
  • make installcheck no longer works after relocation as it doesn't find do_tests.sh.
@tammoippen

This comment has been minimized.

Show comment
Hide comment
@tammoippen

tammoippen Apr 25, 2016

Contributor

@heplesser

  • After moving the directory into which NEST was installed, I had to adjust bin/nest_vars.sh in that directory and then source that file into the shell to make the change effective. Is that right? Why did NEST work properly before relocating, even though I did not source nest_vars.sh then?

The default directories (within CMAKE_INSTALL_PREFIX) are "hardcoded" into NEST. This is why NEST is working, if it is not relocated. When you relocate, you have to change the "hardcoded" values with environmental variables, as nest_vars.sh do.

  • I tried to build MyModule after relocating the install dir and after sourcing nest_vars.sh. Cmake for MyModule still reported that it would install the module to lib/nest in the original install location, not in the new location. $NEST_MODULE_PATH contained the new location.

You are right. nest-config is hardcoded to contain the CMAKE_INSTALL_PREFIX. I will have a look into this.

  • make installcheck no longer works after relocation as it doesn't find do_tests.sh.

make thinks, it installed everything to CMAKE_INSTALL_PREFIX. When relocating, make does not know, where to look for do_tests.sh. But you can check the installation with

NEST_SOURCE=SKIP exec_prefix=<new-prefix> <new-prefix>/share/nest/extras/do_tests.sh --with-pynest
Contributor

tammoippen commented Apr 25, 2016

@heplesser

  • After moving the directory into which NEST was installed, I had to adjust bin/nest_vars.sh in that directory and then source that file into the shell to make the change effective. Is that right? Why did NEST work properly before relocating, even though I did not source nest_vars.sh then?

The default directories (within CMAKE_INSTALL_PREFIX) are "hardcoded" into NEST. This is why NEST is working, if it is not relocated. When you relocate, you have to change the "hardcoded" values with environmental variables, as nest_vars.sh do.

  • I tried to build MyModule after relocating the install dir and after sourcing nest_vars.sh. Cmake for MyModule still reported that it would install the module to lib/nest in the original install location, not in the new location. $NEST_MODULE_PATH contained the new location.

You are right. nest-config is hardcoded to contain the CMAKE_INSTALL_PREFIX. I will have a look into this.

  • make installcheck no longer works after relocation as it doesn't find do_tests.sh.

make thinks, it installed everything to CMAKE_INSTALL_PREFIX. When relocating, make does not know, where to look for do_tests.sh. But you can check the installation with

NEST_SOURCE=SKIP exec_prefix=<new-prefix> <new-prefix>/share/nest/extras/do_tests.sh --with-pynest
@heplesser

This comment has been minimized.

Show comment
Hide comment
@heplesser

heplesser Apr 25, 2016

Contributor

@tammoippen Thanks for the information. The correct option for testing is --test-pynest, but things fail, since the path to the tests scripts seems to be fixed (I originally installed in nest_tammo and then renamed that dir reloc:

[plesser@Hanss-MacBook-Pro] ~/tmp (0)$ NEST_SOURCE=SKIP exec_prefix=/Users/plesser/tmp/reloc/ /Users/plesser/tmp/reloc/share/nest/extras/do_tests.sh --test-pynest

Phase 1: Testing if SLI can execute scripts and report errors
-------------------------------------------------------------
  Running test 'selftests/test_pass.sli'... Failed: unexpected exit code 126
Running  mpirun -np 1 /Users/plesser/tmp/reloc/bin/nest /Users/plesser/tmp/nest_tammo/share/doc/nest/selftests/test_pass.sli

Apr 25 15:37:57 file [Error]: FileOpenError
    Could not open the following file for reading: 
    "/Users/plesser/tmp/nest_tammo/share/doc/nest/selftests/test_pass.sli".
Contributor

heplesser commented Apr 25, 2016

@tammoippen Thanks for the information. The correct option for testing is --test-pynest, but things fail, since the path to the tests scripts seems to be fixed (I originally installed in nest_tammo and then renamed that dir reloc:

[plesser@Hanss-MacBook-Pro] ~/tmp (0)$ NEST_SOURCE=SKIP exec_prefix=/Users/plesser/tmp/reloc/ /Users/plesser/tmp/reloc/share/nest/extras/do_tests.sh --test-pynest

Phase 1: Testing if SLI can execute scripts and report errors
-------------------------------------------------------------
  Running test 'selftests/test_pass.sli'... Failed: unexpected exit code 126
Running  mpirun -np 1 /Users/plesser/tmp/reloc/bin/nest /Users/plesser/tmp/nest_tammo/share/doc/nest/selftests/test_pass.sli

Apr 25 15:37:57 file [Error]: FileOpenError
    Could not open the following file for reading: 
    "/Users/plesser/tmp/nest_tammo/share/doc/nest/selftests/test_pass.sli".
@tammoippen

This comment has been minimized.

Show comment
Hide comment
@tammoippen

tammoippen Apr 25, 2016

Contributor

@heplesser Thank you. I will have a look.

Contributor

tammoippen commented Apr 25, 2016

@heplesser Thank you. I will have a look.

@tammoippen

This comment has been minimized.

Show comment
Hide comment
@tammoippen

tammoippen Apr 25, 2016

Contributor

@heplesser Please try again. 27f5c20 should fix the testing and ea22760 should fix the installation of MyModule. In both cases nest_vars.sh should be sourced.

Contributor

tammoippen commented Apr 25, 2016

@heplesser Please try again. 27f5c20 should fix the testing and ea22760 should fix the installation of MyModule. In both cases nest_vars.sh should be sourced.

@heplesser

This comment has been minimized.

Show comment
Hide comment
@heplesser

heplesser Apr 25, 2016

Contributor

Testing after relocation works for SLI tests, but something strange happens for the PyNEST tests. When running make installcheck, 524 Python tests are run. When running after relocation as suggested above, only 241 are run. I have traced this back to nosetests being used when running via make installcheck, but the normal Python test harness while running do_tests.sh from the command line. I do not understand why this is the case.

Further, note that the following three lines in the installed do_tests.sh file still contain absolute locations:

TEST_BASEDIR=${NEST_DOC_DIR:-"/Users/plesser/tmp/nest_tammo/share/doc/nest"}

PYTHON_HARNESS="${NEST_DATA_DIR:-/Users/plesser/tmp/nest_tammo/share/nest}/extras/do_tests.py"

nosetests -v --with-xunit --xunit-file=${TEST_OUTDIR}/pynest_tests.xml ${NEST_PYTHON_PREFIX:-/Users/plesser/tmp/nest_tammo/lib/python2.7/site-packages}/nest/tests 2>&1 \
            | tee -a "${TEST_LOGFILE}" | grep -i --line-buffered "\.\.\. ok\|fail\|skip\|error" | sed 's/^/  /'
Contributor

heplesser commented Apr 25, 2016

Testing after relocation works for SLI tests, but something strange happens for the PyNEST tests. When running make installcheck, 524 Python tests are run. When running after relocation as suggested above, only 241 are run. I have traced this back to nosetests being used when running via make installcheck, but the normal Python test harness while running do_tests.sh from the command line. I do not understand why this is the case.

Further, note that the following three lines in the installed do_tests.sh file still contain absolute locations:

TEST_BASEDIR=${NEST_DOC_DIR:-"/Users/plesser/tmp/nest_tammo/share/doc/nest"}

PYTHON_HARNESS="${NEST_DATA_DIR:-/Users/plesser/tmp/nest_tammo/share/nest}/extras/do_tests.py"

nosetests -v --with-xunit --xunit-file=${TEST_OUTDIR}/pynest_tests.xml ${NEST_PYTHON_PREFIX:-/Users/plesser/tmp/nest_tammo/lib/python2.7/site-packages}/nest/tests 2>&1 \
            | tee -a "${TEST_LOGFILE}" | grep -i --line-buffered "\.\.\. ok\|fail\|skip\|error" | sed 's/^/  /'
@heplesser

This comment has been minimized.

Show comment
Hide comment
@heplesser

heplesser Apr 25, 2016

Contributor

Building MyModule after relocation works.

Contributor

heplesser commented Apr 25, 2016

Building MyModule after relocation works.

@heplesser

This comment has been minimized.

Show comment
Hide comment
@heplesser

heplesser Apr 25, 2016

Contributor

The first line of sli.el also contains the original install path explicitly.

Contributor

heplesser commented Apr 25, 2016

The first line of sli.el also contains the original install path explicitly.

@heplesser

This comment has been minimized.

Show comment
Hide comment
@heplesser

heplesser Apr 26, 2016

Contributor

It turned out that the problem with missing Python tests when running do_tests.sh form the command line instead of via make installcheck were due to the way we check for xunit support in nosetests. If do_tests.sh was run from a directory containing any failing Python tests, it would think that nosetests did not support xunit. This is fixed by #321.

Contributor

heplesser commented Apr 26, 2016

It turned out that the problem with missing Python tests when running do_tests.sh form the command line instead of via make installcheck were due to the way we check for xunit support in nosetests. If do_tests.sh was run from a directory containing any failing Python tests, it would think that nosetests did not support xunit. This is fixed by #321.

@tammoippen

This comment has been minimized.

Show comment
Hide comment
@tammoippen

tammoippen Apr 26, 2016

Contributor

@heplesser

Further, note that the following three lines in the installed do_tests.sh file still contain absolute locations:

TEST_BASEDIR=${NEST_DOC_DIR:-"/Users/plesser/tmp/nest_tammo/share/doc/nest"}
PYTHON_HARNESS="${NEST_DATA_DIR:-/Users/plesser/tmp/nest_tammo/share/nest}/extras/do_tests.py"
nosetests -v --with-xunit --xunit-file=${TEST_OUTDIR}/pynest_tests.xml ${NEST_PYTHON_PREFIX:-/Users/plesser/tmp/nest_tammo/lib/python2.7/site-packages}/nest/tests 2>&1 \
            | tee -a "${TEST_LOGFILE}" | grep -i --line-buffered "\.\.\. ok\|fail\|skip\|error" | sed 's/^/  /'

I used parameter substitution here: ${NEST_DOC_DIR:-"/Users/plesser/tmp/nest_tammo/share/doc/nest"} . So if NEST_DOC_DIR is set then use its value, otherwise use the previous install directory.

Contributor

tammoippen commented Apr 26, 2016

@heplesser

Further, note that the following three lines in the installed do_tests.sh file still contain absolute locations:

TEST_BASEDIR=${NEST_DOC_DIR:-"/Users/plesser/tmp/nest_tammo/share/doc/nest"}
PYTHON_HARNESS="${NEST_DATA_DIR:-/Users/plesser/tmp/nest_tammo/share/nest}/extras/do_tests.py"
nosetests -v --with-xunit --xunit-file=${TEST_OUTDIR}/pynest_tests.xml ${NEST_PYTHON_PREFIX:-/Users/plesser/tmp/nest_tammo/lib/python2.7/site-packages}/nest/tests 2>&1 \
            | tee -a "${TEST_LOGFILE}" | grep -i --line-buffered "\.\.\. ok\|fail\|skip\|error" | sed 's/^/  /'

I used parameter substitution here: ${NEST_DOC_DIR:-"/Users/plesser/tmp/nest_tammo/share/doc/nest"} . So if NEST_DOC_DIR is set then use its value, otherwise use the previous install directory.

@heplesser

This comment has been minimized.

Show comment
Hide comment
@heplesser

heplesser Apr 26, 2016

Contributor

@tammoippen Thanks for the shell-lesson :)!

Things work fine now, so 👍 from me.

Contributor

heplesser commented Apr 26, 2016

@tammoippen Thanks for the shell-lesson :)!

Things work fine now, so 👍 from me.

@tammoippen

This comment has been minimized.

Show comment
Hide comment
@tammoippen

tammoippen Apr 26, 2016

Contributor

Yes, sli.el.in has quite a lot references to @CMAKE_INSTALL_FULL_DOCDIR@, @CMAKE_INSTALL_FULL_DATADIR@ and also to the sources (line 93) @PROJECT_SOURCE_DIR@. I think the Emacs support itself might need some polish: the access to the sources is necessary to get SVN information. The file psvn.el is an interface for the revision control tool subversion - we should "upgrade" to use the Git supplied git.el extension. In 37b7077 I refactored these @- replacement into lisp variables, but I am not a regular emacs user / lisp programmer. Maybe someone with more experience can check this and do the polish. I will open an issue for for this.

Edit: see #322

Contributor

tammoippen commented Apr 26, 2016

Yes, sli.el.in has quite a lot references to @CMAKE_INSTALL_FULL_DOCDIR@, @CMAKE_INSTALL_FULL_DATADIR@ and also to the sources (line 93) @PROJECT_SOURCE_DIR@. I think the Emacs support itself might need some polish: the access to the sources is necessary to get SVN information. The file psvn.el is an interface for the revision control tool subversion - we should "upgrade" to use the Git supplied git.el extension. In 37b7077 I refactored these @- replacement into lisp variables, but I am not a regular emacs user / lisp programmer. Maybe someone with more experience can check this and do the polish. I will open an issue for for this.

Edit: see #322

@sanjayankur31

This comment has been minimized.

Show comment
Hide comment
@sanjayankur31

sanjayankur31 Apr 26, 2016

Contributor

It builds well here. I placed it in /opt/ and everything seems to have gone into the right place. I built it with MPI and had to add the required snippet in my ~/.nestrc before I ran make installcheck to run the tests. (Should nest have created a nestrc file for me when it ran the first time - I had one, but it was empty?)

When I was looking at /opt/share/doc/nest/examples/nestrc.sli, I noticed that the file still starts with:

% NEST has created this file for you from the template in
% <prefix>/share/doc/nest/examples/nestrc.sli.

Should <prefix> also be replaced with CMAKE_INSTALL_PREFIX here? Another configure_file directive in the CMakeLists.txt file? I'll let you know if anything else comes up. Thanks!

Contributor

sanjayankur31 commented Apr 26, 2016

It builds well here. I placed it in /opt/ and everything seems to have gone into the right place. I built it with MPI and had to add the required snippet in my ~/.nestrc before I ran make installcheck to run the tests. (Should nest have created a nestrc file for me when it ran the first time - I had one, but it was empty?)

When I was looking at /opt/share/doc/nest/examples/nestrc.sli, I noticed that the file still starts with:

% NEST has created this file for you from the template in
% <prefix>/share/doc/nest/examples/nestrc.sli.

Should <prefix> also be replaced with CMAKE_INSTALL_PREFIX here? Another configure_file directive in the CMakeLists.txt file? I'll let you know if anything else comes up. Thanks!

tammoippen added some commits Apr 20, 2016

Use of GNUInstallDirs for standard pathnames
Replace pathname variables in cmake by standard ones.
fix MyModule for use with GNUInstallDirs
Rename sli and nest library
Previously libnest_lib.so and libsli_lib.so were installed.
Now they are called libnest.so and libsli.so
Improve MyModule and nest-config
Also use data and doc install dir from NEST too
Rename NEST_INSTALL ...
use GNUInstallDirs after setting datadir
make do_test.sh relocatable
only when `NEST_DATA_DIR` and `NEST_DOC_DIR` and `NEST_PYTHON_PREFIX` are defined, like in `nest_vars.sh`
Make nest-config relocatable
when NEST_INSTALL_DIR is set, e.g. via nest_vars.sh, nest-config will return the correct values
some documentation and formatting
also `exec_prefix` is not used during do_tests.sh
@tammoippen

This comment has been minimized.

Show comment
Hide comment
@tammoippen

tammoippen May 9, 2016

Contributor

@sanjayankur31

(Should nest have created a nestrc file for me when it ran the first time - I had one, but it was empty?)

Yes, NEST should have created a .nestrc file with some default content (on first execution) and it is required to edit the file for the mpirun command, if build with mpi. What do you mean with 'it was empty'? It should be a copy of nestrc.sli. Have you sourced /opt/bin/nest_vars.sh which sets some necessary environment variables for the relocated NEST.

Should <prefix> also be replaced with CMAKE_INSTALL_PREFIX here?

I do not think, that this is necessary: the same .nestrc file can be used for multiple installations of nest and the <prefix> can change between them.

@sanjayankur31 If you are satisfied with this PR, please consider giving me a 👍 .

Contributor

tammoippen commented May 9, 2016

@sanjayankur31

(Should nest have created a nestrc file for me when it ran the first time - I had one, but it was empty?)

Yes, NEST should have created a .nestrc file with some default content (on first execution) and it is required to edit the file for the mpirun command, if build with mpi. What do you mean with 'it was empty'? It should be a copy of nestrc.sli. Have you sourced /opt/bin/nest_vars.sh which sets some necessary environment variables for the relocated NEST.

Should <prefix> also be replaced with CMAKE_INSTALL_PREFIX here?

I do not think, that this is necessary: the same .nestrc file can be used for multiple installations of nest and the <prefix> can change between them.

@sanjayankur31 If you are satisfied with this PR, please consider giving me a 👍 .

@sanjayankur31

This comment has been minimized.

Show comment
Hide comment
@sanjayankur31

sanjayankur31 May 9, 2016

Contributor

Yes, NEST should have created a .nestrc file with some default content (on first execution) and it is required to edit the file for the mpirun command, if build with mpi. What do you mean with 'it was empty'? It should be a copy of nestrc.sli. Have you sourced /opt/bin/nest_vars.sh which sets some necessary environment variables for the relocated NEST.

Well, it was an empty file with nothing in it. Maybe I didn't source nest_vars. I'll look at it again later The rest looks good to me. 👍

Contributor

sanjayankur31 commented May 9, 2016

Yes, NEST should have created a .nestrc file with some default content (on first execution) and it is required to edit the file for the mpirun command, if build with mpi. What do you mean with 'it was empty'? It should be a copy of nestrc.sli. Have you sourced /opt/bin/nest_vars.sh which sets some necessary environment variables for the relocated NEST.

Well, it was an empty file with nothing in it. Maybe I didn't source nest_vars. I'll look at it again later The rest looks good to me. 👍

@heplesser

This comment has been minimized.

Show comment
Hide comment
@heplesser

heplesser May 11, 2016

Contributor

Two thumbs up, so I will merge.

Contributor

heplesser commented May 11, 2016

Two thumbs up, so I will merge.

@heplesser heplesser merged commit a1d443b into nest:master May 11, 2016

1 check passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details

@tammoippen tammoippen deleted the tammoippen:independant_nest branch May 18, 2016

@steffengraber steffengraber referenced this pull request in steffengraber/nest-simulator May 20, 2016

Merged

Help build #1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment