Problems building whizard

[tmadlener]

I am unable to get whizard to build within spack. I get the following error:

==> Installing whizard
==> No binary for whizard found: installing from source
==> Warning: Suspicious requests to set or unset 'CC' found
==> Warning: 	    	env.set('CC', os.path.join(link_dir, compiler.link_paths['cc'])) at /spack/lib/spack/spack/build_environment.py:225
==> Warning: 	--->	env.set('CXX', self.compiler.cxx) at /spack/var/spack/repos/builtin/packages/whizard/package.py:78
==> Warning: Suspicious requests to set or unset 'CXX' found
==> Warning: 	    	env.set('CXX', os.path.join(link_dir, compiler.link_paths['cxx'])) at /spack/lib/spack/spack/build_environment.py:228
==> Warning: 	--->	env.set('FC', self.compiler.fc) at /spack/var/spack/repos/builtin/packages/whizard/package.py:79
==> Warning: Suspicious requests to set or unset 'F77' found
==> Warning: 	    	env.set('F77', os.path.join(link_dir, compiler.link_paths['f77'])) at /spack/lib/spack/spack/build_environment.py:231
==> Warning: 	--->	 at /spack/var/spack/repos/builtin/packages/whizard/package.py:81
==> Warning: Suspicious requests to set or unset 'FC' found
==> Warning: 	    	env.set('FC', os.path.join(link_dir, compiler.link_paths['fc'])) at /spack/lib/spack/spack/build_environment.py:234
==> Warning: 	--->	env.set('F77', self.compiler.fc) at /spack/var/spack/repos/builtin/packages/whizard/package.py:80
==> Fetching https://whizard.hepforge.org/downloads/?f=whizard-2.8.5.tar.gz
######################################################################## 100.0%
==> whizard: Executing phase: 'autoreconf'
==> whizard: Executing phase: 'configure'
==> whizard: Executing phase: 'build'
==> Error: ProcessError: Command exited with status 2:
    'make' '-j12'

4 errors found in build log:
     801    cp -f bigarray_module.ml bigarray_compat.ml
     802      OCAMLC    bigarray_compat.cmi
     803      OCAMLC    events.cmi
     804      OCAMLOPT  bigarray_compat.cmx
     805      OCAMLOPT  float.cmx
     806    File "bigarray_compat.ml", line 1:
  >> 807    Error: Could not find the .cmi file for interface bigarray_compat.mli.
     808      OCAMLC    diffmaps.cmi
  >> 809    make[2]: *** [bigarray_compat.cmx] Error 2
     810    make[2]: *** Waiting for unfinished jobs....
     811    make[2]: Leaving directory `/tmp/root/spack-stage/spack-stage-whizard-2.8.5-v7sytfvqvg5rlybfssrkdslaemrat2qr/spack-src/circe2/src'
  >> 812    make[1]: *** [all-recursive] Error 1
     813    make[1]: Leaving directory `/tmp/root/spack-stage/spack-stage-whizard-2.8.5-v7sytfvqvg5rlybfssrkdslaemrat2qr/spack-src/circe2'
  >> 814    make: *** [all-recursive] Error 1

See build log for details:
  /tmp/root/spack-stage/spack-stage-whizard-2.8.5-v7sytfvqvg5rlybfssrkdslaemrat2qr/spack-build-out.txt

I can consistently reproduce it using the following container running centos7: https://hub.docker.com/repository/docker/tmadlener/ilcsof-cvmfs-install-centos7

One way to reproduce it is to first build a suitable gcc (e.g. gcc@8.3.0 or gcc@9.3.0) using spack inside the container and then try to build whizard. However, this will take a few hours until all the necessary dependencies are built. A quicker way is to use the cvmfs installed packages as upstream. To do so, make sure to locally mount /cvmfs/sw.hsf.org and then start the container

docker run -it -v /cvmfs/sw.hsf.org:/cvmfs/sw.hsf.org tmadlener/ilcsof-cvmfs-install-centos7

Inside the container, setup spack and the correct environment

git clone https://github.com/spack/spack
git clone https://github.com/key4hep/k4-spack
. spack/share/spack/setup-env.sh
mkdir -p ~/.spack/linux
cp k4-spack/config/packages.yaml ~/.spack/linux

Add the upstream repository and load the pre-installed gcc@8.3.0 and add it to the known compilers for spack

cat <<EOT >> $HOME/.spack/linux/upstreams.yaml
upstreams:
  spack-instance-1:
      install_tree: /cvmfs/sw.hsf.org/spackages/
EOT

spack load gcc@8.3.0 && spack compiler find

Now check that all the dependencies are present (specify a pythia8 version that is already installed, to not have to build the new default version 8303).

spack spec -Il whizard ^pythia8@8244

This should show only whizard@2.8.5/v7sytfv as missing package and trying to install it leads to the above described error

spack install whizard ^pythia8@8244

For completeness spack debug report yields the following as of writing this issue.

* **Spack:** 0.15.4-1359-796561e
* **Python:** 2.7.5
* **Platform:** linux-centos7-skylake

[tmadlener]

Actually, it seems that using the cvmfs packages as upstream repository, leads to another issue, described in #72.

To really only get the error described here the above set of commands has to be slightly modified. Most importantly

Add the upstream repository and load the pre-installed gcc@8.3.0 and add it to the known compilers for spack

cat <<EOT >> $HOME/.spack/linux/upstreams.yaml
upstreams:
 spack-instance-1:
     install_tree: /cvmfs/sw.hsf.org/spackages/
EOT

spack load gcc@8.3.0 && spack compiler find

needs to be replaced by simply setting up the compiler and then going through building whizard and building all its dependencies:

. /cvmfs/sw.hsf.org/key4hep/setup.sh
spack compiler find
spack install whizard hepmc=2 ^pythia8@8244

Using hepmc=2 ensures that it will be built only once, but it can potentially also be omitted. This will build whizard and all its dependencies. Hence, it might take some time to finish.

[tmadlener]

This seems to be caused by a race condition. Running only one build process via spack install -j 1 whizard hepmc=2 ^pythia8@8244, allows me to successfully build whizard.

It is not entirely clear to me what causes the race condition. It looks like it is related to ocaml, but it could also be that one of the dependencies is not set correctly in the makefile. In any case, it looks like the gains in building whizard with more than one process in parallel are rather small and so maybe we can consider forcing it to be compiled with only one process?

[tmadlener]

From a few more brief tests, it seems that the race condition does not occur when using 4 parallel build jobs. Nevertheless, I have created spack/spack#19270 that forces whizard to be built sequentially, so that at least this issue can be closed.