Update yoda.sh by cholmcc · Pull Request #5221 · alisw/alidist

cholmcc · 2023-11-10T17:35:56Z

Fix for ROOT dependency loading in the generated module file.

cholmcc · 2023-11-11T21:36:35Z

Post-mortem on tests

build/O2/alidist-cs8: A QC test multinode_test times out - not related - False negative
build/O2/alidist-dataflow-cs8: Four QC tests times out - not related - False negative
build/alidist/O2Suite/o2/macOS: Fails because ALICE fork of Apache Arrow does not protect against PREALLOCATE already defined in system headers - False negative
build/alidist/O2Suite/o2/macOS-arm: Same as above - False negative

Yours,
Christian

jackal1-66 · 2023-11-13T21:26:17Z

@cholmcc I enabled auto-merge, however I don't have the privileges of merging the PR by myself. I think @ktf should be able to

cholmcc · 2023-11-14T08:43:52Z

Hi all,

@jackal1-66 - perhaps a re-run of the checks will work. The timeouts of QC tests have been fixed somehow. The problem on MacOSX persists - I've made a pull request against ALICE's arrow fork.

Yours,
Christian

jackal1-66 · 2023-11-14T14:09:25Z

Not able to re-run the tests @cholmcc. You could try pushing an empty commit maybe. Otherwise @ktf I believe is the one with privileges to restart them...

cholmcc · 2023-11-16T23:45:57Z

Hi,

Did this fix the issue we had with module load yoda and module load rivet? Thanks.

Yours,
Christian

maireiphc · 2023-11-17T03:48:49Z

Hello @cholmcc,

it seems so:
on lxplus, alienv enter AliGenerators (loading the latest compiled version available on CVMFS = AliGenerators/v20231116-1) and it does not complain about missing or unknown "\ROOT"
Similarly no issue pointed while enter in fresh environment, specific alienv enter Rivet or
alienv enter YODA

Thanks.

cholmcc · 2023-11-17T08:02:02Z

Hi Antonin,

it seems so: on lxplus, ....

Great news. That means we can move on to other things :-) i.e., getting Rivet integrated into O2Physics. Some follow up questions:

Once we have the Rivet integration in O2Physics (presumably), should we then make Rivet at build requirement of O2Physics? The above mentioned branch sets up the Rivet dependency as a recommendation, not a requirement.
As for deployment (of Rivet and Rivet O2Physics integration) on HyperLoop/Grid, how is that handled? Are there some special steps that need to be taken?

Thanks again.

Yours,
Christian

jackal1-66 · 2023-11-17T08:32:07Z

Ciao @cholmcc @maireiphc, i tested on lxplus and indeed everything is working well, however on the GRID it's another thing. I just checked using yesterday's version of AliGenerators and this is the output I'm getting:

WARNING: While bind mounting '/cvmfs:/cvmfs': destination is already in the mount point list
In file included from /cvmfs/alice.cern.ch/el7-x86_64/Packages/Rivet/rivet-3.1.8-3/include/Rivet/Jet.fhh:6,
                 from /cvmfs/alice.cern.ch/el7-x86_64/Packages/Rivet/rivet-3.1.8-3/include/Rivet/ParticleBase.hh:5,
                 from /cvmfs/alice.cern.ch/el7-x86_64/Packages/Rivet/rivet-3.1.8-3/include/Rivet/Particle.hh:6,
                 from /cvmfs/alice.cern.ch/el7-x86_64/Packages/Rivet/rivet-3.1.8-3/include/Rivet/Event.hh:6,
                 from /cvmfs/alice.cern.ch/el7-x86_64/Packages/Rivet/rivet-3.1.8-3/include/Rivet/Analysis.hh:7,
                 from ALICE_2022_I2593302.cc:3:
/cvmfs/alice.cern.ch/el7-x86_64/Packages/Rivet/rivet-3.1.8-3/include/Rivet/Tools/RivetFastJet.hh:7:10: fatal error: fastjet/JetDefinition.hh: No such file or directory
    7 | #include "fastjet/JetDefinition.hh"
      |          ^~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
make: *** [tmp.c6P6ZCkM.o] Error 1

Do you think that your latest upgrade of FastJet could fix this @maireiphc? @cholmcc do you have any fast fix in case?

cholmcc · 2023-11-17T16:00:39Z

Hi Marco,

Yes, I believe you have to get a newer version of FastJet on Grid. The Rivet bootstrap script has FastJet version 3.4.0. However, I think 3.2.x, as used by alidist should do fine too. Don't know what version is on Grid though. The header is present in the ALICE github repo.

Can you perhaps say what you tried to do? Perhaps the include path isn't set correctly. Thanks.

Yours,
Christian

jackal1-66 · 2023-11-20T09:09:13Z

Ciao Christian,
the only thing I try doing is loading AliGenerators and running Rivet on a EPOS generated hepmc through a FIFO.

Cheers,

Marco

cholmcc · 2023-11-20T13:47:47Z

Hi Marco,

Ciao Christian, the only thing I try doing is loading AliGenerators and running Rivet on a EPOS generated hepmc through a FIFO.

In case I want to try my self, it would be good if you could be a little more specific. Like

where? (CVMFS or Grid)
how? (command line)

It looks like you're trying to compile something, so the question is what are you compiling and how?

Thanks.

Yours,
Christian

jackal1-66 · 2023-11-20T14:07:37Z

Ciao Christian,
I found where the problem is exactly, and it is visible also on LXPLUS:

[AliGenerators] ALICE_2019_I1735351 > rivet-build RivetALICE_2019_I1735351.so ALICE_2019_I1735351.cc
g++ -o "RivetALICE_2019_I1735351.so" -shared -fPIC  -I/cvmfs/alice.cern.ch/el7-x86_64/Packages/Rivet/rivet-3.1.8-4/include -I/cvmfs/alice.cern.ch/el7-x86_64/Packages/HepMC3/3.2.5-100/include  -DENABLE_HEPMC_3=true -I/usr/include -I/usr/include -I/include -I/include   -pedantic -Wall -Wno-long-long -Wno-format -Werror=uninitialized -Werror=delete-non-virtual-dtor -fopenmp -fPIC -g -O2 -std=c++17   -Wl,--no-as-needed  -L/local/workspace/DailyBuilds/DailyAliGenerators/daily-tags.iKD2od7Gn0/slc7_x86-64/cgal/4.6.3-128/lib -L/cvmfs/alice.cern.ch/el7-x86_64/Packages/GMP/v6.2.1-43/lib -Wl,--no-as-needed -L/cvmfs/alice.cern.ch/el7-x86_64/Packages/Rivet/rivet-3.1.8-4/lib -L/cvmfs/alice.cern.ch/el7-x86_64/Packages/HepMC3/3.2.5-100/lib -L/lib -L/lib -lfastjettools -lfastjet -lCGAL -lgmp -L/local/workspace/DailyBuilds/DailyAliGenerators/daily-tags.iKD2od7Gn0/slc7_x86-64/cgal/4.6.3-128/lib -L/cvmfs/alice.cern.ch/el7-x86_64/Packages/GMP/v6.2.1-43/lib -Wl,-rpath,/cvmfs/alice.cern.ch/el7-x86_64/Packages/GMP/v6.2.1-43/lib -lm -lfastjetplugins -lsiscone_spherical -lsiscone -lfastjetcontribfragile -lfastjettools  -lRivet    ALICE_2019_I1735351.cc 
In file included from /cvmfs/alice.cern.ch/el7-x86_64/Packages/Rivet/rivet-3.1.8-4/include/Rivet/Jet.fhh:6,
                 from /cvmfs/alice.cern.ch/el7-x86_64/Packages/Rivet/rivet-3.1.8-4/include/Rivet/ParticleBase.hh:5,
                 from /cvmfs/alice.cern.ch/el7-x86_64/Packages/Rivet/rivet-3.1.8-4/include/Rivet/Particle.hh:6,
                 from /cvmfs/alice.cern.ch/el7-x86_64/Packages/Rivet/rivet-3.1.8-4/include/Rivet/Event.hh:6,
                 from /cvmfs/alice.cern.ch/el7-x86_64/Packages/Rivet/rivet-3.1.8-4/include/Rivet/Analysis.hh:7,
                 from ALICE_2019_I1735351.cc:2:
/cvmfs/alice.cern.ch/el7-x86_64/Packages/Rivet/rivet-3.1.8-4/include/Rivet/Tools/RivetFastJet.hh:7:10: fatal error: fastjet/JetDefinition.hh: No such file or directory
    7 | #include "fastjet/JetDefinition.hh"
      |          ^~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
make: *** [tmp.a1zMs3cy.o] Error 1

I believe this might be caused by multiple FastJet versions definitions, as echo $FASTJET shows version 3.4.1_1.0.52-alice2 installed, while the fastjet.sh script of alidist shows version 3.2.x.
Best,

Marco

maireiphc · 2023-11-20T14:57:20Z

Copy-paste of a just-posted comment in PR #5241 (already merged).
The tag written in fastjet.sh is not considered, it is overwritten by alidist/defaults-generators.
I asked for an update of this Fastjet version in defaults-generators two days ago, precisely towards
v3.4.1_1.052-alice2 #5241.

My 2 cents

This update of fastjet version does not seem to be transparent. @jackal1-66 have spot some issue among fastjet versions:

See the parallel PRs (both already merged):
- on the alidist/yoda.sh side: Update yoda.sh #5221 (line of dev: making rivet into 0²)
- on the alidist/defaults-generators: alidist/defaults-generators.sh - update fastjet version to match O² standards #5241 (line of dev : keep defaults-recipes on the same footing for 0² and AliGenerators)
The current fastjet status with merged PR is:
- alidist/defaults-O2 (since 2023-11-09) :
  - v3.4.1_1.052-alice2 (since 2023-11-09), -alice2 mods. See https://github.com/alisw/alidist/pull/5209/files for fixes relating to C++20 std
  - v3.4.1_1.052-alice1 (since 2023-05-25), first coming-in of v3.4.1 Bump FastJet to v3.4.1_1.052-alice1 #4998
- alidist/fastjet.sh (since 2018) : v3.2.1_1.024-alice3 (although other parts of the recipe are regularly updated, in sync with O² developments)
- alidist/defaults-generators (since 2023-11-18) : v3.4.0_1.045-alice1 to v3.4.1_1.052-alice2

- FastJet and YODA export `FASTJET_ROOT` and `YODA_ROOT`, respectively - Rivet build-requires `cgal` so that `CGAL_ROOT` _values_ in `rivet-config` and `rivet-build` gets properly substituted at build-time with references to environment variable `CGAL_ROOT` at run-time See also comment at alisw#5221

cholmcc · 2023-11-21T10:22:08Z

Hi,

I reformat the command line and log quoted above, just to make things clearer. Note the lines marked with <==== Hmmm - those are of concern :-/

[AliGenerators] ALICE_2019_I1735351 > rivet-build RivetALICE_2019_I1735351.so ALICE_2019_I1735351.cc
g++ \
-o "RivetALICE_2019_I1735351.so" \
-shared -fPIC  \
-I/cvmfs/alice.cern.ch/el7-x86_64/Packages/Rivet/rivet-3.1.8-4/include \
-I/cvmfs/alice.cern.ch/el7-x86_64/Packages/HepMC3/3.2.5-100/include  \
-DENABLE_HEPMC_3=true \
-I/usr/include \
-I/usr/include \
-I/include \                         <==== Hmmm
-I/include   \                       <==== Hmmm 
-pedantic \
-Wall -Wno-long-long \
-Wno-format \
-Werror=uninitialized \
-Werror=delete-non-virtual-dtor \
-fopenmp \
-fPIC \
-g \
-O2 \
-std=c++17   \
-Wl,--no-as-needed  \
-L/local/workspace/DailyBuilds/DailyAliGenerators/daily-tags.iKD2od7Gn0/slc7_x86-64/cgal/4.6.3-128/lib \
-L/cvmfs/alice.cern.ch/el7-x86_64/Packages/GMP/v6.2.1-43/lib \
-Wl,--no-as-needed \
-L/cvmfs/alice.cern.ch/el7-x86_64/Packages/Rivet/rivet-3.1.8-4/lib \
-L/cvmfs/alice.cern.ch/el7-x86_64/Packages/HepMC3/3.2.5-100/lib \
-L/lib \                                   <=== Hmmm
-L/lib \                                   <=== Hmmm
-lfastjettools \
-lfastjet \
-lCGAL \
-lgmp \
-L/local/workspace/DailyBuilds/DailyAliGenerators/daily-tags.iKD2od7Gn0/slc7_x86-64/cgal/4.6.3-128/lib \
-L/cvmfs/alice.cern.ch/el7-x86_64/Packages/GMP/v6.2.1-43/lib \
-Wl,-rpath,/cvmfs/alice.cern.ch/el7-x86_64/Packages/GMP/v6.2.1-43/lib \
-lm \
-lfastjetplugins \
-lsiscone_spherical \
-lsiscone \
-lfastjetcontribfragile \
-lfastjettools  \
-lRivet    \
ALICE_2019_I1735351.cc 
...
/cvmfs/alice.cern.ch/el7-x86_64/Packages/Rivet/rivet-3.1.8-4/include/Rivet/Tools/RivetFastJet.hh:7:10: \ 
   fatal error: fastjet/JetDefinition.hh: No such file or directory
    7 | #include "fastjet/JetDefinition.hh"
      |          ^~~~~~~~~~~~~~~~~~~~~~~~~~

On lxplus, after

alienv enter AliGenerators

I check which rivet-build and see a line like

ifastjet="$FASTJET_ROOT/include"
test -n "$ifastjet" && mycppflags="$mycppflags -I${ifastjet}"
...
lfastjet="-L$FASTJET_ROOT/lib -lfastjettools -lfastjet -lCGAL -lgmp -L/local/workspace/DailyBuilds/DailyAliGenerators/daily-tags.iKD2od7Gn0/slc7_x86-64/cgal/4.6.3-128/lib -L$GMP_ROOT/lib -Wl,-rpath,$GMP_ROOT/lib -lm -lfastjetplugins -lsiscone_spherical -lsiscone -lfastjetcontribfragile -lfastjettools"
test -n "$lfastjet" && myldflags="$myldflags ${lfastjet}"

However, the environment variable FASTJET_ROOT is not set

> test "x$FASTJET_ROOT" = x && echo "FASTJET_ROOT not set!" || echo "FASTJET_ROOT=$FASTJET_ROOT" 
FASTJET_ROOT not set

So if I do,

> FASTJET_ROOT=/cvmfs/alice.cern.ch/el7-x86_64/Packages/fastjet/v3.4.1_1.052-alice2-2 rivet-build ALICE_YYYY_I1234567.cc

it should work, right? Wrong, I get

... fatal error: YODA/AnalysisObject.h: No such file or directory
    5 | #include "YODA/AnalysisObject.h"
      |          ^~~~~~

because YODA_ROOT isn't set either

> test "x$YODA_ROOT" = x && echo "YODA_ROOT not set" || echo "YODA_ROOT=$YODA_ROOT" 
YODA_ROOT not set

So let's do

> FASTJET_ROOT=$FASTJET \
   YODA_ROOT=/cvmfs/alice.cern.ch/el7-x86_64/Packages/YODA/yoda-1.9.7-3 \
  rivet-build ALICE_YYYY_I1234567.cc

which almost works, except I get

ld: cannot find -lCGAL: No such file or directory

This ties into the weird flag

-L/local/workspace/DailyBuilds/DailyAliGenerators/daily-tags.iKD2od7Gn0/slc7_x86-64/cgal/4.6.3-128/lib

above. This was _supposed to have been -L${CGAL_ROOT}/lib but is instead hard-wired to the build machine path.

So I try

> FASTJET_ROOT=$FASTJET    YODA_ROOT=/cvmfs/alice.cern.ch/el7-x86_64/Packages/YODA/yoda-1.9.7-3   rivet-build ALICE_YYYY_I1234567.cc -L$CGAL_ROOT/lib

Yeah, that worked!

OK, so what is the problem and possibly the solution.

Let's just take a step back. When building the Rivet module, there's sed code that substitute build-time variables from dependencies and build-dependencies in rivet-build (and rivet-config). E.g., the value $FASTJET_ROOT is substituted with the reference \$FASTJET_ROOT. This is so that the module becomes "relocatable".

For FastJet and YODA, the problem is clearly that the modules do not define FASTJET_ROOT and YODA_ROOT. That should be a relatively easy fix to do.
For CGAL it's a bit different. CGAL_ROOT seems to be defined at build-time of the Rivet module, but for some reason the replacement isn't done. That's because the Rivet module does not build-require cgal.

I have made a pull-request

Yours,
Christian

* Update yoda.sh Fix for ROOT dependency loading in the generated module file. * Empty commit to trigger CI

Update yoda.sh

65f94ef

Fix for ROOT dependency loading in the generated module file.

cholmcc requested a review from a team as a code owner November 10, 2023 17:35

jackal1-66 previously approved these changes Nov 11, 2023

View reviewed changes

jackal1-66 enabled auto-merge (squash) November 13, 2023 21:24

Empty commit to trigger CI

43d4f5b

auto-merge was automatically disabled November 15, 2023 20:41
Head branch was pushed to by a user without write access

cholmcc dismissed jackal1-66’s stale review via 43d4f5b November 15, 2023 20:41

jackal1-66 enabled auto-merge (squash) November 16, 2023 08:43

jackal1-66 approved these changes Nov 16, 2023

View reviewed changes

jackal1-66 merged commit 0129e22 into alisw:master Nov 16, 2023

maireiphc mentioned this pull request Nov 20, 2023

alidist/defaults-generators.sh - update fastjet version to match O² standards #5241

Merged

cholmcc mentioned this pull request Nov 21, 2023

Updates for Rivet - Yoda, Fastjet, Cgal env var #5245

Closed

cholmcc deleted the patch-2 branch January 18, 2024 00:49

MichaelLettrich pushed a commit to MichaelLettrich/alidist that referenced this pull request Oct 3, 2024

Update yoda.sh (alisw#5221)

f94105a

* Update yoda.sh Fix for ROOT dependency loading in the generated module file. * Empty commit to trigger CI

Conversation

cholmcc commented Nov 10, 2023

Uh oh!

cholmcc commented Nov 11, 2023

Post-mortem on tests

Uh oh!

jackal1-66 commented Nov 13, 2023

Uh oh!

cholmcc commented Nov 14, 2023

Uh oh!

jackal1-66 commented Nov 14, 2023

Uh oh!

cholmcc commented Nov 16, 2023

Uh oh!

maireiphc commented Nov 17, 2023

Uh oh!

cholmcc commented Nov 17, 2023

Uh oh!

jackal1-66 commented Nov 17, 2023

Uh oh!

cholmcc commented Nov 17, 2023

Uh oh!

jackal1-66 commented Nov 20, 2023

Uh oh!

cholmcc commented Nov 20, 2023

Uh oh!

jackal1-66 commented Nov 20, 2023

Uh oh!

maireiphc commented Nov 20, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cholmcc commented Nov 21, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

maireiphc commented Nov 20, 2023 •

edited

Loading