Adaptations for the release validation #134

dberzano · 2018-05-10T10:25:37Z

This is to track the work we are doing for making AliDPG scripts compatible with the release validation.

Runs only during the validation, not on Grid

dberzano · 2018-05-19T12:43:50Z

AOD/main_AODtrain.C

@@ -119,7 +119,7 @@ void main_AODtrain(Int_t merge=0)

  PrintSettings();

-  if (merge || doCDBconnect) {
+  if ((merge || doCDBconnect) && !gSystem->Getenv("OCDB_PATH")) {


Since OCDB_PATH is not set for Grid jobs, nothing changes on the Grid.

Hi Dario,
on a hopefully short time scale, the macro AODtrainRawAndMC.C will replace the two macros AODtrain.C and AODtrainsim.C, in order to have the same code to produce AODs for raw data and MC.
So, I was thinking that to avoid problems in the future, it could be worth to apply this modification also to main_AODtrainRawAndMC.C.

Thanks!

Ciao, Francesco

Let me update the PR. I have also noticed that AOD macros don't load the custom OCDB.

dberzano · 2018-05-19T12:44:17Z

DataProc/CPass0/main_recCPass0.C

@@ -109,6 +109,9 @@ void main_recCPass0(const char *filename="raw.root",Int_t nevents=-1, const char
  else {
    // setup ocdb by custom (if any) or default settings
    if (gSystem->AccessPathName("localOCDBaccessConfig.C", kFileExists)==0) {        
+      TString envRunNumBuf = gSystem->Getenv("ALIEN_JDL_LPMRUNNUMBER");


Executed only if localOCDBaccessConfig.C is there, which is not the case on the Grid.

dberzano · 2018-05-19T12:45:05Z

DataProc/CPass0/mergeMakeOCDB.byComponent.perStage.sh

@@ -262,6 +262,35 @@ main()
    echo "***********************" 2>&1 | tee -a ocdb.log
    echo aliroot -b -q "makeOCDB.C($run, \"$ocdb\", \"$defaultOCDB\", $detectorBitsQualityFlag)" 2>&1 | tee -a ocdb.log
    aliroot -b -q "makeOCDB.C($run, \"$ocdb\", \"$defaultOCDB\", $detectorBitsQualityFlag)" 2>&1 | tee -a ocdb.log
+    if [[ $ALIEN_JDL_CREATEOCDBARCHIVE ]]; then


Environment variable ALIEN_JDL_CREATEOCDBARCHIVE (which would be CreateOCDBArchive on a JDL) is not defined on the Grid, so this part should not affect Grid jobs either.

dberzano · 2018-05-19T12:48:35Z

DataProc/CPass0/mergeMakeOCDB.byComponent.perStage.sh

@@ -472,7 +501,11 @@ copyScripts()

 extractFileNamesFromXMLCollection()
 {
-    grep turl|sed 's|^.*turl\s*=\s*"\s*\([a-zA-Z]*://.*\.root\).*$|\1|g'
+    if [[ ! $ROOTSYS || ! $ALIDPG_ROOT ]]; then


This part does affect Grid jobs. As a matter of fact, we've replaced a, er, "homemade" XML parser with a proper one, see the new readAliEnCollection.sh script. The reason for this replacement is that it misbehaved for generic URLs.

We have tested it successfully: the parser is used if ROOT and AliDPG are both available in the job's environment (which should always be the case), otherwise we fall back to the old method.

dberzano · 2018-05-19T12:48:54Z

DataProc/CPass1/main_recCPass1.C

@@ -86,6 +86,9 @@ void main_recCPass1(const char *filename="raw.root",Int_t nevents=-1, const char
  else {
    // setup ocdb by custom (if any) or default settings
    if (gSystem->AccessPathName("localOCDBaccessConfig.C", kFileExists)==0) {
+      TString envRunNumBuf = gSystem->Getenv("ALIEN_JDL_LPMRUNNUMBER");


Same as for CPass0: does not affect Grid jobs.

dberzano · 2018-05-19T12:49:09Z

DataProc/CPass1/mergeMakeOCDB.byComponent.perStage.sh

@@ -261,6 +261,35 @@ main()
    echo "***********************" 2>&1 | tee -a ocdb.log
    echo aliroot -b -q "makeOCDB.C($run, \"$ocdb\", \"$defaultOCDB\", $detectorBitsQualityFlag)" 2>&1 | tee -a ocdb.log
    aliroot -b -q "makeOCDB.C($run, \"$ocdb\", \"$defaultOCDB\", $detectorBitsQualityFlag)" 2>&1 | tee -a ocdb.log
+    if [[ $ALIEN_JDL_CREATEOCDBARCHIVE ]]; then


Same as for CPass0: does not affect Grid jobs.

dberzano · 2018-05-19T12:49:25Z

DataProc/CPass1/mergeMakeOCDB.byComponent.perStage.sh

@@ -470,7 +499,11 @@ copyScripts()

 extractFileNamesFromXMLCollection()
 {
-    grep turl|sed 's|^.*turl\s*=\s*"\s*\([a-zA-Z]*://.*\.root\).*$|\1|g'
+    if [[ ! $ROOTSYS || ! $ALIDPG_ROOT ]]; then


Same as for CPass0: this part does affect Grid jobs but it should work.

dberzano · 2018-05-19T12:50:10Z

DataProc/Common/readAliEnCollection.C

@@ -0,0 +1,18 @@
+void readAliEnCollection(const char *in, const char *out) {


This is the new XML parser. It is based on ROOT. We always build our ROOT with XML support, so this will work everywhere, including the Grid.

dberzano · 2018-05-19T12:50:25Z

DataProc/Common/readAliEnCollection.sh

@@ -0,0 +1,7 @@
+#!/bin/bash


This script merely invokes the ROOT-based XML parser.

dberzano · 2018-05-19T12:52:02Z

DataProc/MergeOutputs/merge.C

@@ -1,11 +1,14 @@
-void merge(const char *filesToMerge="FilterEvents_Trees.root")
+void merge(const char *filesToMerge="FilterEvents_Trees.root", const char *inputCollection="wn.xml")


This change introduces an argument to the merge function, allowing to specify a different XML collection. Since Grid JDLs don't use the second parameter, they will always fall back to the default one, wn.xml. This part does affect Grid jobs but it was tested to work.

dberzano · 2018-05-19T12:52:27Z

DataProc/MergeOutputs/merge.C

-  if (!gGrid || !gGrid->IsConnected()) {
-    ::Error("QAtrain", "No grid connection");
-    return;
+  TString noGridConnect = gSystem->Getenv("ALIEN_JDL_NOGRIDCONNECT");


That envvar is not defined for Grid jobs, so this section does not affect them.

dberzano · 2018-05-19T12:53:12Z

DataProc/MergeOutputs/mergeByComponent.C

@@ -181,6 +179,12 @@ void CopyCPass(const char* alienFileList, const char* outputFileList, Int_t time
    TString src("");
    src.ReadLine(inputFile);
    if (src.IsNull()) continue;
+    if (!counter && !gGrid) {


Conditionally connect to the Grid. Grid input files are always alien://... so it should work there.

dberzano · 2018-05-19T12:54:06Z

DataProc/PPass/main_rec.C

@@ -8,6 +8,14 @@ void main_rec(const char *filename="raw.root", const char* options="")


  AliReconstruction rec;
+


I have noticed that current reconstruction macros for PPass do not allow to reconstruct only part of the chunk (unlike CPass0 and 1). This variable (not used on the Grid) adds this capability consistently for every reconstruction job.

dberzano · 2018-05-19T12:54:38Z

DataProc/PPass/main_rec.C

    // setup ocdb by custom (if any) or default settings
    if (gSystem->AccessPathName("OCDBconfig.C", kFileExists)==0) {
      gROOT->ProcessLine("OCDBconfig.C");
    }
+    else if (gSystem->AccessPathName("localOCDBaccessConfig.C", kFileExists)==0) {


If the override OCDB macro is there, we load it. This does not affect Grid jobs.

dberzano · 2018-05-19T12:55:06Z

DataProc/TPCSPCalibration/createOCDBTPCSPDistCalib.sh

@@ -89,6 +89,36 @@ if [[ ! -f "$locMacro" ]] ; then cp $macroName ./ ; fi

 time aliroot -b -q "${locMacro}(\"$inputFileList\", $startRun, $endRun, \"$targetOCDBDir\",$corr ,$dist)" >> ocdb.log

+if [[ $ALIEN_JDL_CREATEOCDBARCHIVE ]]; then


Same as above. Does not affect Grid jobs.

dberzano · 2018-05-19T12:55:23Z

DataProc/TPCSPCalibration/procResidData.C

@@ -81,6 +88,15 @@ void procResidData(int mode,                       // processing mode
  //
 }

+//________________________________________________________________________
+void loadLocalOCDBIfExists(AliTPCDcalibRes *clb) {
+  if (gSystem->AccessPathName("localOCDBaccessConfig.C", kFileExists) == 0) {


This part, again, does not affect Grid jobs.

dberzano · 2018-05-19T12:56:00Z

DataProc/TPCSPCalibration/procTBinMap.sh

@@ -69,7 +69,7 @@ extractEnvVars()
 [[ $# -lt 3 ]] &&  Usage && exit 1
 echo "Arguments: 1 = $1, 2 = $2, 3 = $3"

-source $ALICE_PHYSICS/PWGPP/scripts/alilog4bash.sh  
+[[ -e $ALICE_PHYSICS/PWGPP/scripts/alilog4bash.sh ]] && source "$ALICE_PHYSICS"/PWGPP/scripts/alilog4bash.sh || source "$ALICE_ROOT"/libexec/alilog4bash.sh


This fixes a problem never noticed on the Grid: that file was moved and TPC SP jobs haven't had any log output since a while.

dberzano · 2018-05-19T12:56:40Z

QA/main_QAtrain_duo.C

@@ -146,7 +146,7 @@ void main_QAtrain_duo(const char *suffix="", Int_t run = 0,
  PrintSettings();

  TString cdbString(cdb);
-  if (cdbString.Contains("raw://"))
+  if (cdbString.Contains("raw://") && !gSystem->Getenv("OCDB_PATH"))


OCDB_PATH not set on the Grid, does not affect Grid jobs.

dberzano · 2018-05-19T12:57:27Z

bin/train_merge.sh

@@ -51,7 +51,7 @@ fi

 if [ -f "QAtrain_duo.C" ]; then
    echo "QAtrain_duo macro found"
-    if grep -q -E -e "/cpass1|cpass1/" *.xml; then
+    if grep -q -E -e "/cpass1|cpass1/" *.xml || grep -q -E -e "/cpass1|cpass1/" "$1"; then


Does not affect Grid jobs, where input collection is always a XML file. It makes it work for text files too, in the release validation case (used if we don't put our files in zip archives).

dberzano · 2018-05-19T12:59:13Z

Hello @chiarazampolli, I would merge it too. I have added comments to the code on this PR to better clarify what parts affect/do not affect Grid jobs. Parts that "affect" Grid jobs should be harmless, or fix unnoticed misbehaviours. Those comments will be helpful in tracking potential mishaps during the next reconstruction. Nothing bad should happen, but you can never tell 😄

fprino · 2018-05-19T13:22:20Z

Hi,

thanks for the work. I agree to proceed merging the pull request.
I added a comment about the AOD macro, but that modification can be done also at a later stage.

Cheers, Francesco

dberzano · 2018-05-19T14:15:37Z

@fprino I have done the requested modification in AODtrainRawAndMC, so we're more future-proof. I think you can merge if you agree, and backport to the v5-09-XX branch too.

fprino · 2018-05-21T07:23:10Z

Hi Dario, Matteo,

thanks for the work and for the additional modification for the AOD filtering.
I have merged the PR.

Ciao, Francesco

chiarazampolli · 2018-05-22T11:50:55Z

Ciao,

(sorry for the delay, I was in a working session with a colleague till now).

I want also to thank a lot Dario and Matteo for the huge work!!

Chiara

dberzano force-pushed the fix-things branch 17 times, most recently from 4414a82 to ac23d5b Compare May 17, 2018 10:48

dberzano force-pushed the fix-things branch from ac23d5b to d68a891 Compare May 19, 2018 00:40

Dario Berzano added 9 commits May 19, 2018 04:52

MergeByCompoent: AliEn connect only if needed

6f9a556

Create OCDB archive with access macro if requested

eb805c5

Properly parse XML collections

f0dc1a4

Merge: support optional XML and no alien: connect

9b1b5f5

Grab alilog4bash.sh from the correct place

2e126bb

See alisw/AliRoot#325

Add local OCDB access to TPCSPCalibration

c680a34

Add SetRun() before specific storages in CPass0/1

045c23f

Runs only during the validation, not on Grid

Make QAtrain_duo work without AliEn

755a84f

Enable TPCSPC to upload OCDB to dummy tarball

a75ed79

dberzano force-pushed the fix-things branch 3 times, most recently from 306a1a1 to 42576c9 Compare May 19, 2018 03:21

dberzano commented May 19, 2018

View reviewed changes

DataProc/Common/readAliEnCollection.sh

@@ -0,0 +1,7 @@

#!/bin/bash

Copy link

Contributor Author

dberzano May 19, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This script merely invokes the ROOT-based XML parser.

dberzano commented May 19, 2018

View reviewed changes

Dario Berzano added 2 commits May 19, 2018 16:07

Do not init Grid in AODtrainRawAndMC if unneeded

1487c0d

Load localOCDB* macro in AOD trains if possible

51a3a28

dberzano mentioned this pull request May 21, 2018

Final steps for AliDPG validation alisw/release-validation#51

Closed

10 tasks

fprino merged commit d36d2a2 into alisw:master May 21, 2018

dberzano mentioned this pull request Jun 5, 2018

New tag from v5-09-XX #140

Closed

dberzano mentioned this pull request Aug 22, 2018

Missing TGrid::Connect() on the Grid #153

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adaptations for the release validation #134

Adaptations for the release validation #134

dberzano commented May 10, 2018

dberzano May 19, 2018

fprino May 19, 2018

dberzano May 19, 2018

dberzano May 19, 2018

dberzano May 19, 2018

dberzano May 19, 2018

dberzano May 19, 2018

dberzano May 19, 2018

dberzano May 19, 2018

dberzano May 19, 2018

dberzano May 19, 2018

dberzano May 19, 2018

dberzano May 19, 2018

dberzano May 19, 2018

dberzano May 19, 2018

dberzano May 19, 2018

dberzano May 19, 2018

dberzano May 19, 2018

dberzano May 19, 2018

dberzano May 19, 2018

dberzano May 19, 2018

dberzano commented May 19, 2018

fprino commented May 19, 2018

dberzano commented May 19, 2018

fprino commented May 21, 2018

chiarazampolli commented May 22, 2018

		@@ -0,0 +1,18 @@
		void readAliEnCollection(const char in, const char out) {

		@@ -1,11 +1,14 @@
		void merge(const char *filesToMerge="FilterEvents_Trees.root")
		void merge(const char filesToMerge="FilterEvents_Trees.root", const char inputCollection="wn.xml")

		@@ -8,6 +8,14 @@ void main_rec(const char filename="raw.root", const char options="")


		AliReconstruction rec;

		@@ -89,6 +89,36 @@ if [[ ! -f "$locMacro" ]] ; then cp $macroName ./ ; fi

		time aliroot -b -q "${locMacro}(\"$inputFileList\", $startRun, $endRun, \"$targetOCDBDir\",$corr ,$dist)" >> ocdb.log

		if [[ $ALIEN_JDL_CREATEOCDBARCHIVE ]]; then

Adaptations for the release validation #134

Adaptations for the release validation #134

Conversation

dberzano commented May 10, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dberzano commented May 19, 2018

fprino commented May 19, 2018

dberzano commented May 19, 2018

fprino commented May 21, 2018

chiarazampolli commented May 22, 2018