Possible bug in ldmx root #1169

DhruvanshuSan · 2023-06-12T22:07:04Z

Describe the bug
When I try to run a root macro inside container, the root is unable to read the path of the input file properly. However, outside container it runs properly.

To Reproduce
Steps to reproduce the behavior:

source ldmx-env.sh
ldmx root -l -q extractResponse_v3.C'$\"/path/to/file.root\"$'
Error coming up (Using an example input file):

-bash-4.2$ ldmx root -l -q extractResponse_v3.C'$\"/sdf/group/ldmx/users/dhparmar/datafiles/LDMX_RAWData/hits/unpacked_reprocessed_ldmx_captan_out_07-04-2022_12-22-00__41_digi_linearize_hits.root\"$' INFO: /etc/singularity/ exists; cleanup by system administrator is not complete (see https://apptainer.org/docs/admin/latest/singularity_migration.html)

Processing extractResponse_v3.C(/sdf/group/ldmx/users/dhparmar/datafiles/LDMX_RAWData/hits/unpacked_reprocessed_ldmx_captan_out_07-04-2022_12-22-00__41_digi_linearize_hits.root)... input_line_17:2:21: error: expected expression extractResponse_v3(/sdf/group/ldmx/users/dhparmar/datafiles/LDMX_RAWData/hits/unpacked_reprocessed_ldmx_captan_out_07-04-2022_12-22-00__41_digi_linearize_hits.root) /* '.x' tries to invoke a func... ^ input_line_17:2:22: error: use of undeclared identifier 'sdf' extractResponse_v3(/sdf/group/ldmx/users/dhparmar/datafiles/LDMX_RAWData/hits/unpacked_reprocessed_ldmx_captan_out_07-04-2022_12-22-00__41_digi_linearize_hits.root) /* '.x' tries to invoke a func... ^ input_line_17:2:26: error: use of undeclared identifier 'group' extractResponse_v3(/sdf/group/ldmx/users/dhparmar/datafiles/LDMX_RAWData/hits/unpacked_reprocessed_ldmx_captan_out_07-04-2022_12-22-00__41_digi_linearize_hits.root) /* '.x' tries to invoke a func... ^ input_line_17:2:32: error: use of undeclared identifier 'ldmx' extractResponse_v3(/sdf/group/ldmx/users/dhparmar/datafiles/LDMX_RAWData/hits/unpacked_reprocessed_ldmx_captan_out_07-04-2022_12-22-00__41_digi_linearize_hits.root) /* '.x' tries to invoke a func... ^ input_line_17:2:37: error: use of undeclared identifier 'users' extractResponse_v3(/sdf/group/ldmx/users/dhparmar/datafiles/LDMX_RAWData/hits/unpacked_reprocessed_ldmx_captan_out_07-04-2022_12-22-00__41_digi_linearize_hits.root) /* '.x' tries to invoke a func... ^ input_line_17:2:43: error: use of undeclared identifier 'dhparmar' extractResponse_v3(/sdf/group/ldmx/users/dhparmar/datafiles/LDMX_RAWData/hits/unpacked_reprocessed_ldmx_captan_out_07-04-2022_12-22-00__41_digi_linearize_hits.root) /* '.x' tries to invoke a func... ^ input_line_17:2:52: error: use of undeclared identifier 'datafiles' extractResponse_v3(/sdf/group/ldmx/users/dhparmar/datafiles/LDMX_RAWData/hits/unpacked_reprocessed_ldmx_captan_out_07-04-2022_12-22-00__41_digi_linearize_hits.root) /* '.x' tries to invoke a func... ^ input_line_17:2:62: error: use of undeclared identifier 'LDMX_RAWData' extractResponse_v3(/sdf/group/ldmx/users/dhparmar/datafiles/LDMX_RAWData/hits/unpacked_reprocessed_ldmx_captan_out_07-04-2022_12-22-00__41_digi_linearize_hits.root) /* '.x' tries to invoke a func... ^ input_line_17:2:75: error: use of undeclared identifier 'hits' extractResponse_v3(/sdf/group/ldmx/users/dhparmar/datafiles/LDMX_RAWData/hits/unpacked_reprocessed_ldmx_captan_out_07-04-2022_12-22-00__41_digi_linearize_hits.root) /* '.x' tries to invoke a func... ^ input_line_17:2:80: error: use of undeclared identifier 'unpacked_reprocessed_ldmx_captan_out_07' extractResponse_v3(/sdf/group/ldmx/users/dhparmar/datafiles/LDMX_RAWData/hits/unpacked_reprocessed_ldmx_captan_out_07-04-2022_12-22-00__41_digi_linearize_hits.root) /* '.x' tries to invoke a func... ^ input_line_17:2:127: error: no matching literal operator for call to 'operator""_12' with argument of type 'unsigned long long' or 'const char *', and no matching literal operator template extractResponse_v3(/sdf/group/ldmx/users/dhparmar/datafiles/LDMX_RAWData/hits/unpacked_reprocessed_ldmx_captan_out_07-04-2022_12-22-00__41_digi_linearize_hits.root) /* '.x' tries to invoke a func... ^ input_line_17:2:136: error: no matching literal operator for call to 'operator""__41_digi_linearize_hits.root' with argument of type 'unsigned long long' or 'const char *', and no matching literal operator template extractResponse_v3(/sdf/group/ldmx/users/dhparmar/datafiles/LDMX_RAWData/hits/unpacked_reprocessed_ldmx_captan_out_07-04-2022_12-22-00__41_digi_linearize_hits.root) /* '.x' tries to invoke a func... ^

Desired behavior
The macro is supposed to take the file as input and process it.

Screenshots
If applicable, add screenshots to help explain your problem.

Environment:

Inside the container: ROOT 6.22/08
- Output of ldmx-container-config
Outside the container:
- OS of computer you are running on : Linux
- Version of Geant4, ROOT, ONNXRuntime, Xerces, and gcc

Additional context
We are giving the path in a different manner than what you would normally give because inside container it gives a bash error due to parentheses not recognized (it is a common unexpected token error). I discussed this issue with @tomeichlersmith on slack, he helped me to get rid of the bash error but we were not able to resolve the error of not able to read path properly.

The text was updated successfully, but these errors were encountered:

tomeichlersmith · 2023-06-12T22:17:32Z

To add some more context, as implied by the example input file, this is being run on SLAC SDF which uses singularity as its container runner.

I tried to replicate this error on my computer which uses docker and I was unable to see the same error - this is what caused me to elevate this to possible bug.
I tried a simple replacement

// file test.C
int test(std::string path) {
  std::cout << path << std::endl;
  return 0;
}

to check various things (the -l flag drops the root banner and the -q flag exits after running the macro)

[ldmx] tom@iroh:~/ldmx$ ldmx root -l -q test.C'("/path/to/file.root")'                                                                                                                                    
/etc/entry.sh: eval: line 64: syntax error near unexpected token `('
[ldmx] tom@iroh:~/ldmx$ ldmx root -l -q test.C'\("/path/to/file.root"\)'

Processing test.C(/path/to/file.root)...
input_line_9:2:7: error: expected expression
 test(/path/to/file.root) /* '.x' tries to invoke a function with the same name as the macro */
      ^
input_line_9:2:8: error: use of undeclared identifier 'path'
 test(/path/to/file.root) /* '.x' tries to invoke a function with the same name as the macro */
       ^
input_line_9:2:13: error: use of undeclared identifier 'to'
 test(/path/to/file.root) /* '.x' tries to invoke a function with the same name as the macro */
            ^
input_line_9:2:16: error: use of undeclared identifier 'file'
 test(/path/to/file.root) /* '.x' tries to invoke a function with the same name as the macro */
               ^
[ldmx] tom@iroh:~/ldmx$ ldmx root -l -q test.C'\(\"/path/to/file.root\"\)'

Processing test.C("/path/to/file.root")...
/path/to/file.root
(int) 0

Using the same test macro, I can confirm the bug on SDF:

[ldmx] eichl008@sdf-login04:~/ldmx$ ldmx root -l -q test.C'("/path/to/file.root")'                                                                                                                         
/etc/entry.sh: eval: line 64: syntax error near unexpected token `('
[ldmx] eichl008@sdf-login04:~/ldmx$ ldmx root -l -q test.C'\("/path/to/file.root"\)'

Processing test.C(/path/to/file.root)...
input_line_9:2:7: error: expected expression
 test(/path/to/file.root) /* '.x' tries to invoke a function with the same name as the macro */
      ^
input_line_9:2:8: error: use of undeclared identifier 'path'
 test(/path/to/file.root) /* '.x' tries to invoke a function with the same name as the macro */
       ^
input_line_9:2:13: error: use of undeclared identifier 'to'
 test(/path/to/file.root) /* '.x' tries to invoke a function with the same name as the macro */
            ^
input_line_9:2:16: error: use of undeclared identifier 'file'
 test(/path/to/file.root) /* '.x' tries to invoke a function with the same name as the macro */
               ^
[ldmx] eichl008@sdf-login04:~/ldmx$ ldmx root -l -q test.C'\(\"/path/to/file.root\"\)'

Processing test.C(/path/to/file.root)...
input_line_9:2:7: error: expected expression
 test(/path/to/file.root) /* '.x' tries to invoke a function with the same name as the macro */
      ^
input_line_9:2:8: error: use of undeclared identifier 'path'
 test(/path/to/file.root) /* '.x' tries to invoke a function with the same name as the macro */
       ^
input_line_9:2:13: error: use of undeclared identifier 'to'
 test(/path/to/file.root) /* '.x' tries to invoke a function with the same name as the macro */
            ^
input_line_9:2:16: error: use of undeclared identifier 'file'
 test(/path/to/file.root) /* '.x' tries to invoke a function with the same name as the macro */
               ^

tomeichlersmith · 2023-06-12T22:27:00Z

In the meantime...

While waiting for this bug to be patched, I have a quick alternative. Write a wrapper script to run root inside the container. This skips one of the levels of bash deduction so it makes things easier.

#!/bin/bash
#  This simple bash script is merely here to pass the root running into the container
#    ldmx ./this-file.sh /path/to/file.root
#  make sure to make this file executable with `chmod +x this-file.sh`
root "macro.C(\"$1\")"

tomeichlersmith · 2023-06-12T22:32:12Z

Unfortunately, this appears to be a singularity issue and not an issue with our environment script. the reason I say this is because the arguments to root get all the way to the singularity call without being simplified by our bash functions.

[ldmx] eichl008@sdf-login04:~/ldmx$ (set -x; ldmx root -q -l 'test.C\(\"/path/to/file.root\"\)')
+ ldmx root -q -l 'test.C\(\"/path/to/file.root\"\)'
+ [[ 4 == \0 ]]
+ case $1 in
+ __ldmx_run_here root -q -l 'test.C\(\"/path/to/file.root\"\)'
+ local _old_pwd=/sdf/home/e/eichl008
++ pwd -P
+ local _pwd=/sdf/group/ldmx/users/eichl008/.
+ __ldmx_is_mounted /sdf/group/ldmx/users/eichl008/.
++ cd /sdf/group/ldmx/users/eichl008/.
++ pwd -P
+ local full=/sdf/group/ldmx/users/eichl008
+ for _already_mounted in '${LDMX_CONTAINER_MOUNTS[@]}'
+ [[ /sdf/group/ldmx/users/eichl008/ = /sdf/group/ldmx/users/eichl008/* ]]
+ return 0
+ cd /sdf/group/ldmx/users/eichl008
+ __ldmx_run /sdf/group/ldmx/users/eichl008/. root -q -l 'test.C\(\"/path/to/file.root\"\)'
+ local csv_list=/tmp/.X11-unix
+ for dir_to_mount in '"${LDMX_CONTAINER_MOUNTS[@]}"'
+ csv_list=/sdf/group/ldmx/users/eichl008,/tmp/.X11-unix
+ local env_list
+ singularity run --no-home --cleanenv --env LDMX_BASE=/sdf/group/ldmx/users/eichl008,DISPLAY= --bind /sdf/group/ldmx/users/eichl008,/tmp/.X11-unix ldmx_dev_latest.sif /sdf/group/ldmx/users/eichl008/. root -q -l 'test.C\(\"/path/to/file.root\"\)'

Processing test.C(/path/to/file.root)...
input_line_9:2:7: error: expected expression
 test(/path/to/file.root) /* '.x' tries to invoke a function with the same name as the macro */
      ^
input_line_9:2:8: error: use of undeclared identifier 'path'
 test(/path/to/file.root) /* '.x' tries to invoke a function with the same name as the macro */
       ^
input_line_9:2:13: error: use of undeclared identifier 'to'
 test(/path/to/file.root) /* '.x' tries to invoke a function with the same name as the macro */
            ^
input_line_9:2:16: error: use of undeclared identifier 'file'
 test(/path/to/file.root) /* '.x' tries to invoke a function with the same name as the macro */
               ^
+ return 1
+ local rc=1
+ cd -
+ export OLDPWD=/sdf/home/e/eichl008
+ OLDPWD=/sdf/home/e/eichl008
+ return 1
+ return 1

which I can confirm by running singularity directly

[ldmx] eichl008@sdf-login04:~/ldmx$ singularity run --no-home --cleanenv --env LDMX_BASE=/sdf/group/ldmx/users/eichl008 --bind /sdf/group/ldmx/users/eichl008/ ldmx_dev_latest.sif /sdf/group/ldmx/users/eichl008/. root -q -l 'test.C\(\"/path/to/file.root\"\)'

Processing test.C(/path/to/file.root)...
input_line_9:2:7: error: expected expression
 test(/path/to/file.root) /* '.x' tries to invoke a function with the same name as the macro */
      ^
input_line_9:2:8: error: use of undeclared identifier 'path'
 test(/path/to/file.root) /* '.x' tries to invoke a function with the same name as the macro */
       ^
input_line_9:2:13: error: use of undeclared identifier 'to'
 test(/path/to/file.root) /* '.x' tries to invoke a function with the same name as the macro */
            ^
input_line_9:2:16: error: use of undeclared identifier 'file'
 test(/path/to/file.root) /* '.x' tries to invoke a function with the same name as the macro */
               ^

bryngemark · 2023-06-13T14:38:26Z

hi @DhruvanshuSan , i don’t know what @tomeichlersmith and you discussed outside of this thread, but since I was a bit surprised by your quotation mark placement, I just tried your command from the original issue (using some random macro I had lying around) with my local install of root:

$ root -l -q myTestMacro.C'\(\"/path/to/file.root\"\)'
Warning in <TApplication::GetOptions>: macro myTestMacro.C\ not found

and without the \ before the ( and "

$ root -l -q 'myTestMacro.C('"/path/to/file.root"')'

Processing myTestMacro.C(/path/to/file.root)...
input_line_21:2:14: error: expected expression
 myTestMacro(/path/to/file.root) /* invoking function corresponding to '.x' */
             ^
input_line_21:2:15: error: use of undeclared identifier 'path'
 myTestMacro(/path/to/file.root) /* invoking function corresponding to '.x' */
              ^
input_line_21:2:20: error: use of undeclared identifier 'to'
 myTestMacro(/path/to/file.root) /* invoking function corresponding to '.x' */
                   ^
input_line_21:2:23: error: use of undeclared identifier 'file'
 myTestMacro(/path/to/file.root) /* invoking function corresponding to '.x' */
                      ^

while if I use slashes and the placement I typically use (note: ` surrounds the whole file name and function arguments) , I see what I would expect:

$ root -l -q 'myTestMacro.C('\"/path/to/file.root\"')'
Processing myTestMacro.C("/path/to/file.root")...
Error in <TFile::TFile>: file /path/to/file.root does not exist
...

tomeichlersmith · 2023-06-13T14:52:49Z

Thank you for the additional escape sequences to try @bryngemark - I did not think of those!

DhruvanshuSan · 2023-06-13T16:23:21Z

Hi @bryngemark, thanks for the additional input. To be fair, yesterday @tomeichlersmith and I were just playing around with different escape sequences to get rid of the bash unexpected token error. I tried the suggestions which you gave and I am also able to run macro outside container with your last suggestion (approach 2) : root -l -q 'myTestMacro.C('\"/path/to/file.root\"')'

Also, I was able to reproduce the same error (macro not found error) when I typed it this way (approach 1): root -l -q myTestMacro.C'$\"/path/to/file.root\"$'

So in a sense, it is something weird. When we go for approach 1 as we were trying yesterday, it accepts the command giving the error as described in the main bug report inside the container but gives the macro not found error outside container. Approach 2 works normally fine and as expected outside the container, but gives the same bash unexpected token which we were facing before.

So @tomeichlersmith, I am not sure if root version also has something to do with it in addition to the singularity issue. My personal pc has root version 6.24/02 and docker and hence it runs smoothly on my pc. And I recall that you mentioned the container has root version 6.22/08 and singularity container configuration.

@bryngemark, to give more context, I was trying to run SLURM job scripts on sdf machine to run the mip response macro on multiple files and it was unable to find root in SLURM. Hence the solution was to run it inside the container and that is where all this trouble started.

bryngemark · 2023-06-20T13:44:05Z

hi both, so it looks like we need different sets of escape sequences for running locally, with docker, and with singularity, which in ROOT should translate to .x myTestMacro.C("/path/to/file.root") ?

locally this works:
root -l -q 'myTestMacro.C('\"/path/to/file.root\"')'

with docker I have to do

$ ldmx root -l -q 'myTestMacro.C\(\"/path/to/file.root\"\)'

Processing myTestMacro.C("/path/to/file.root")...

and with singularity we need... @tomeichlersmith's patch wrapper? I have run root macros with singularity before, on SLAC LSBatch, for instance the gain extraction macro used earlier on in the processing chain that @DhruvanshuSan is working on. @DhruvanshuSan are you using that same set of bash scripts (perhaps modified) that I have shared in the repo?

DhruvanshuSan · 2023-06-21T01:25:32Z

Hi @bryngemark , so I do am using slightly modified versions of the root macros which you shared in the repo. I am not using the bash scripts shared by you in the repo (I am assuming you talking about this one : https://github.com/DhruvanshuSan/TrigScint/blob/trunk/util/runReformatting.sh)

I made a simple bash script to run the macro which has commands of the type
ldmx root -l -q MacroName.C'("path/to/file.root")'

Note the above bash script format is not yet tested on the escape sequences suggested here.

Btw @bryngemark , did you try the escape sequences inside ldmx container on the sdf machine?

bryngemark · 2023-06-21T14:17:49Z

yes that's the bash script i'm referring to. i ran it on the LS batch queues, so not with slurm on sdf, with singularity. running the command the way it's done that script might help us further to disentangle what's going on.

DhruvanshuSan · 2023-06-23T04:22:33Z

Ok, I can try to run that bash script on sdf machine to check. So what nature of input does the script asks for? Is it like a text file with list of commands of the type ldmx root -l -q MacroName.C'("path/to/file.root")'?

bryngemark · 2023-06-26T10:06:49Z

@DhruvanshuSan it is this script: https://github.com/LDMX-Software/TrigScint/blob/56ceac8786421e2f4bbdabc5059d665af80cf002/util/runReformatting.sh

The only non-optional argument is a .txt file that lists input files to run the reformatting on. However you don't need to execute the full chain. Take the command line used for ROOT macro running on e.g.

 root -l -b -q ''$configDir'/../util/extractPedsAndGains.C+("'${linearRootFile}'", '${deadChannel}', "conv", '${logVerbosity}')'

that you can see here as inspiration and run just that line, or start from a reformatted .root file, or so.

DhruvanshuSan · 2023-06-28T17:30:08Z

Hi @bryngemark & @tomeichlersmith , I figured out a way to get rid of the issues we facing so far. One can try the following bash script :

#!/bin/bash
source /sdf/group/ldmx/users/dhparmar/ldmx-sw/scripts/ldmx-env.sh
ldmx root <<EOF
.x macroName.C("/path/to/file.root")
EOF

So the trick which worked is to run the macro inside the container in interactive mode and pass the argument to the macro via "here document" input redirection bash routine.

I also modified my SLURM script to incorporate this :

#!/bin/bash
#SBATCH --job-name=ldmx_response #name of the job
#SBATCH --nodes=1             #node count
#SBATCH --cpus-per-task=4   #cpu-cores per task
#SBATCH --mail-type=END,FAIL
#SBATCH --mem=2G   #memory per node
#SBATCH --time=00:30:00 #total run time
#SBATCH --output=SLURM_logs/ldmx_response_output_%A_%a.out
#SBATCH --error=SLURM_logs/ldmx_response_error_%A_%a.out
#SBATCH --array=1-298

#set number of threads
export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK

#time in milliseconds each thread should wait after completing execution of
#parallel region before sleeping
export KMP_BLOCKTIME=0

#bind threads to cores
export KMP_AFFINITY=granularity=fine,compact,0,0

cd ldmx-sw
source scripts/ldmx-env.sh
cd TrigScint/util
FILENAME="/sdf/group/ldmx/users/dhparmar/CommandList_ResponseSLURM.txt"
CommandName=`sed -n ${SLURM_ARRAY_TASK_ID}p $FILENAME`
pwd
echo $CommandName
ldmx root <<EOF
$CommandName
EOF

where $CommandName are of the type .x macroName("path/to/file.root").

And this works as expected. I tested my SLURM job array script and it processed all the files as expected.

Location of SLURM script on my device: /sdf/group/ldmx/users/dhparmar/TrySLURM.sh

Please let me know if the above scripts are working for you guys as well.

tomeichlersmith · 2023-07-05T16:20:39Z

Thank you @DhruvanshuSan , I've added this solution to the FAQ on the website and linked back to this issue for anyone searching for more context.

LDMX-Software/ldmx-software.github.io@f5af2b8

tomeichlersmith closed this as completed Jul 5, 2023

tomeichlersmith added this to the v3.3.0 milestone Sep 13, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Possible bug in ldmx root #1169

Possible bug in ldmx root #1169

DhruvanshuSan commented Jun 12, 2023 •

edited

tomeichlersmith commented Jun 12, 2023

tomeichlersmith commented Jun 12, 2023

tomeichlersmith commented Jun 12, 2023

bryngemark commented Jun 13, 2023 •

edited

tomeichlersmith commented Jun 13, 2023

DhruvanshuSan commented Jun 13, 2023 •

edited

bryngemark commented Jun 20, 2023

DhruvanshuSan commented Jun 21, 2023

bryngemark commented Jun 21, 2023

DhruvanshuSan commented Jun 23, 2023

bryngemark commented Jun 26, 2023

DhruvanshuSan commented Jun 28, 2023 •

edited

tomeichlersmith commented Jul 5, 2023

Possible bug in ldmx root #1169

Possible bug in ldmx root #1169

Comments

DhruvanshuSan commented Jun 12, 2023 • edited

tomeichlersmith commented Jun 12, 2023

tomeichlersmith commented Jun 12, 2023

In the meantime...

tomeichlersmith commented Jun 12, 2023

bryngemark commented Jun 13, 2023 • edited

tomeichlersmith commented Jun 13, 2023

DhruvanshuSan commented Jun 13, 2023 • edited

bryngemark commented Jun 20, 2023

DhruvanshuSan commented Jun 21, 2023

bryngemark commented Jun 21, 2023

DhruvanshuSan commented Jun 23, 2023

bryngemark commented Jun 26, 2023

DhruvanshuSan commented Jun 28, 2023 • edited

tomeichlersmith commented Jul 5, 2023

DhruvanshuSan commented Jun 12, 2023 •

edited

bryngemark commented Jun 13, 2023 •

edited

DhruvanshuSan commented Jun 13, 2023 •

edited

DhruvanshuSan commented Jun 28, 2023 •

edited