Running a Ganga job locally through a Docker container throws syntax error

[dg1223]

Here's how to reproduce the issue. I have Ganga installed inside a virtual environment called gangaenv.

Activate Ganga

cd gangaenv/
. bin/activate
ganga

Run a Ganga job inside a Docker container

j = Job(name='dockertest')
j.virtualization = Docker(image="cern/c2mon-es")
j.submit()

Here's the error that I get:

Ganga In [8]: j.submit()
Ganga Out [8]: 1

[20:47:18]
Ganga In [9]:   File "/home/shamir/gangadir/workspace/shamir/LocalXML/12/input/__jobscript__", line 230
    execmd = ['docker', 'run', '--rm', '-v', workdir+":"+"/work_dir"] +
                                                                      ^
SyntaxError: invalid syntax

I was able to hack the input file that is created by the job locally and successfully run the job. But it doesn't solve the core issue. Here's how I did it:

1. Open /home/[your_linux_username]/gangadir/workspace/[your_linux_username]/LocalXML/[job_number]/input/__jobscript__

2. Go to line 230

3. Add a space and an escape character \ after the + sign.

       execmd = ['docker', 'run', '--rm', '-v', workdir+":"+"/work_dir"] + \

4. Go to line 253

5. Repeat step 3

   execmd = [binary, '--quiet', 'run', '--rm', '--volume', workdir+":"+"/work_dir"] + \

6. Save file

7. Force job status to fail → j..force_status('failed')

8. Resubmit job → j.resubmit()

Seems to solve the issue.

I believe I found the main script Docker.py that is causing this error. However, applying the same fix on the script doesn't resolve the issue. I tried digging further to see how the script is called but couldn't figure it out in a short period. I tried to apply the following fix:

1. Open ganga/GangaCore/Lib/Virtualization/Docker.py
2. In line 81, add a space and an escape character \ after the + sign.
3. In line 104, add a space and an escape character \ after the + sign.
4. Save
5. Submit a new job -> the issue persists

I might be able to solve this issue if I get some guidance from someone who's experienced with the codebase (or if I invest a lot more time :) )

[egede]

Your analysis to find the syntax error looks perfectly correct. I am confused by why that did not work. Are few potential pitfalls:

• Did you create a new job from scratch or just copy the previous job. If you just copy the previous one, it will not rewrite the jobscript.
• Are you sure that you changed the code that is actually executed. If you did a pip install ganga and at the same time checked out ganga from GitHub, you may well have two instances of the code and edit the wrong one.

If you make a pull request with the suggested change above, I am happy to test it.

[dg1223]

Thanks for the hints! I created new jobs every time but I think I checked out ganga from GitHub as well as pip installed ganga at the same time. I'll rerun the tests and confirm tomorrow.

[dg1223]

So, I was indeed running pip install ganga but making changes locally :)

Installed Ganga from source and tested the fix. It worked. Opened a PR #2301