Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Do not use a PVC between the C++ writer and runner containers #24

Closed
gordonwatts opened this issue Sep 16, 2019 · 1 comment
Closed
Assignees

Comments

@gordonwatts
Copy link
Collaborator

Currently

The transformer is two containers as python 3 and the ATLAS runtime do not get along well. The first container writes the C++ code, and the second compiles and runs it. To communicate between the two everything is written to a Kubernetes PVC.

Why change

PVC's are generally a pain in the but and add another point of configuration. Don't use them unless there is a real reason. Given it takes ~100's of ms to write these files, caching is really not needed.

Change to

Two options were considered:

  • Write a zip of the files to a database
  • Write a zip of the files into the message that is sent to the runner container.

The latter seems to be the most simple solution. The files are quite small. Especially for something like ServiceX, where the queries are not building 100's of histograms. I expect the C++ zip to be well under 100K and probably more like 10K or so.

@gordonwatts
Copy link
Collaborator Author

This has been changed in the referenced repos now. While the code isn't perfect, it does work. It would be best to do a code review of this rabbit mq code at some point.

Ready to be closed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants