Dealing with OOM plugin instances

[jennydaman]

How best to deal with OOM?

History

The problem of OOM was not considered in the system design of ChRIS. There is no way to indicate the reason for a job's failure from Kubernetes to CUBE. As a hacky solution, I added a feature where Kubernetes' "termination reason" would be added to the plugin logs. FNNDSC/pman@cb57cdf

Current Problems

Plugins have a min_memory_limit and a max_memory_limit. However it's inconvenient to specify a value for memory_limit when creating plugin instances, so practically speaking, min_memory_limit is the default memory_limit 99% of the time. When a plugin instance is OOMKilled, our course of action is usually to increase the value of min_memory_limit in the source code, let CI/CD push out a new version, and bump the version.

Things to Explore

How can we design ChRIS_ui so that it's easier to set the memory_limit of plugin instances created by a workflow?

Should OOMKilled and other Kubernetes-side termination reasons be recognized as statuses by CUBE, so that termination reasons can be queried for more easily and correctly?

Other workflow engines, e.g. cromwell, can automatically retry jobs with more memory¹. Should we copy this behavior? (Should we try to replace pfcon entirely with cromwell? That's another topic...)

Footnotes

1. https://cromwell.readthedocs.io/en/latest/cromwell_features/RetryWithMoreMemory/ ↩