DM-43584: Add an option to turn off caching entirely #149
Conversation
| # in chronological order. Turn off for large development runs. | ||
| cache_calibs = bool(int(os.environ.get("DEBUG_CACHE_CALIBS", '1'))) | ||
| # TODO: workaround for DM-40193 | ||
| cache_anything = bool(int(os.environ.get("DEBUG_CACHE_INPUTS", '1'))) |
There was a problem hiding this comment.
The transitions from Helm values true/false to K8s YAML 1/0 to Python True/False are a little mind-bending, but I guess it's better than getting true and True confused.
There was a problem hiding this comment.
Unfortunately, since environment variables must be strings, and any nonempty string casts to bool as True, there needs to be an explicit translation step somewhere. I picked this convention mainly because it mirrors the existing LSST_DISABLE_BUCKET_VALIDATION.
| self.butler.registry.removeCollection(member) | ||
| self.butler.removeRuns(calib_runs, unstore=True) | ||
|
|
||
| # TODO: workaround for DM-40193 |
There was a problem hiding this comment.
If this moved up to just after the refresh(), couldn't you put a return after it and skip all the other work, while still being nearly as compact code-wise? But I guess it isn't really that much extra work, and in fact getting rid of other dataset types might speed up the ... query.
There was a problem hiding this comment.
Nope, doesn't work -- I don't know if it's extra registry entries or something else, but the rearranged order leaves about 190 MB in the pods instead of 2 MB. I'm worried that may grow in time, so I'll go back to the original implementation.
kfindeisen
force-pushed
the
tickets/DM-43584
branch
from
afb6a22 to
d36c631
Compare
This option resets the local repo to a clean slate on every run, preventing disk leaks entirely.
kfindeisen
force-pushed
the
tickets/DM-43584
branch
from
d36c631 to
aded10c
Compare
2 participants
This PR adds an emergency option that turns off dataset caching entirely, for situations where the local repo must not be allowed to grow at all.
Closes #148.