You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Currently all (or at least some) of our containers are ignoring the SIGTERM signal from k8s. When this is the case k8s waits for a default duration of 30 secs and then sends the forceful termination SIGKILL signal.
This is essentially preventing the creation of autosave branches in the case when creating such a branch takes more than 30 seconds. This leads to lost work - especially in cases where the repo has a lot of LFS files and it takes a bit of time for the autosave branch to be crated.
In addition to this we should also properly handle any large uncommitted files that should go to LFS and put them in LFS in the autosave branch. We can potentially use renku save for this purpose.
The text was updated successfully, but these errors were encountered:
We can use tini to make sure all containers get and respect the SIGTERM signal. But a solution that does this without using an additional library would be even nicer.
Currently all (or at least some) of our containers are ignoring the
SIGTERM
signal from k8s. When this is the case k8s waits for a default duration of 30 secs and then sends the forceful terminationSIGKILL
signal.This is essentially preventing the creation of autosave branches in the case when creating such a branch takes more than 30 seconds. This leads to lost work - especially in cases where the repo has a lot of LFS files and it takes a bit of time for the autosave branch to be crated.
In addition to this we should also properly handle any large uncommitted files that should go to LFS and put them in LFS in the autosave branch. We can potentially use
renku save
for this purpose.The text was updated successfully, but these errors were encountered: