Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Properly handle LFS files and long-stopping sessions when creating autosaves #920

Closed
olevski opened this issue Feb 10, 2022 · 1 comment · Fixed by #951
Closed

Properly handle LFS files and long-stopping sessions when creating autosaves #920

olevski opened this issue Feb 10, 2022 · 1 comment · Fixed by #951
Assignees
Labels
bug Something isn't working

Comments

@olevski
Copy link
Member

olevski commented Feb 10, 2022

Currently all (or at least some) of our containers are ignoring the SIGTERM signal from k8s. When this is the case k8s waits for a default duration of 30 secs and then sends the forceful termination SIGKILL signal.

This is essentially preventing the creation of autosave branches in the case when creating such a branch takes more than 30 seconds. This leads to lost work - especially in cases where the repo has a lot of LFS files and it takes a bit of time for the autosave branch to be crated.

In addition to this we should also properly handle any large uncommitted files that should go to LFS and put them in LFS in the autosave branch. We can potentially use renku save for this purpose.

@olevski olevski added the bug Something isn't working label Feb 10, 2022
@olevski
Copy link
Member Author

olevski commented Feb 10, 2022

We can use tini to make sure all containers get and respect the SIGTERM signal. But a solution that does this without using an additional library would be even nicer.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant