Add -ness checks and refactor migrations #1674
Conversation
dhageman
force-pushed
the
inesscheck
branch
5 times, most recently
from
e49c674
to
2f1f75b
Compare
|
I just ran through the code one more time. The k8s job get created as expected.
Then runs to completion. The logs for the migration container show the migrations have all been completed, as can be seen by the output of: |
| @@ -0,0 +1,57 @@ | |||
| --- | |||
|
|
|||
| - name: Check for pending migrations | |||
There was a problem hiding this comment.
👍
This task ensures that the k8s job is only run if there are new schema changes to be migrated.
And the unique name based on the version and the random hash ensure that there are no conflicts with other k8s jobs from previous migrations.
| {{ lookup("template", "../common/templates/labels/version.yaml.j2") | indent(width=4) | trim }} | ||
| spec: | ||
| template: | ||
| spec: |
There was a problem hiding this comment.
@dhageman One thought I had is that we might want to be mindful of cleaning up these migration jobs in case a deployment gets in a bad state and winds up creating them in a loop after the timeout is reached. The job containers are automatically cleaned up after 3600 seconds, which is pretty reasonable.
If we need to, we can adjust this via .spec.ttlSecondsAfterFinished. Though that is probably a good default to start with since it strikes a nice balance between "not cluttering things" and having logs stick around long enough for debugging.
There was a problem hiding this comment.
You make an excellent point. There is definitely a balance there.
|
@dhageman thank you for your time and effort to develop this solution. I think it is robust and will be a good pattern. |
SUMMARY
This PR implements liveness & readiness probes supporting the split web/task container configuration. It also adjusts database migrations to be done in the init container of the task pod.
Addresses #414 & #926
ISSUE TYPE
ADDITIONAL INFORMATION
It should be acknowledged that this is not the first work on implementing these features.