Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Removed AirflowDB #322

Merged
merged 8 commits into from Sep 20, 2023
Merged

Removed AirflowDB #322

merged 8 commits into from Sep 20, 2023

Conversation

dervoeti
Copy link
Member

@dervoeti dervoeti commented Sep 6, 2023

Description

Closes #259. As discussed in the issue, the scheduler takes care of the initialization of the database.

  • Removed .spec.clusterConfig.databaseInitialization (this was just to enable logging for the database initialization job, which doesn't exist anymore)
  • Removed all the AirflowDB related code
  • Moved database init commands from a separate Job to the command of the scheduler Pod
  • Changed Pod Management Policy to OrderedReady for the scheduler, to make sure multiple scheduler Pods start after another and the init commands don't run in parallel
  • Adjusted the docs. I removed the waiting for the database initialization job and just wrote "After a while, all the Pods in the StatefulSets should be ready". We could also add a wait command for the scheduler here: kubectl rollout status --watch --timeout=5m statefulset/airflow-scheduler-default

Definition of Done Checklist

  • Not all of these items are applicable to all PRs, the author should update this template to only leave the boxes in that are relevant
  • Please make sure all these things are done and tick the boxes

Author

Reviewer

Acceptance

@maltesander maltesander self-requested a review September 7, 2023 11:59
maltesander
maltesander previously approved these changes Sep 7, 2023
Copy link
Member

@maltesander maltesander left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! There is still a conflict.

rust/operator-binary/src/airflow_controller.rs Outdated Show resolved Hide resolved
dervoeti and others added 2 commits September 7, 2023 15:52
Co-authored-by: Malte Sander <malte.sander.it@gmail.com>
@dervoeti
Copy link
Member Author

dervoeti commented Sep 7, 2023

Thanks for reviewing! Conflict is resolved :)

@dervoeti
Copy link
Member Author

We decided we want to move database initialization into a separate Init Container (see stackabletech/superset-operator#404). Will add this to this PR shortly.

@dervoeti
Copy link
Member Author

Update: We decided against moving database initialization to an init container, since currently the logs of an init container don't get captured, if the init container fails. Database initialization stays in the scheduler container for now. Also, the CRD change was approved, so this could be merged I'd say.

@maltesander maltesander added changelog/highlight Items worth mentioning in the Platform changelog changelog/breaking Breaking changes changelog/crd-change Any issue that changes a CRD (breaking and non-breaking changes) should get this label labels Sep 20, 2023
Copy link
Member

@maltesander maltesander left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@dervoeti dervoeti added this pull request to the merge queue Sep 20, 2023
Merged via the queue into main with commit dbcfb2f Sep 20, 2023
29 of 30 checks passed
@dervoeti dervoeti deleted the fix/remove-airflowdb branch September 20, 2023 12:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
changelog/breaking Breaking changes changelog/crd-change Any issue that changes a CRD (breaking and non-breaking changes) should get this label changelog/highlight Items worth mentioning in the Platform changelog
Projects
None yet
Development

Successfully merging this pull request may close these issues.

AirflowDB never recovers out of failed state
2 participants