Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PROJQUAY-1397 - Add a DEBUG flag to the executors to prevent cleanup #631

Merged
merged 1 commit into from Dec 21, 2020

Conversation

kleesc
Copy link
Member

@kleesc kleesc commented Dec 18, 2020

When set to true, DEBUG will prevent the build nodes from shutting
down after the quay-builder service is done or fails, and will prevent the
build manager from cleaning up the instances (terminating EC2
instances or deleting k8s jobs).

This will allow debugging builder node issues, and should not be set
in a production environment.

The lifetime service will still exist. i.e The instance will still
shutdown after ~2h (EC2 instances will terminate, k8s jobs will
complete)

Setting DEBUG will also affect ALLOWED_WORKER_COUNT, as the
unterminated instances/jobs will still count towards the total number
of running workers.

Issue: https://issues.redhat.com/browse/PROJQUAY-1397
Changelog:

Docs:

Testing:

  • Set DEBUG to True in the executor(s)' config
  • Start a build from Quay
  • After a build completes or fails, the worker nodes should still be running in the executors.

Details:

@kleesc kleesc force-pushed the add-debug-flag-to-build-worker branch from 88dce11 to af2fa1b Compare December 18, 2020 19:26
@kleesc kleesc changed the title Add a DEBUG flag to the executors to prevent cleanup PROJQUAY-1397 - Add a DEBUG flag to the executors to prevent cleanup Dec 18, 2020
When set to true, DEBUG will prevent the build nodes from shutting
down after the quay-builder service is done or fails, and will prevent the
build manager from cleaning up the instances (terminating EC2
instances or deleting k8s jobs).

This will allow debugging builder node issues, and should not be set
in a production environment.

The lifetime service will still exist. i.e The instance will still
shutdown after ~2h (EC2 instances will terminate, k8s jobs will
complete)

Setting DEBUG will also affect ALLOWED_WORKER_COUNT, as the
unterminated instances/jobs will still count towards the total number
of running workers.
@kleesc kleesc force-pushed the add-debug-flag-to-build-worker branch from af2fa1b to 52f465c Compare December 18, 2020 19:28
Copy link
Contributor

@kurtismullins kurtismullins left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

It might be helpful to add this to the documentation.

@kleesc kleesc merged commit 6b1c1fb into quay:master Dec 21, 2020
@kleesc kleesc deleted the add-debug-flag-to-build-worker branch December 21, 2020 15:31
kleesc added a commit to kleesc/quay that referenced this pull request Dec 21, 2020
When set to true, DEBUG will prevent the build nodes from shutting
down after the quay-builder service is done or fails, and will prevent the
build manager from cleaning up the instances (terminating EC2
instances or deleting k8s jobs).

This will allow debugging builder node issues, and should not be set
in a production environment.

The lifetime service will still exist. i.e The instance will still
shutdown after ~2h (EC2 instances will terminate, k8s jobs will
complete)

Setting DEBUG will also affect ALLOWED_WORKER_COUNT, as the
unterminated instances/jobs will still count towards the total number
of running workers.
kleesc added a commit that referenced this pull request Dec 21, 2020
When set to true, DEBUG will prevent the build nodes from shutting
down after the quay-builder service is done or fails, and will prevent the
build manager from cleaning up the instances (terminating EC2
instances or deleting k8s jobs).

This will allow debugging builder node issues, and should not be set
in a production environment.

The lifetime service will still exist. i.e The instance will still
shutdown after ~2h (EC2 instances will terminate, k8s jobs will
complete)

Setting DEBUG will also affect ALLOWED_WORKER_COUNT, as the
unterminated instances/jobs will still count towards the total number
of running workers.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
2 participants