This repository was archived by the owner on May 6, 2020. It is now read-only.
-
Notifications
You must be signed in to change notification settings - Fork 53
This repository was archived by the owner on May 6, 2020. It is now read-only.
Controller crashes if a deployment is in progress when it's started #903
Copy link
Copy link
Closed
Description
Repro steps:
- Start a Deis cluster with the
DEIS_KUBERNETES_DEPLOYMENTSenv var set to"true"in the controller RC manifest (see https://deis.com/docs/workflow/applications/deploying-apps/#tuning-application-settings for details on turning on deployments) - Create a new app (
deis create --no-remote pulltest) deis pullthe app (deis pull deis/example-go -a pulltest)- Scale the app to something high, so the deployment takes a long time (
deis scale cmd=1000 -a pulltest) - Delete the controller RC (
kubectl delete rc deis-controller --namespace=deis) - Create the controller RC again (
kubectl create -f $(helmc home)/workspace/charts/workflow-v2.2.0/manifests/deis-controller-rc.yaml) - See logs of the controller pod after it's in the
RUNNINGstate (kubectl logs -f <controller pod>)
After step 7, you should see a traceback that looks like:
Deploying available applications
Traceback (most recent call last):
File "/app/api/models/app.py", line 528, in deploy
raise AlreadyExists('Deployment for {} is already in progress'.format(name))
api.exceptions.AlreadyExists: Deployment for pulltest-cmd is already in progress
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/app/manage.py", line 12, in <module>
execute_from_command_line(sys.argv)
File "/usr/local/lib/python3.5/dist-packages/django/core/management/__init__.py", line 353, in execute_from_command_line
utility.execute()
File "/usr/local/lib/python3.5/dist-packages/django/core/management/__init__.py", line 345, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/usr/local/lib/python3.5/dist-packages/django/core/management/base.py", line 348, in run_from_argv
self.execute(*args, **cmd_options)
File "/usr/local/lib/python3.5/dist-packages/django/core/management/base.py", line 399, in execute
output = self.handle(*args, **options)
File "/app/api/management/commands/load_db_state_to_k8s.py", line 39, in handle
application.deploy(rel)
File "/app/api/models/app.py", line 548, in deploy
raise ServiceUnavailable(err) from e
api.exceptions.ServiceUnavailable: pulltest-cmd (app::deploy): Deployment for pulltest-cmd is already in progressReactions are currently unavailable