Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

CAAS Charm Upgrade #11395

Merged
merged 1 commit into from
Apr 15, 2020
Merged

CAAS Charm Upgrade #11395

merged 1 commit into from
Apr 15, 2020

Conversation

hpidcock
Copy link
Member

@hpidcock hpidcock commented Apr 2, 2020 •
edited
Loading

Please provide the following details to expedite Pull Request review:

Checklist

  • Checked if it requires a pylibjuju change?
  • Added integration tests for the PR?
  • Added or updated doc.go related to packages changed?
  • Do comments answer the question of why design decisions were made?

Description of change

  • Moved CAAS unit init into uniter and added resolver op. Removed caasunitinit worker.
  • Removed noop upgrade operation.
  • Uniter deploys own charm bundle to fix race conditions with caasoperator.
  • Block action running on outdated charm.
  • Support charm upgrade on CAAS workloads.
  • WatchContainerStart can now watch containers with a regex pattern.
  • Fix unit init process exit race condition for k8s.

QA steps

juju deploy cs:~charmed-osm/mongodb-k8s-24
juju upgrade-charm mongodb-k8s --channel edge
juju run-action mongodb-k8s/0 backup

Documentation changes

N/A

Bug reference

https://bugs.launchpad.net/juju/+bug/1866856

@hpidcock hpidcock requested a review from wallyworld April 2, 2020 05:16
wallyworld
wallyworld reviewed Apr 6, 2020
View reviewed changes
caas/kubernetes/provider/k8s.go Outdated Show resolved Hide resolved
worker/caasoperator/caasoperator.go Outdated Show resolved Hide resolved
worker/caasoperator/initializer.go Show resolved Hide resolved
worker/caasoperator/initializer.go Outdated Show resolved Hide resolved
worker/caasoperator/initializer.go Outdated Show resolved Hide resolved
worker/uniter/remotestate/watcher.go Show resolved Hide resolved
worker/uniter/container/resolver.go Outdated Show resolved Hide resolved
worker/uniter/container/resolver.go Show resolved Hide resolved
worker/uniter/container/resolver.go Outdated Show resolved Hide resolved
cmd/jujud/agent/caasunitinit.go Show resolved Hide resolved
@hpidcock hpidcock force-pushed the charm-upgrade branch 3 times, most recently from 3dca3b5 to e8ac890 Compare April 8, 2020 08:26
@hpidcock hpidcock changed the title WIP CAAS Charm Upgrade CAAS Charm Upgrade Apr 8, 2020
@hpidcock hpidcock requested a review from wallyworld April 8, 2020 08:44
@hpidcock hpidcock force-pushed the charm-upgrade branch 3 times, most recently from 3cba305 to 7bbdc50 Compare April 9, 2020 00:49
@hpidcock
Copy link
Member Author

hpidcock commented Apr 9, 2020

!!build!!

@wallyworld
Copy link
Member

Testing showed an error in show-status-log. Need to investigate.

Time                        Type       Status       Message
09 Apr 2020 12:07:08+10:00  juju-unit  allocating   
09 Apr 2020 12:07:08+10:00  workload   waiting      agent initializing
09 Apr 2020 12:07:23+10:00  workload   maintenance  installing charm software
09 Apr 2020 12:07:23+10:00  juju-unit  executing    running install hook
09 Apr 2020 12:07:24+10:00  workload   maintenance  fetching resource: mysql_image
09 Apr 2020 12:07:24+10:00  workload   maintenance  Configuring mysql container
09 Apr 2020 12:07:24+10:00  workload   active       
09 Apr 2020 12:07:25+10:00  juju-unit  executing    running leader-elected hook
09 Apr 2020 12:07:25+10:00  workload   waiting      waiting for container
09 Apr 2020 12:07:26+10:00  juju-unit  executing    running config-changed hook
09 Apr 2020 12:07:26+10:00  juju-unit  allocating   Started container juju-pod-init
09 Apr 2020 12:07:26+10:00  workload   waiting      Started container juju-pod-init
09 Apr 2020 12:07:27+10:00  juju-unit  executing    running start hook
09 Apr 2020 12:07:30+10:00  juju-unit  failed       resolver loop error
09 Apr 2020 12:07:31+10:00  juju-unit  allocating   Pulling image "mariadb"
09 Apr 2020 12:07:31+10:00  workload   waiting      Pulling image "mariadb"
09 Apr 2020 12:07:34+10:00  juju-unit  idle         
09 Apr 2020 12:07:34+10:00  workload   active

@wallyworld
Copy link
Member

Running a local upgrade-charm failed also. From model logs

2020-04-09 02:07:30 INFO juju.cmd.jujud caasunitinit.go:219 copy /tmp/unit-mariadb-k8s-0172278128/charm => /var/lib/juju/agents/unit-mariadb-k8s-0/charm
2020-04-09 02:07:30 INFO juju.cmd.jujud caasunitinit.go:229 copy /tmp/unit-mariadb-k8s-0172278128/charm => /var/lib/juju/agents/unit-mariadb-k8s-0/charm
: command terminated with exit code 137
application-mariadb-k8s: 12:07:34 ERROR juju.worker.caasoperator trigger running status for caas unit mariadb-k8s/0

@hpidcock
Copy link
Member Author

hpidcock commented Apr 9, 2020

command terminated with exit code 137

Ahhhhh does the mariadb-k8s have a memory limit or is your system at memory limits?

@wallyworld
Copy link
Member

I have lots-o-memory free, no memory limits was explicitly set when deploying. I can do some more digging.

@hpidcock
Copy link
Member Author

After sleeping on it, I know what this is now. The main process of the container is exiting before the signalling process. So then the container runtime sends a SIGKILL to the process we are watching. I'll have a fix for this soon.

wallyworld
wallyworld approved these changes Apr 15, 2020
View reviewed changes
Copy link
Member

@wallyworld wallyworld left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am pretty sure we'll need a follow up PR to get the new local state attributes saved in the controller

caas/kubernetes/provider/exec/status.go Show resolved Hide resolved
caas/kubernetes/provider/exec/status.go Outdated Show resolved Hide resolved
caas/kubernetes/provider/exec/status_test.go Outdated Show resolved Hide resolved
cmd/jujud/agent/caasunitinit.go Outdated Show resolved Hide resolved
cmd/jujud/agent/caasunitinit.go Show resolved Hide resolved
worker/uniter/container/resolver.go Outdated Show resolved Hide resolved
worker/uniter/operation/remoteinit_test.go Outdated Show resolved Hide resolved
worker/uniter/operation/state.go Outdated Show resolved Hide resolved
worker/uniter/remotestate/snapshot.go Outdated Show resolved Hide resolved
@hpidcock
Fix CAAS charm upgrade
73ff3fc 
- Moved CAAS unit init into uniter and added resolver op. Removed
caasunitinit worker.
- Removed noop upgrade operation.
- Uniter deploys own charm bundle to fix race conditions with
caasoperator.
- Block action running on outdated charm.
- Support charm upgrade on CAAS workloads.
- WatchContainerStart can now watch containers with a regex pattern.
- Fix unit init process exit race condition for k8s.
@hpidcock
Copy link
Member Author

$$merge$$

2 similar comments
@hpidcock
Copy link
Member Author

$$merge$$

@hpidcock
Copy link
Member Author

$$merge$$

@jujubot jujubot merged commit 5a6fb25 into juju:develop Apr 15, 2020
@wallyworld
Copy link
Member

juju actions not upgraded

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@wallyworld wallyworld wallyworld approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@hpidcock @wallyworld @jujubot