state: clean up machine-bound storage properly #7052

Merged
merged 1 commit into from Mar 1, 2017

Conversation

Projects
None yet
4 participants
Member

axw commented Mar 1, 2017

Description of change

We were previously relying on a combination of kludges to
get non-detachable machine storage to be removed along
with a machine. Due to recent changes to expose more of
storage (i.e.g for removal and detachment, etc.), machine
storage stopped getting removed along with the machine.

For "non-detachable" storage, the one-and-only attachment
may still be destroyed and removed independently of the
volume/filesystem. For example, if you destroy a unit on
a machine, and the unit has a "loop" volume, then the
volume will be marked Dying when the unit is destroyed.
That will cascade and mark the volume attachment as
Dying; the volume will be detached, at which point the
volume attachment is removed from the model. At this
point there is no association remaining between the machine
and the volume.

To fix this, we now record the ID of the machine to which
a volume or filesystem is inherently bound (if it is at all)
on the volume/filesystem document. We can then immediately
identify all volumes/filesystems that are inherently bound to
a machine when the machine is to be removed from state,
and remove them as well. We also use this to identify whether
the storage is detachable in various parts.

QA steps

SCENARIO 1

  1. juju bootstrap localhost
  2. juju deploy postgresql --storage pgdata=rootfs
    (wait for unit to become idle)
  3. juju remove-application postgresql
    (wait for unit to be removed)
  4. confirm there is no storage remaining

SCENARIO 2
As above, but migrate the model to another controller
between steps 2 and 3.

SCENARIO 3
Run the "assess_storage.py" test from juju-ci-tools.

Documentation changes

None.

Bug reference

https://bugs.launchpad.net/bugs/1667883

mjs approved these changes Mar 1, 2017

Great stuff

state/cleanup.go
@@ -541,6 +541,14 @@ func cleanupDyingMachineResources(m *Machine) error {
return errors.Annotate(err, "getting machine volume attachments")
}
for _, va := range volumeAttachments {
+ detachable, err := isDetachableVolumeTag(m.st, va.Volume())
@mjs

mjs Mar 1, 2017

Contributor

if detachable isn't needed beyond here then I'd tend to roll the call into the if so that detachable doesn't escape, and you save a few lines.

@axw

axw Mar 1, 2017

Member

Done

state/cleanup.go
@@ -556,6 +564,14 @@ func cleanupDyingMachineResources(m *Machine) error {
return errors.Annotate(err, "getting machine filesystem attachments")
}
for _, fsa := range filesystemAttachments {
+ detachable, err := isDetachableFilesystemTag(m.st, fsa.Filesystem())
@mjs

mjs Mar 1, 2017

Contributor

as above

@axw

axw Mar 1, 2017

Member

done

+ // to be Dying, then Dead, and finally removed. Only once the attachment
+ // is removed will the filesystem transition to Dead and then be removed.
+ // Therefore, there may be filesystems that are bound, but not attached,
+ // to the machine.
@mjs

mjs Mar 1, 2017

Contributor

Thanks for the nice description.

state/filesystem_test.go
@@ -936,7 +947,7 @@ func (s *FilesystemStateSuite) testFilesystemAttachmentParamsConcurrent(c *gc.C,
func (s *FilesystemStateSuite) TestFilesystemAttachmentParamsConcurrentRemove(c *gc.C) {
// this creates a filesystem mounted at "/srv".
- filesystem, machine := s.setupFilesystemAttachment(c, "rootfs")
+ filesystem, machine := s.setupFilesystemAttachment(c, "environscoped")
@mjs

mjs Mar 1, 2017

Contributor

environ?

@axw

axw Mar 1, 2017

Member

legacy, renamed to modelscoped

state/migration_import.go
@@ -1507,6 +1507,13 @@ func (i *importer) addVolume(volume description.Volume) error {
Info: info,
AttachmentCount: len(attachments),
}
+ detachable, err := isDetachableVolumePool(i.st, volume.Pool())
@mjs

mjs Mar 1, 2017

Contributor

as above

@axw

axw Mar 1, 2017

Member

done

state/migration_import.go
@@ -1603,6 +1610,13 @@ func (i *importer) addFilesystem(filesystem description.Filesystem) error {
Info: info,
AttachmentCount: len(attachments),
}
+ detachable, err := isDetachableFilesystemPool(i.st, filesystem.Pool())
@mjs

mjs Mar 1, 2017

Contributor

as above

@axw

axw Mar 1, 2017

Member

done

+ if err != nil {
+ return errors.Trace(err)
+ }
+ if len(attachments) != 1 {
@mjs

mjs Mar 1, 2017

Contributor

maybe add a comment about this along the lines of multiple attachments meaning it must be detachable

@axw

axw Mar 1, 2017

Member

it's more about being defensive; I've added a comment

+ if err != nil {
+ return errors.Trace(err)
+ }
+ if len(attachments) != 1 {
@mjs

mjs Mar 1, 2017

Contributor

as above

@axw

axw Mar 1, 2017

Member

ditto

- // e.g. NFS shares, then we need to check the filesystem info
- // to decide whether or not to remove.
+// isDetachableFilesystemPool reports whether or not the given
+// storage pool will create a filesystem that is not inherently
@wallyworld

wallyworld Mar 1, 2017

Owner

"is not inherently a machine" seems clunky
"is not inherently bound to a machine" perhaps?

@axw

axw Mar 1, 2017

Member

I accidentally a word. Fixed

Member

axw commented Mar 1, 2017

$$merge$$

Contributor

jujubot commented Mar 1, 2017

Status: merge request accepted. Url: http://juju-ci.vapour.ws:8080/job/github-merge-juju

Contributor

jujubot commented Mar 1, 2017

Build failed: Tests failed
build url: http://juju-ci.vapour.ws:8080/job/github-merge-juju/10389

Member

axw commented Mar 1, 2017

$$merge$$

Contributor

jujubot commented Mar 1, 2017

Status: merge request accepted. Url: http://juju-ci.vapour.ws:8080/job/github-merge-juju

@jujubot jujubot merged commit 751ea06 into juju:develop Mar 1, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment