storageprovisioner: fix watching of volume-backed filesystems #7346

Merged
merged 4 commits into from May 17, 2017

Conversation

Projects
None yet
3 participants
Member

axw commented May 16, 2017

Description of change

A recent change was made to the state package's filesystem and filesystem attachment watchers. The change was to filter the results based on the state of volume attachments. This doesn't work without also watching the volume attachment collection for changes, so that change has been reverted.

In its place, we introduce composite watchers in the apiserver/storageprovisioner/internal/filesystemwatcher package. These watchers compose watchers for model and machine filesystems, and volume attachments, to produce answers that the storage provisioner worker expects.

There are some incidental bug fixes, found while verifying this change:

  • the gce provider's AttachVolumes has been fixed to set DeviceLink instead of DevicePath, as in CreateVolumes
  • the storage provisioner will now react to block device changes when there are incomplete filesystem attachments, as well as incomplete filesystems
  • the storage provisioner removes attachments when they are Dead, not immediately after detaching the storage. This avoids an ugly "not found" error.

QA steps

$ juju bootstrap google
$ juju deploy postgresql --storage pgdata=gce,10G
(wait)
$ juju detach-storage pgdata/0
(wait)
$ juju attach-storage postgresql/0 pgdata/0
(storage should show up as attached)

Documentation changes

None.

Bug reference

None.

I think this seems ok, but I might be misunderstanding the semantics of the machineFilesystemsWatcher as I don't get why all 3 contained watchers need to fire before an event is omitted.

+ // The tests all operate on machine "0", and the watchers
+ // should ignore attachments for other machines, so we should
+ // never get here.
+ panic("should not get here")
@wallyworld

wallyworld May 16, 2017

Owner

Instead of panicing, why not return an error and let the caller's c.Assert(err, jc.ErrorIsNil) fail the test. IMO since the panic would result from a logic error ("code should do such and such"), any such violation is a test failure.

@axw

axw May 17, 2017

Member

Done.

+// results from machine- and model-scoped watchers, to conform to the behaviour
+// of the storageprovisioner worker. The model-level storageprovisioner watches
+// model-scoped filesystems that have no backing volume. The machine-level worker
+// watches both machine-scoped filesytems, and model-scoped filesystems attached
@wallyworld

wallyworld May 16, 2017

Owner

"that have no backing volume" also?

@axw

axw May 17, 2017

Member

I added "whose backing volumes are" between "filesystems" and "attached"

+// WatchModelFilesystems returns a strings watcher that reports model-scoped
+// filesystems that have no backing volume. Volume-backed filesystems are
+// always managed by the machine to which they are attached.
+func (fw Watchers) WatchModelFilesystems() state.StringsWatcher {
@wallyworld

wallyworld May 16, 2017

Owner

might be verbose, but could it be WatchOrphanedModelFilesystems() or WatchUnbackedModelFileSystems() or something? the method name doesn't imply that there's a condition on which model filesystems are watched

@axw

axw May 17, 2017

Member

I'll change it to "ModelManaged" and "MachineManaged", to make it clear that we're interested in machines that are managed by the model-level or machine-level storage provisioner.

+ w.changes = make(set.Strings)
+ out = nil
+ }
+ if machineFilesystemsReceived &&
@wallyworld

wallyworld May 16, 2017

Owner

Is that right? All 3 underlying watchers have to fire before we send any events?

@axw

axw May 17, 2017

Member

Yes. The way all watchers work is to first give the initial state of the world in a single event, and then later changes can report on individual aspects. If we were to report the machine-scoped filesystems without also reporting the model-scoped ones, then the worker would think that the model-scoped ones were not part of the model any more.

state/watcher/watchertest/strings.go
+ C chan []string
+}
+
+func NewStringsWatcher(ch chan []string) *StringsWatcher {
@wallyworld

wallyworld May 16, 2017

Owner

should comment here and above

@axw

axw May 17, 2017

Member

done

Member

axw commented May 17, 2017

$$merge$$

Contributor

jujubot commented May 17, 2017

Status: merge request accepted. Url: http://juju-ci.vapour.ws:8080/job/github-merge-juju

Contributor

jujubot commented May 17, 2017

Build failed: Tests failed
build url: http://juju-ci.vapour.ws:8080/job/github-merge-juju/10905

axw added some commits May 14, 2017

worker/storageprovisioner: handle Dead attachments
We previously handled Dying storage attachments by
destroying and then removing them. The watcher will
still observe the lifecycle state going from Dying
to Dead, so we defer the removal until the Dead
state is observed.
Member

axw commented May 17, 2017

$$merge$$

Contributor

jujubot commented May 17, 2017

Status: merge request accepted. Url: http://juju-ci.vapour.ws:8080/job/github-merge-juju

Contributor

jujubot commented May 17, 2017

Build failed: Tests failed
build url: http://juju-ci.vapour.ws:8080/job/github-merge-juju/10906

Member

axw commented May 17, 2017

$$merge$$

Contributor

jujubot commented May 17, 2017

Status: merge request accepted. Url: http://juju-ci.vapour.ws:8080/job/github-merge-juju

@jujubot jujubot merged commit 91b7af8 into juju:feature-persistent-storage May 17, 2017

1 check failed

github-check-merge-juju Built PR, ran unit tests, and tested LXD deploy. Use !!.*!! to request another build. IE, !!build!!, !!retry!!
Details
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment