[FLOC-4452] Delay invariant checks until all diff operations have been applied #2839

wallrj · 2016-06-21T15:48:30Z

Fixes: https://clusterhq.atlassian.net/browse/FLOC-4452

…r so as not to trigger the invariant check until all attributes have been set.

wallrj · 2016-06-22T13:59:52Z

Well I think I've fixed one version of this error,

[root@ip-172-31-0-243 ~]# find /data/jenkins-jobs/ClusterHQ-flocker/jobs/invariant-error-FLOC-4452/ -type f -name 'remote_logs.log' | xargs grep --count "Either all or none of"
/data/jenkins-jobs/ClusterHQ-flocker/jobs/invariant-error-FLOC-4452/jobs/run_acceptance_loopback_on_AWS_CentOS_7_for_flocker.acceptance/builds/1/archive/remote_logs.log:0
/data/jenkins-jobs/ClusterHQ-flocker/jobs/invariant-error-FLOC-4452/jobs/_run_acceptance_on_AWS_Ubuntu_Trusty_with_EBS/builds/1/archive/remote_logs.log:0
/data/jenkins-jobs/ClusterHQ-flocker/jobs/invariant-error-FLOC-4452/jobs/_run_acceptance_on_Rackspace_CentOS_7_with_Cinder/builds/1/archive/remote_logs.log:6
/data/jenkins-jobs/ClusterHQ-flocker/jobs/invariant-error-FLOC-4452/jobs/run_acceptance_loopback_on_AWS_Ubuntu_Trusty_for_flocker.acceptance/builds/1/archive/remote_logs.log:0
/data/jenkins-jobs/ClusterHQ-flocker/jobs/invariant-error-FLOC-4452/jobs/__main_multijob/workspace/run_acceptance_loopback_on_AWS_CentOS_7_for_flocker.acceptance/remote_logs.log:0
/data/jenkins-jobs/ClusterHQ-flocker/jobs/invariant-error-FLOC-4452/jobs/__main_multijob/workspace/run_acceptance_loopback_on_AWS_Ubuntu_Trusty_for_flocker.acceptance/remote_logs.log:0
/data/jenkins-jobs/ClusterHQ-flocker/jobs/invariant-error-FLOC-4452/jobs/_run_acceptance_on_AWS_CentOS_7_with_EBS/builds/1/archive/remote_logs.log:4
/data/jenkins-jobs/ClusterHQ-flocker/jobs/invariant-error-FLOC-4452/jobs/_run_acceptance_on_GCE_CentOS_7_with_GCE/builds/1/archive/remote_logs.log:0
/data/jenkins-jobs/ClusterHQ-flocker/jobs/invariant-error-FLOC-4452/jobs/_run_acceptance_on_Rackspace_Ubuntu_Trusty_with_Cinder/builds/2/archive/remote_logs.log:0
/data/jenkins-jobs/ClusterHQ-flocker/jobs/invariant-error-FLOC-4452/jobs/_run_acceptance_on_GCE_Ubuntu_Trusty_with_GCE/builds/1/archive/remote_logs.log:0
You have mail in /var/spool/mail/root
[root@ip-172-31-0-243 ~]# find /data/jenkins-jobs/ClusterHQ-flocker/jobs/cancel-timer-FLOC-4450/ -type f -name 'remote_logs.log' | xargs grep --count "Either all or none of"
/data/jenkins-jobs/ClusterHQ-flocker/jobs/cancel-timer-FLOC-4450/jobs/run_acceptance_loopback_on_AWS_CentOS_7_for_flocker.acceptance/builds/1/archive/remote_logs.log:0
/data/jenkins-jobs/ClusterHQ-flocker/jobs/cancel-timer-FLOC-4450/jobs/_run_acceptance_on_AWS_Ubuntu_Trusty_with_EBS/builds/1/archive/remote_logs.log:16
/data/jenkins-jobs/ClusterHQ-flocker/jobs/cancel-timer-FLOC-4450/jobs/_run_acceptance_on_AWS_Ubuntu_Trusty_with_EBS/builds/2/archive/remote_logs.log:8
/data/jenkins-jobs/ClusterHQ-flocker/jobs/cancel-timer-FLOC-4450/jobs/_run_acceptance_on_Rackspace_CentOS_7_with_Cinder/builds/1/archive/remote_logs.log:14
/data/jenkins-jobs/ClusterHQ-flocker/jobs/cancel-timer-FLOC-4450/jobs/run_acceptance_loopback_on_AWS_Ubuntu_Trusty_for_flocker.acceptance/builds/1/archive/remote_logs.log:0
/data/jenkins-jobs/ClusterHQ-flocker/jobs/cancel-timer-FLOC-4450/jobs/__main_multijob/workspace/run_acceptance_loopback_on_AWS_CentOS_7_for_flocker.acceptance/remote_logs.log:0
/data/jenkins-jobs/ClusterHQ-flocker/jobs/cancel-timer-FLOC-4450/jobs/__main_multijob/workspace/run_acceptance_loopback_on_AWS_Ubuntu_Trusty_for_flocker.acceptance/remote_logs.log:0
/data/jenkins-jobs/ClusterHQ-flocker/jobs/cancel-timer-FLOC-4450/jobs/_run_acceptance_on_AWS_CentOS_7_with_EBS/builds/5/archive/remote_logs.log:12
/data/jenkins-jobs/ClusterHQ-flocker/jobs/cancel-timer-FLOC-4450/jobs/_run_acceptance_on_AWS_CentOS_7_with_EBS/builds/1/archive/remote_logs.log:16
/data/jenkins-jobs/ClusterHQ-flocker/jobs/cancel-timer-FLOC-4450/jobs/_run_acceptance_on_GCE_CentOS_7_with_GCE/builds/1/archive/remote_logs.log:8
/data/jenkins-jobs/ClusterHQ-flocker/jobs/cancel-timer-FLOC-4450/jobs/_run_acceptance_on_GCE_CentOS_7_with_GCE/builds/2/archive/remote_logs.log:8
/data/jenkins-jobs/ClusterHQ-flocker/jobs/cancel-timer-FLOC-4450/jobs/_run_acceptance_on_Rackspace_Ubuntu_Trusty_with_Cinder/builds/1/archive/remote_logs.log:22
/data/jenkins-jobs/ClusterHQ-flocker/jobs/cancel-timer-FLOC-4450/jobs/_run_acceptance_on_Rackspace_Ubuntu_Trusty_with_Cinder/builds/2/archive/remote_logs.log:16
/data/jenkins-jobs/ClusterHQ-flocker/jobs/cancel-timer-FLOC-4450/jobs/_run_acceptance_on_GCE_Ubuntu_Trusty_with_GCE/builds/1/archive/remote_logs.log:8
/data/jenkins-jobs/ClusterHQ-flocker/jobs/cancel-timer-FLOC-4450/jobs/_run_acceptance_on_GCE_Ubuntu_Trusty_with_GCE/builds/2/archive/remote_logs.log:8

http://ci-live.clusterhq.com:8080/job/ClusterHQ-flocker/job/invariant-error-FLOC-4452/view/cron/job/_run_acceptance_on_AWS_CentOS_7_with_EBS/ does still have the "Either all or none of" invariant error, but the stack trace suggests that those instances were not using the updated code....which is worrying and inexplicable right now.

cb133303-63a6-4542-acb5-4c85592e808e -> /1
2016-06-21 16:30:20.238646Z
  message_type: 'twisted:log'
  _HOSTNAME: 'ip-172-31-0-162.us-west-1.compute.internal'
  _PROCESS_NAME: 'flocker-container-agent.service'
  error: True
  message: 'Unhandled Error
         |  Traceback (most recent call last):
         |    File "/opt/flocker/lib/python2.7/site-packages/twisted/protocols/amp.py", line 1021, in _commandReceived
         |      deferred = self.dispatchCommand(box)
         |    File "/opt/flocker/lib/python2.7/site-packages/twisted/protocols/amp.py", line 1079, in dispatchCommand
         |      return maybeDeferred(responder, box)
         |    File "/opt/flocker/lib/python2.7/site-packages/twisted/internet/defer.py", line 150, in maybeDeferred
         |      result = f(*args, **kw)
         |    File "/opt/flocker/lib/python2.7/site-packages/twisted/protocols/amp.py", line 1166, in doit
         |      return maybeDeferred(aCallable, **kw).addCallback(
         |  --- <exception caught here> ---
         |    File "/opt/flocker/lib/python2.7/site-packages/twisted/internet/defer.py", line 150, in maybeDeferred
         |      result = f(*args, **kw)
         |    File "/opt/flocker/lib/python2.7/site-packages/flocker/control/_protocol.py", line 1152, in cluster_updated_diff
         |      self._current_state
         |    File "/opt/flocker/lib/python2.7/site-packages/flocker/control/_diffing.py", line 115, in apply
         |      obj = c.apply(obj)
         |    File "/opt/flocker/lib/python2.7/site-packages/flocker/control/_diffing.py", line 79, in apply
         |      return obj.transform(self.path, self.value)
         |    File "/opt/flocker/lib/python2.7/site-packages/pyrsistent/_pclass.py", line 136, in transform
         |      return transform(self, transformations)
         |    File "/opt/flocker/lib/python2.7/site-packages/pyrsistent/_transformations.py", line 65, in transform
         |      r = _do_to_path(r, path, command)
         |    File "/opt/flocker/lib/python2.7/site-packages/pyrsistent/_transformations.py", line 74, in _do_to_path
         |      return _update_structure(structure, kvs, path[1:], command)
         |    File "/opt/flocker/lib/python2.7/site-packages/pyrsistent/_transformations.py", line 110, in _update_structure
         |      result = _do_to_path(v, path, command)
         |    File "/opt/flocker/lib/python2.7/site-packages/pyrsistent/_transformations.py", line 74, in _do_to_path
         |      return _update_structure(structure, kvs, path[1:], command)
         |    File "/opt/flocker/lib/python2.7/site-packages/pyrsistent/_transformations.py", line 110, in _update_structure
         |      result = _do_to_path(v, path, command)
         |    File "/opt/flocker/lib/python2.7/site-packages/pyrsistent/_transformations.py", line 74, in _do_to_path
         |      return _update_structure(structure, kvs, path[1:], command)
         |    File "/opt/flocker/lib/python2.7/site-packages/pyrsistent/_transformations.py", line 113, in _update_structure
         |      return e.persistent()
         |    File "/opt/flocker/lib/python2.7/site-packages/pyrsistent/_precord.py", line 149, in persistent
         |      check_global_invariants(result, cls._precord_invariants)
         |    File "/opt/flocker/lib/python2.7/site-packages/pyrsistent/_field_common.py", line 22, in check_global_invariants
         |      raise InvariantException(error_codes, (), \'Global invariant failed\')
         |  pyrsistent._checked_types.InvariantException: Global invariant failed, invariant_errors=[Either all or none of set([\'paths\', \'manifestations\', \'devices\']) must be set.], missing_fields=[]
         |  '

(note that there's no sign of the new _EvolverProxy.commit in that stack trace.)

wallrj · 2016-06-22T14:06:32Z

There's another example of the invariant error in a different PClass

grep 'Global invariant' /data/jenkins-jobs/ClusterHQ-flocker/jobs/invariant-error-FLOC-4452/jobs/_run_acceptance_on_GCE_Ubuntu_Trusty_with_GCE/builds/1/archive/remote_logs.log
...

Unhandled Error
Traceback (most recent call last):
  File "/opt/flocker/local/lib/python2.7/site-packages/twisted/protocols/amp.py", line 1021, in _commandReceived
    deferred = self.dispatchCommand(box)
  File "/opt/flocker/local/lib/python2.7/site-packages/twisted/protocols/amp.py", line 1079, in dispatchCommand
    return maybeDeferred(responder, box)
  File "/opt/flocker/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 150, in maybeDeferred
    result = f(*args, **kw)
  File "/opt/flocker/local/lib/python2.7/site-packages/twisted/protocols/amp.py", line 1166, in doit
    return maybeDeferred(aCallable, **kw).addCallback(
--- <exception caught here> ---
  File "/opt/flocker/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 150, in maybeDeferred
    result = f(*args, **kw)
  File "/opt/flocker/local/lib/python2.7/site-packages/flocker/control/_protocol.py", line 1157, in cluster_updated_diff
    self._current_configuration
  File "/opt/flocker/local/lib/python2.7/site-packages/flocker/control/_diffing.py", line 179, in apply
    proxy = c.apply(proxy)
  File "/opt/flocker/local/lib/python2.7/site-packages/flocker/control/_diffing.py", line 81, in apply
    self.path[:-1], lambda o: o.set(self.path[-1], self.value)
  File "/opt/flocker/local/lib/python2.7/site-packages/flocker/control/_diffing.py", line 130, in transform
    self.commit()
  File "/opt/flocker/local/lib/python2.7/site-packages/flocker/control/_diffing.py", line 155, in commit
    target,
  File "/opt/flocker/local/lib/python2.7/site-packages/pyrsistent/_pclass.py", line 136, in transform
    return transform(self, transformations)
  File "/opt/flocker/local/lib/python2.7/site-packages/pyrsistent/_transformations.py", line 65, in transform
    r = _do_to_path(r, path, command)
  File "/opt/flocker/local/lib/python2.7/site-packages/pyrsistent/_transformations.py", line 74, in _do_to_path
    return _update_structure(structure, kvs, path[1:], command)
  File "/opt/flocker/local/lib/python2.7/site-packages/pyrsistent/_transformations.py", line 110, in _update_structure
    result = _do_to_path(v, path, command)
  File "/opt/flocker/local/lib/python2.7/site-packages/pyrsistent/_transformations.py", line 74, in _do_to_path
    return _update_structure(structure, kvs, path[1:], command)
  File "/opt/flocker/local/lib/python2.7/site-packages/pyrsistent/_transformations.py", line 110, in _update_structure
    result = _do_to_path(v, path, command)
  File "/opt/flocker/local/lib/python2.7/site-packages/pyrsistent/_transformations.py", line 74, in _do_to_path
    return _update_structure(structure, kvs, path[1:], command)
  File "/opt/flocker/local/lib/python2.7/site-packages/pyrsistent/_transformations.py", line 113, in _update_structure
    return e.persistent()
  File "/opt/flocker/local/lib/python2.7/site-packages/pyrsistent/_pclass.py", line 248, in persistent
    return self._pclass_evolver_original.__class__(**self._pclass_evolver_data)
  File "/opt/flocker/local/lib/python2.7/site-packages/flocker/control/_model.py", line 491, in __new__
    return PClass.__new__(cls, **kwargs)
  File "/opt/flocker/local/lib/python2.7/site-packages/pyrsistent/_pclass.py", line 69, in __new__
    check_global_invariants(result, cls._pclass_invariants)
  File "/opt/flocker/local/lib/python2.7/site-packages/pyrsistent/_field_common.py", line 22, in check_global_invariants
    raise InvariantException(error_codes, (), 'Global invariant failed')
pyrsistent._checked_types.InvariantException: Global invariant failed, invariant_errors=[Application(memory_limit=None, name=u'flocker_acceptance_obsolete_test_containers_ContainerAPITests_test_move_container_with_dataset-975570', links=LinkPSet([]), environment=pmap({}), command_line=UnicodePVector([u'python', u'-c', u'"""\nHTTP server that writes data to a specified file on POST, or reads and\nreturns data from a specified file on GET.\n"""\n\nfrom sys import argv\n\ntry:\n    from urlparse import parse_qs\nexcept ImportError:\n    from cgi import parse_qs\n\nfrom BaseHTTPServer import BaseHTTPRequestHandler, HTTPServer\n\n\nclass Handler(BaseHTTPRequestHandler):\n    def do_POST(s):\n        length = int(s.headers[\'content-length\'])\n        postvars = parse_qs(\n            s.rfile.read(length),\n            keep_blank_values=1\n        )\n        try:\n            with open(argv[1] + \'/test\', "w") as f:\n                f.write(postvars["data"][0])\n        except Exception as e:\n            s.wfile.write(str(e.__class__) + ": " + str(e))\n        s.send_response(200)\n        s.end_headers()\n        s.wfile.write(b"ok")\n        s.wfile.close()\n\n    def do_GET(s):\n        s.send_response(200)\n        s.end_headers()\n        if len(argv) > 1:\n            try:\n                with open(argv[1] + \'/test\', "r") as f:\n                    data = f.read()\n            except Exception as e:\n                s.wfile.write(str(e.__class__) + ": " + str(e))\n            else:\n                s.wfile.write(data)\n        else:\n                s.wfile.write(b"ok")\n        s.wfile.close()\n\nhttpd = HTTPServer((b"0.0.0.0", 8080), Handler)\nhttpd.serve_forever()\n', u'/data']), image=DockerImage(tag=u'2.7-slim', repository=u'python'), restart_policy=RestartNever(), volume=AttachedVolume(mountpoint=FilePath('/data'), manifestation=Manifestation(primary=True, dataset=Dataset(deleted=False, dataset_id=u'4bda5cd0-9377-45e5-ad53-441f12db38ba', maximum_size=1073741824, metadata=pmap({})))), running=True, ports=PortPSet([Port(internal_port=8080, external_port=8080)]), cpu_shares=None) manifestation is not on node], missing_fields=[]

I need to look into that one.

…ns in such a way as to trigger an invariant error

…C-4452

…ariant-error-FLOC-4452

wallrj · 2016-06-24T16:48:21Z

Well I don't fully understand it, but there still seem to be invariant failures in some of our acceptance test environments:

[centos@ip-172-31-0-243 ~]$ find /data/jenkins-jobs/ClusterHQ-flocker/jobs/invariant-error-FLOC-4452 -type f -name 'remote_logs.log' | xargs grep --count "invariant" | sort
...
__main_multijob/workspace/run_acceptance_loopback_on_AWS_CentOS_7_for_flocker.acceptance/remote_logs.log:0
__main_multijob/workspace/run_acceptance_loopback_on_AWS_Ubuntu_Trusty_for_flocker.acceptance/remote_logs.log:0
run_acceptance_loopback_on_AWS_CentOS_7_for_flocker.acceptance/builds/1/archive/remote_logs.log:0
run_acceptance_loopback_on_AWS_CentOS_7_for_flocker.acceptance/builds/2/archive/remote_logs.log:0
run_acceptance_loopback_on_AWS_CentOS_7_for_flocker.acceptance/builds/3/archive/remote_logs.log:0
run_acceptance_loopback_on_AWS_CentOS_7_for_flocker.acceptance/builds/4/archive/remote_logs.log:0
run_acceptance_loopback_on_AWS_CentOS_7_for_flocker.acceptance/builds/5/archive/remote_logs.log:0
run_acceptance_loopback_on_AWS_CentOS_7_for_flocker.acceptance/builds/6/archive/remote_logs.log:0
run_acceptance_loopback_on_AWS_CentOS_7_for_flocker.acceptance/builds/7/archive/remote_logs.log:0
run_acceptance_loopback_on_AWS_Ubuntu_Trusty_for_flocker.acceptance/builds/1/archive/remote_logs.log:0
run_acceptance_loopback_on_AWS_Ubuntu_Trusty_for_flocker.acceptance/builds/2/archive/remote_logs.log:0
run_acceptance_loopback_on_AWS_Ubuntu_Trusty_for_flocker.acceptance/builds/3/archive/remote_logs.log:0
run_acceptance_loopback_on_AWS_Ubuntu_Trusty_for_flocker.acceptance/builds/4/archive/remote_logs.log:0
run_acceptance_loopback_on_AWS_Ubuntu_Trusty_for_flocker.acceptance/builds/5/archive/remote_logs.log:0
run_acceptance_loopback_on_AWS_Ubuntu_Trusty_for_flocker.acceptance/builds/6/archive/remote_logs.log:0
run_acceptance_loopback_on_AWS_Ubuntu_Trusty_for_flocker.acceptance/builds/7/archive/remote_logs.log:0
_run_acceptance_on_AWS_CentOS_7_with_EBS/builds/1/archive/remote_logs.log:12
_run_acceptance_on_AWS_CentOS_7_with_EBS/builds/2/archive/remote_logs.log:12
_run_acceptance_on_AWS_CentOS_7_with_EBS/builds/3/archive/remote_logs.log:12
_run_acceptance_on_AWS_CentOS_7_with_EBS/builds/4/archive/remote_logs.log:4
_run_acceptance_on_AWS_CentOS_7_with_EBS/builds/5/archive/remote_logs.log:4
_run_acceptance_on_AWS_CentOS_7_with_EBS/builds/6/archive/remote_logs.log:4
_run_acceptance_on_AWS_Ubuntu_Trusty_with_EBS/builds/1/archive/remote_logs.log:8
_run_acceptance_on_AWS_Ubuntu_Trusty_with_EBS/builds/2/archive/remote_logs.log:8
_run_acceptance_on_AWS_Ubuntu_Trusty_with_EBS/builds/4/archive/remote_logs.log:0
_run_acceptance_on_AWS_Ubuntu_Trusty_with_EBS/builds/5/archive/remote_logs.log:0
_run_acceptance_on_AWS_Ubuntu_Trusty_with_EBS/builds/6/archive/remote_logs.log:0
_run_acceptance_on_GCE_CentOS_7_with_GCE/builds/1/archive/remote_logs.log:0
_run_acceptance_on_GCE_CentOS_7_with_GCE/builds/2/archive/remote_logs.log:0
_run_acceptance_on_GCE_CentOS_7_with_GCE/builds/3/archive/remote_logs.log:0
_run_acceptance_on_GCE_CentOS_7_with_GCE/builds/4/archive/remote_logs.log:0
_run_acceptance_on_GCE_CentOS_7_with_GCE/builds/6/archive/remote_logs.log:0
_run_acceptance_on_GCE_CentOS_7_with_GCE/builds/7/archive/remote_logs.log:0
_run_acceptance_on_GCE_Ubuntu_Trusty_with_GCE/builds/1/archive/remote_logs.log:8
_run_acceptance_on_GCE_Ubuntu_Trusty_with_GCE/builds/2/archive/remote_logs.log:8
_run_acceptance_on_GCE_Ubuntu_Trusty_with_GCE/builds/4/archive/remote_logs.log:0
_run_acceptance_on_GCE_Ubuntu_Trusty_with_GCE/builds/5/archive/remote_logs.log:0
_run_acceptance_on_Rackspace_CentOS_7_with_Cinder/builds/1/archive/remote_logs.log:14
_run_acceptance_on_Rackspace_CentOS_7_with_Cinder/builds/2/archive/remote_logs.log:12
_run_acceptance_on_Rackspace_CentOS_7_with_Cinder/builds/3/archive/remote_logs.log:12
_run_acceptance_on_Rackspace_CentOS_7_with_Cinder/builds/4/archive/remote_logs.log:4
_run_acceptance_on_Rackspace_CentOS_7_with_Cinder/builds/5/archive/remote_logs.log:8
_run_acceptance_on_Rackspace_CentOS_7_with_Cinder/builds/6/archive/remote_logs.log:4
_run_acceptance_on_Rackspace_Ubuntu_Trusty_with_Cinder/builds/2/archive/remote_logs.log:8
_run_acceptance_on_Rackspace_Ubuntu_Trusty_with_Cinder/builds/3/archive/remote_logs.log:8
_run_acceptance_on_Rackspace_Ubuntu_Trusty_with_Cinder/builds/4/archive/remote_logs.log:8
_run_acceptance_on_Rackspace_Ubuntu_Trusty_with_Cinder/builds/5/archive/remote_logs.log:0
_run_acceptance_on_Rackspace_Ubuntu_Trusty_with_Cinder/builds/6/archive/remote_logs.log:0

And judging by the stack trace it seems that the acceptance test runner is installing the wrong package version

http://ci-live.clusterhq.com:8080/job/ClusterHQ-flocker/job/invariant-error-FLOC-4452/job/_run_acceptance_on_AWS_CentOS_7_with_EBS/6/artifact/remote_logs.log/*view*/

1b7c5314-8a82-4de8-b240-a4f5ff3d497a -> /1
2016-06-24 15:13:52.062164Z
  message_type: 'twisted:log'
  _HOSTNAME: 'ip-172-31-1-149.us-west-1.compute.internal'
  _PROCESS_NAME: 'flocker-container-agent.service'
  error: True
  message: 'Unhandled Error
         |  Traceback (most recent call last):
         |    File "/opt/flocker/lib/python2.7/site-packages/twisted/protocols/amp.py", line 1021, in _commandReceived
         |      deferred = self.dispatchCommand(box)
         |    File "/opt/flocker/lib/python2.7/site-packages/twisted/protocols/amp.py", line 1079, in dispatchCommand
         |      return maybeDeferred(responder, box)
         |    File "/opt/flocker/lib/python2.7/site-packages/twisted/internet/defer.py", line 150, in maybeDeferred
         |      result = f(*args, **kw)
         |    File "/opt/flocker/lib/python2.7/site-packages/twisted/protocols/amp.py", line 1166, in doit
         |      return maybeDeferred(aCallable, **kw).addCallback(
         |  --- <exception caught here> ---
         |    File "/opt/flocker/lib/python2.7/site-packages/twisted/internet/defer.py", line 150, in maybeDeferred
         |      result = f(*args, **kw)
         |    File "/opt/flocker/lib/python2.7/site-packages/flocker/control/_protocol.py", line 1152, in cluster_updated_diff
         |      self._current_state
         |    File "/opt/flocker/lib/python2.7/site-packages/flocker/control/_diffing.py", line 115, in apply
         |      obj = c.apply(obj)
         |    File "/opt/flocker/lib/python2.7/site-packages/flocker/control/_diffing.py", line 79, in apply
         |      return obj.transform(self.path, self.value)
         |    File "/opt/flocker/lib/python2.7/site-packages/pyrsistent/_pclass.py", line 136, in transform
         |      return transform(self, transformations)
         |    File "/opt/flocker/lib/python2.7/site-packages/pyrsistent/_transformations.py", line 65, in transform
         |      r = _do_to_path(r, path, command)
         |    File "/opt/flocker/lib/python2.7/site-packages/pyrsistent/_transformations.py", line 74, in _do_to_path
         |      return _update_structure(structure, kvs, path[1:], command)
         |    File "/opt/flocker/lib/python2.7/site-packages/pyrsistent/_transformations.py", line 110, in _update_structure
         |      result = _do_to_path(v, path, command)
         |    File "/opt/flocker/lib/python2.7/site-packages/pyrsistent/_transformations.py", line 74, in _do_to_path
         |      return _update_structure(structure, kvs, path[1:], command)
         |    File "/opt/flocker/lib/python2.7/site-packages/pyrsistent/_transformations.py", line 110, in _update_structure
         |      result = _do_to_path(v, path, command)
         |    File "/opt/flocker/lib/python2.7/site-packages/pyrsistent/_transformations.py", line 74, in _do_to_path
         |      return _update_structure(structure, kvs, path[1:], command)
         |    File "/opt/flocker/lib/python2.7/site-packages/pyrsistent/_transformations.py", line 113, in _update_structure
         |      return e.persistent()
         |    File "/opt/flocker/lib/python2.7/site-packages/pyrsistent/_precord.py", line 149, in persistent
         |      check_global_invariants(result, cls._precord_invariants)
         |    File "/opt/flocker/lib/python2.7/site-packages/pyrsistent/_field_common.py", line 22, in check_global_invariants
         |      raise InvariantException(error_codes, (), \'Global invariant failed\')
         |  pyrsistent._checked_types.InvariantException: Global invariant failed, invariant_errors=[Either all or none of set([\'paths\', \'manifestations\', \'devices\']) must be set.], missing_fields=[]
         |  '

NB no sign of the new _EvolverProxy.commit function.

Which is a bit worrying!

wallrj · 2016-06-24T16:53:15Z

Ah! I know why. I think this is our test_upgrade test striking again

[FLOC-4450] Cancel ping timeout action when AMP connection is lost #2836 (comment)

The errors are occurring during the short period while the cluster is running the downgraded version of flocker. (1.13.0)

…C-4452

exarkun · 2016-06-24T18:57:19Z

flocker/control/testtools.py

@@ -341,7 +371,7 @@ def deployment_strategy(
        draw(
            lease_strategy(
                dataset_id=st.just(dataset_id),
-                node_uuid=st.just(node_uuid)
+                node_id=st.just(node_uuid)


Was this simply broken before? It looks like yes but it went unnoticed because with no "stateful" applications there were never any leases.

Yep, it was broken and the node_strategy is also broken in the way it supplies UUID(...) versus unicode(UUID(...)) to Dataset.dataset_id....I need to investigate that.
I just added the new stateful_application parameter to avoid triggering the bug in the existing tests.

exarkun · 2016-06-28T16:48:52Z

Thanks Richard. Left my comments inline. Looks good for the most part though I wasn't able to follow all of the implementation (left comments where I was lost). The test suite looks good though so I'm happy to have you address the noted points to your satisfaction and then merge - apart from the issue with the upgrade test? What's the story there? Does that test just need to be made more robust as a separate unit of work? Thanks again.

…C-4452

…r the special case where the root object is replaced with a new, unrelated object.

…thod.

…rm is not _IEvolvable.

…parate proxies for set and record types

wallrj · 2016-07-11T09:48:57Z

Ignoring:

Flaky: http://ci-live.clusterhq.com:8080/job/ClusterHQ-flocker/job/invariant-error-FLOC-4452/job/run_trial_for_ebs_storage_driver_on_CentOS_7_flocker.node.agents.functional/14/ (although I haven't seen this fail very often.)
Flaky: http://ci-live.clusterhq.com:8080/job/ClusterHQ-flocker/job/invariant-error-FLOC-4452/job/run_trial_on_AWS_Ubuntu_Xenial_flocker.control/5/ (common hypothesis FailedHealthCheck error)
Invariant errors in the acceptance test remote_logs only occur during the test_upgrade tests while the 1.13 packages are installed.

avg-I · 2016-07-11T16:31:18Z

flocker/control/_diffing.py

+
+class _IRecordType(Interface):
+    """
+    The operations that can be performed when transforming a ``PSet`` object.


s/PSet/PMap/ probably

avg-I · 2016-07-11T17:10:03Z

@wallrj Richard, sorry it took me this long to review this change and to produce just a few minor comments. It took me quite a while to understand all the changed and related code.

I didn't realize that you merged the PR some hours ago, so this is a belated LGTM from me.

wallrj added 2 commits June 20, 2016 11:40

A failing test

ff5f016

Add a proxy to collect and apply adjacent set operations to an evolve…

835627a

…r so as not to trigger the invariant check until all attributes have been set.

wallrj added 12 commits June 22, 2016 17:58

Add logging so I can see what diffs are failing.

2e00493

Tests for the diffs which modify a Node applications and manifestatio…

962daa5

…ns in such a way as to trigger an invariant error

A more flexible approach....not quite complete

adcf738

Work in progress

0bec10c

Handle add and remove of Pyrsistent objects

a9ce128

Handle the freezing of map like objects and sets differently.

5502b7b

remove unused import

ae06a41

oops, I removed too much

19d0d86

When removing an item attempt to remove it from the evolver too.

ac9e6cb

Merge remote-tracking branch 'origin/master' into invariant-error-FLO…

68018e1

…C-4452

docstrings

a982d77

Merge remote-tracking branch 'origin/cancel-timer-FLOC-4450' into inv…

beac6b7

…ariant-error-FLOC-4452

Merge remote-tracking branch 'origin/master' into invariant-error-FLO…

5352c86

…C-4452

wallrj changed the title ~~[FLOC-4452] Collect and apply adjacent set operations to an evolver so as not to trigger the invariant checks until all attributes have been set~~ [FLOC-4452] Delay invariant checks until all diff operations have been applied Jun 24, 2016

exarkun reviewed Jun 24, 2016
View reviewed changes

wallrj added 3 commits July 5, 2016 11:17

A clearer set up for the expected test object.

c4e2815

Use twisted MonkeyPatcher directly....maybe overkill?!

f8cfdd6

Explain the purpose of the DiffTestObjInvariant class.

1e512d5

wallrj mentioned this pull request Jul 7, 2016

[DO NOT REVIEW OR MERGE] Move pyrsistent serialization and hashing helpers to a new module. #2858

Open

Merge remote-tracking branch 'origin/master' into invariant-error-FLO…

b0549a4

…C-4452

wallrj mentioned this pull request Jul 7, 2016

Allow multiple transform operations to be performed before checking invariants tobgu/pyrsistent#89

Open

A note about why we're importing a private function from pyrsistent

ff7730f

wallrj mentioned this pull request Jul 7, 2016

[FLOC-4421] Diffing of objects #2806

Merged

wallrj added 6 commits July 7, 2016 16:00

Add a attribute to the _Set operation and a new _Replace operation fo…

d452558

…r the special case where the root object is replaced with a new, unrelated object.

Use Zope Interface to mark classes that support Pyrsistent evolver me…

a24841f

…thod.

Give clearer errors when the object supplied to _EvolverProxy.transfo…

14e2af6

…rm is not _IEvolvable.

docstring

574cf92

Refactor things...a lot...to get rid of hasattr checks and to have se…

7ec0238

…parate proxies for set and record types

Add missing docstrings

7abd8a8

wallrj merged commit 9f9b2f7 into master Jul 11, 2016

wallrj deleted the invariant-error-FLOC-4452 branch July 11, 2016 09:51

avg-I reviewed Jul 11, 2016
View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[FLOC-4452] Delay invariant checks until all diff operations have been applied #2839

[FLOC-4452] Delay invariant checks until all diff operations have been applied #2839

wallrj commented Jun 21, 2016 •

edited

Loading

wallrj commented Jun 22, 2016

wallrj commented Jun 22, 2016

wallrj commented Jun 24, 2016

wallrj commented Jun 24, 2016

exarkun Jun 24, 2016

wallrj Jun 24, 2016

exarkun commented Jun 28, 2016

wallrj commented Jul 11, 2016

avg-I Jul 11, 2016

avg-I commented Jul 11, 2016

[FLOC-4452] Delay invariant checks until all diff operations have been applied #2839

[FLOC-4452] Delay invariant checks until all diff operations have been applied #2839

Conversation

wallrj commented Jun 21, 2016 • edited Loading

wallrj commented Jun 22, 2016

wallrj commented Jun 22, 2016

wallrj commented Jun 24, 2016

wallrj commented Jun 24, 2016

exarkun Jun 24, 2016

Choose a reason for hiding this comment

wallrj Jun 24, 2016

Choose a reason for hiding this comment

exarkun commented Jun 28, 2016

wallrj commented Jul 11, 2016

avg-I Jul 11, 2016

Choose a reason for hiding this comment

avg-I commented Jul 11, 2016

wallrj commented Jun 21, 2016 •

edited

Loading