After updating to 0.95.1 I now get "INCOMPATIBLE CLUSTER" when doing 'addnode' #371

Closed
CarlosBorroto opened this Issue Feb 24, 2014 · 10 comments

Comments

Projects
None yet
2 participants

Log:

$ starcluster addnode science-cluster
StarCluster - (http://star.mit.edu/cluster) (v. 0.95.1)
Software Tools for Academics and Researchers (STAR)
Please submit bug reports to starcluster@mit.edu

!!! ERROR - INCOMPATIBLE CLUSTER: science-cluster
!!! ERROR -
!!! ERROR - The cluster 'science-cluster' is not compatible with
!!! ERROR - StarCluster 0.95.1. Possible reasons are:
!!! ERROR -
!!! ERROR - 1. The '@sc-science-cluster' group was created using an
!!! ERROR - incompatible version of StarCluster (stable or development).
!!! ERROR -
!!! ERROR - 2. The '@sc-science-cluster' group was manually created
!!! ERROR - outside of StarCluster.
!!! ERROR -
!!! ERROR - 3. One of the nodes belonging to '@sc-science-cluster' was
!!! ERROR - manually created outside of StarCluster.
!!! ERROR -
!!! ERROR - 4. StarCluster was interrupted very early on when first
!!! ERROR - creating the cluster's security group.
!!! ERROR -
!!! ERROR - In any case 'science-cluster' and its nodes cannot be used
!!! ERROR - with this version of StarCluster (0.95.1).
!!! ERROR -
!!! ERROR - The cluster 'science-cluster' currently has 2 active nodes.
!!! ERROR -
!!! ERROR - Please terminate the cluster using:
!!! ERROR -
!!! ERROR -     $ starcluster terminate --force science-cluster
Owner

jtriley commented Feb 25, 2014

@cjav Usually you get a warning about a version mismatch between the cluster and client but in this case there was an error loading the cluster metadata. Can you check your $HOME/.starcluster/logs/debug.log for the error and paste the exception (search for IncompatibleCluster)?

For now you can revert back to 0.95 with:

$ pip install StarCluster==0.95
Owner

jtriley commented Feb 25, 2014

Also, what version did you use to create the cluster?

@jtriley I'm sure I used 0.95.1. I even terminate the cluster and recreated it just to be sure.

This is the exception I see in the log:

Traceback (most recent call last):
  File "/Users/cborroto/.virtualenvs/sciencemodule-dev/lib/python2.7/site-packages/starcluster/cli.py", line 274, in main
    sc.execute(args)
  File "/Users/cborroto/.virtualenvs/sciencemodule-dev/lib/python2.7/site-packages/starcluster/commands/addnode.py", line 129, in execute
    no_create=self.opts.no_create)
  File "/Users/cborroto/.virtualenvs/sciencemodule-dev/lib/python2.7/site-packages/starcluster/cluster.py", line 185, in add_nodes
    cl = self.get_cluster(cluster_name)
  File "/Users/cborroto/.virtualenvs/sciencemodule-dev/lib/python2.7/site-packages/starcluster/cluster.py", line 67, in get_cluster
    load_volumes=load_volumes)
  File "/Users/cborroto/.virtualenvs/sciencemodule-dev/lib/python2.7/site-packages/starcluster/cluster.py", line 614, in load_receipt
    raise exception.IncompatibleCluster(self.cluster_group)
IncompatibleCluster: INCOMPATIBLE CLUSTER: science-cluster

Thanks for looking in to this!
Carlos

Also in the log and I think related:

Traceback (most recent call last):
  File "/Users/cborroto/.virtualenvs/sciencemodule-dev/lib/python2.7/site-packages/starcluster/cluster.py", line 591, in load_receipt
    self.update(self._get_settings_from_tags())
  File "/Users/cborroto/.virtualenvs/sciencemodule-dev/lib/python2.7/site-packages/starcluster/cluster.py", line 721, in _get_settings_from_tags
    cluster.update(self._load_chunked_tags(sg, static.CORE_TAG))
  File "/Users/cborroto/.virtualenvs/sciencemodule-dev/lib/python2.7/site-packages/starcluster/cluster.py", line 715, in _load_chunked_tags
    return utils.decode_uncompress_load(chunks, use_json=True)
  File "/Users/cborroto/.virtualenvs/sciencemodule-dev/lib/python2.7/site-packages/starcluster/utils.py", line 611, in decode_uncompress_load
    return serializer.loads(zlib.decompress(string.decode('base64')))
error: Error -3 while decompressing data: incorrect header check
2014-02-24 19:37:41,008 PID: 18912 cli.py:300 - ERROR - INCOMPATIBLE CLUSTER: science-cluster
Owner

jtriley commented Feb 25, 2014

@cjav Just to clarify - this happens with any cluster you create with 0.95.1? Would you mind joining #starcluster on IRC (freenode) some time today and we can try to debug.

Owner

jtriley commented Feb 25, 2014

@cjav Ugh, just found the bug. Going to put out another patch release today. Thanks for reporting!

jtriley closed this in d161cd4 Feb 25, 2014

Owner

jtriley commented Feb 25, 2014

@cjav Should be fixed now in 0.95.2 (released on PyPI)

Thanks! I'll be happy to join you on IRC if you still need help with anything.

Owner

jtriley commented Feb 25, 2014

@cjav Can you confirm this fixes the issue for you? Thanks!

Working nicely. Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment