Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Cluster has extra nodes post-upgrade #4260

Closed
otoolep opened this issue Sep 28, 2015 · 8 comments
Closed

Cluster has extra nodes post-upgrade #4260

otoolep opened this issue Sep 28, 2015 · 8 comments
Assignees
@otoolep
Milestone
0.9.5

Comments

@otoolep
Copy link
Contributor

otoolep commented Sep 28, 2015

Started with #4212 (comment)

A cluster running 0.9.2 after upgrade to a 0.9.4, has extra nodes in the system.

> show servers
id  cluster_addr            raft
1   db2.domain.com:8088 true
2   db1.domain.com:8088 true
3   db3.domain.com:8088 true
4   10.0.9.39:8088      false
5   10.0.11.23:8088     false
6   10.0.12.32:8088     false
@otoolep otoolep added this to the 0.9.5 milestone Sep 28, 2015
@otoolep
Copy link
Contributor Author

otoolep commented Sep 28, 2015

@jwilder @corylanou

@otoolep
Copy link
Contributor Author

otoolep commented Sep 28, 2015

cc @ecables

@otoolep otoolep self-assigned this Sep 28, 2015
@otoolep otoolep mentioned this issue Sep 28, 2015
Merged
6 tasks
@otoolep
Copy link
Contributor Author

otoolep commented Sep 28, 2015

@ecables -- can we get another look at SHOW SHARDS?

@ecables
Copy link

ecables commented Sep 29, 2015

Do you just want the output for _internal?

db1:

name: _internal
---------------
id  start_time      end_time        expiry_time     owners
61  2015-09-24T00:00:00Z    2015-09-25T00:00:00Z    2015-10-02T00:00:00Z    3
62  2015-09-24T00:00:00Z    2015-09-25T00:00:00Z    2015-10-02T00:00:00Z    1
63  2015-09-24T00:00:00Z    2015-09-25T00:00:00Z    2015-10-02T00:00:00Z    2
65  2015-09-25T00:00:00Z    2015-09-26T00:00:00Z    2015-10-03T00:00:00Z    1,2,3
67  2015-09-26T00:00:00Z    2015-09-27T00:00:00Z    2015-10-04T00:00:00Z    3,1,2
69  2015-09-27T00:00:00Z    2015-09-28T00:00:00Z    2015-10-05T00:00:00Z    2,3,1
72  2015-09-28T00:00:00Z    2015-09-29T00:00:00Z    2015-10-06T00:00:00Z    2,3,1
74  2015-09-29T00:00:00Z    2015-09-30T00:00:00Z    2015-10-07T00:00:00Z    3,1,2

db2:

name: _internal
---------------
id  start_time      end_time        expiry_time     owners
93  2015-09-24T00:00:00Z    2015-09-25T00:00:00Z    2015-10-02T00:00:00Z    3
94  2015-09-24T00:00:00Z    2015-09-25T00:00:00Z    2015-10-02T00:00:00Z    1
95  2015-09-24T00:00:00Z    2015-09-25T00:00:00Z    2015-10-02T00:00:00Z    2
96  2015-09-25T00:00:00Z    2015-09-26T00:00:00Z    2015-10-03T00:00:00Z    1,2,3
97  2015-09-26T00:00:00Z    2015-09-27T00:00:00Z    2015-10-04T00:00:00Z    3,1,2
98  2015-09-27T00:00:00Z    2015-09-28T00:00:00Z    2015-10-05T00:00:00Z    2,3,1
99  2015-09-28T00:00:00Z    2015-09-29T00:00:00Z    2015-10-06T00:00:00Z    2,3,1
101 2015-09-29T00:00:00Z    2015-09-30T00:00:00Z    2015-10-07T00:00:00Z    3,1,2

Interestingly, only one node still has the dupe servers; the other two nodes only show three servers in the SHOW SERVERS output; here's that output from db3:

name: _internal
---------------
id  start_time      end_time        expiry_time     owners
113 2015-09-24T00:00:00Z    2015-09-25T00:00:00Z    2015-10-02T00:00:00Z    6
114 2015-09-24T00:00:00Z    2015-09-25T00:00:00Z    2015-10-02T00:00:00Z    1
115 2015-09-24T00:00:00Z    2015-09-25T00:00:00Z    2015-10-02T00:00:00Z    2
116 2015-09-24T00:00:00Z    2015-09-25T00:00:00Z    2015-10-02T00:00:00Z    3
117 2015-09-24T00:00:00Z    2015-09-25T00:00:00Z    2015-10-02T00:00:00Z    4
118 2015-09-24T00:00:00Z    2015-09-25T00:00:00Z    2015-10-02T00:00:00Z    5
119 2015-09-25T00:00:00Z    2015-09-26T00:00:00Z    2015-10-03T00:00:00Z    4,5,6
120 2015-09-25T00:00:00Z    2015-09-26T00:00:00Z    2015-10-03T00:00:00Z    1,2,3
121 2015-09-26T00:00:00Z    2015-09-27T00:00:00Z    2015-10-04T00:00:00Z    6,1,2
122 2015-09-26T00:00:00Z    2015-09-27T00:00:00Z    2015-10-04T00:00:00Z    3,4,5
123 2015-09-27T00:00:00Z    2015-09-28T00:00:00Z    2015-10-05T00:00:00Z    2,3,4
124 2015-09-27T00:00:00Z    2015-09-28T00:00:00Z    2015-10-05T00:00:00Z    5,6,1
125 2015-09-28T00:00:00Z    2015-09-29T00:00:00Z    2015-10-06T00:00:00Z    5,6,1
126 2015-09-28T00:00:00Z    2015-09-29T00:00:00Z    2015-10-06T00:00:00Z    2,3,4
129 2015-09-29T00:00:00Z    2015-09-30T00:00:00Z    2015-10-07T00:00:00Z    6,1,2
130 2015-09-29T00:00:00Z    2015-09-30T00:00:00Z    2015-10-07T00:00:00Z    3,4,5

> show servers
id  cluster_addr            raft
1   db2.domain.com:8088 true
2   db1.domain.com:8088 true
3   db3.domain.com:8088 true
4   10.10.9.19:8088     false
5   10.10.11.23:8088        false
6   10.10.11.22:8088        false

@otoolep
Copy link
Contributor Author

otoolep commented Sep 29, 2015

Thanks @ecables -- no, I want it for all databases.

@otoolep
Copy link
Contributor Author

otoolep commented Sep 29, 2015

I want to see the ownership of every single shard on your system.

@otoolep
Copy link
Contributor Author

otoolep commented Sep 29, 2015

A simple restart of a 0.9.4 cluster, changing the hostnames at the command line, does not result in this issue. This is not surprising, but I wanted to confirm this.

@otoolep
Copy link
Contributor Author

otoolep commented Oct 28, 2015

We believe this issue was caused due to an unsupported upgrade path, and don't plan to fix it going forward as it should be a non-issue now.

@otoolep otoolep closed this as completed Oct 28, 2015
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@otoolep otoolep

Labels
None yet
Projects
None yet
Milestone
0.9.5
Development

No branches or pull requests
2 participants
@otoolep @ecables