-
Notifications
You must be signed in to change notification settings - Fork 3.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[0.9.5-nightly-952f1d5] not all nodes have measurements in _internal database #4212
Comments
That's probably because the first shard groups of the _internal database are not fully replicated. Show us the output of 'SHOW SHARDS'. |
Requested output below:
|
Yeah -- you need to bump the replication factor of your Make sense? |
Okay, I altered the retention policy for
The output of
Do I need to restart Influx for this change to take effect? |
@ecables -- what will happen is this change will take effect when the next set of shards -- called "shard groups" are created. That group will be created a few minutes before the current shards expire. |
Well, I ended up restarting the nodes (db1, db2, and db3), thinking that might trigger this, and on two of the nodes I'm seeing owners with ID 4, 5, and 6 for _internal, despite there only being 3 nodes in the cluster. Looking at db1:
In the above output, servers 4-6 are the unresolved versions of 1-3. db2 shows only 3:
db3 also shows 6 servers as well:
Thoughts? We've had weird issues like this in the past, where 3 new unresolved versions of the existing servers show up, and then create errors in the logs when writes to unknown nodes occur. |
@jwilder -- any thoughts? |
To be clear @ecables -- you are not running 6-node clusters, right? When the name resolution didn't happen, everything got doubled? |
No, we only have 3 nodes, and the 6 nodes shown merely represent the hostname representation and the unresolved representation (IP address). |
Yeah, that's what I assumed, but just wanted to be sure. I've CC @jwilder who most recently worked on the clustering code. |
@ecables When you restart the nodes, is the same |
No changes were made to the configuration during the restart; before starting the cluster during the upgrade from 0.9.4.1 -> 0.9.5-nightly I ensured that the |
I also want to contribute that this cluster has not been wiped since around 0.9.2. At one point we were using the "peers = []" config directory which was specified via IP addresses at the time. Around the 0.9.3 train the peers.json config file changed from permitting IPs to permitting hostnames in the config. peers.json was then changed to use hostnames rather than IPs. I'm assuming this MAY be stale information because of that. If so is there a way to gracefully yank these out of the cluster? The other funny thing to note here is that these nodes come and go as we restart the cluster. |
Right now we do not have configuration support for removing nodes from the cluster -- the We may be able to do some work here, to help, but this could be awkward. Your cluster is not a fully healthy state, and it may be due to the upgrade from 0.9.2 to the latest code. |
I am going to close this issue for now, as I believe the system is behaving as expected. |
With regards to the replication of the internal stats, that is. |
After some recent reports, this does not appear to be solved. |
It may be related to the extra nodes on the cluster, which will hopefully be cleaned up by #4310 |
Patch ready for merging, 4 green CI builds. https://circleci.com/gh/influxdb/influxdb/tree/drop_node_non_raft |
This is an issue for an older version of InfluxDB and clustering is no longer supported in the open source version. I'm going to close this issue. Thank you. |
Just upgraded to 0.9.5-nightly-952f1d5, and I noticed that only one of the nodes has measurements in the _internal database.
db1:
db2:
db3:
The text was updated successfully, but these errors were encountered: