Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Restarting influxdb overwrites default shardspace config #886

Closed
XANi opened this issue Aug 29, 2014 · 6 comments
Closed

Restarting influxdb overwrites default shardspace config #886

XANi opened this issue Aug 29, 2014 · 6 comments

Comments

@XANi
Copy link

XANi commented Aug 29, 2014

If graphite plugin is active while creating DB it messes up shardspaces(v0.8.0).

My shardspace config: https://gist.github.com/XANi/b22ad3d1174bdc851281

Example: 2 node cluster; graphite plugin set up to write into "stats" database

curl -X POST "http://localhost:8086/cluster/database_configs/stat?u=root&p=root" -d @/tmp/shardspace.json
curl -X POST "http://localhost:8086/cluster/database_configs/stats?u=root&p=root" -d @/tmp/shardspace.json

results in

ss1

after I change to write to 'stat' database, restart nodes and start writing, it okay... but after restarting one node other database's shardspace get messed up too:
ss2

So it looks to me like:

  • spamming writes to unexisting db at it's creation can create "default" shardspace (probably if write comes between db create and adding its shardspaces)
  • on restart whatever current default shardspace is, it gets some of it's parameters overwritten
@XANi XANi changed the title Graphite plugin breaks pre-configured shard spaces Restarting influxdb overwrites default shardspace config Aug 29, 2014
@jvshahid jvshahid added this to the Next release milestone Aug 29, 2014
@jvshahid jvshahid added the bug label Aug 29, 2014
@mradamczyk
Copy link

Same happens to me even without restarting influx and without having graphite.
https://groups.google.com/forum/?utm_medium=email&utm_source=footer#!msg/influxdb/c0CHA4ehcFM/7KBEv-PIt9EJ

@XANi
Copy link
Author

XANi commented Aug 29, 2014

Yeah I think it is unrelated to, just much easier to repeat under load. I "fixed" it by just leaving default shardspace and adding ones I wanted on top of it

@pauldix
Copy link
Member

pauldix commented Sep 4, 2014

If you send writes to a non-existant database, the graphite plugin will create it. If you create a database and have no shard spaces created, it will create a default one for you.

I think the solution here is simple. Create the database AND your shard spaces before writing any data to InfluxDB. Is this not an option?

@XANi
Copy link
Author

XANi commented Sep 5, 2014

It does not create database in 0.8, it does complain in logs that there is no database ;]

And I did that. Turned off graphite plugin, created DB, turned plugin back on. Result ? Ok at first, after restarting database this happened:
img

original parameters for shard creation in https://gist.github.com/XANi/b22ad3d1174bdc851281

2 database, method of doing that was a bit different:

  • Both of them only had long_term and short_term shardspaces, both with target 2
  • One with "default" was created while graphite plugin was running, It looks after DB creation but before shard creation it got write, so default was autocreated.
  • Other looked fine.... until after restart. long_term regexp and replica count was reset after restarting innodb

Only way I managed to make it work is:

  • Naming first shard default with whatever parameters, so influxdb can freely break it on restarts
  • then making rest "as i wanted them"
  • effectively shardspace default exists, and it is rewritten by influxdb on restart with default parameters, but all data is catched by other shardspaces

@XANi XANi closed this as completed Sep 5, 2014
@XANi XANi reopened this Sep 5, 2014
@pauldix
Copy link
Member

pauldix commented Sep 5, 2014

I think I can reproduce this. Will be looking into it today

pauldix added a commit that referenced this issue Sep 5, 2014
Fixes #886. Shard spaces would not have compiled regexes when the server is restarted and the cluster config is pulled from a raft snapshot. A call to MatchSeries would then reset the regex for the shard space. BAAAAAD.
pauldix added a commit that referenced this issue Sep 5, 2014
This will help users recover from #886. It's dangerous functionality because it only changes the metadata. Will document and tell people to use with caution.
pauldix added a commit that referenced this issue Sep 8, 2014
Fixes #886. Shard spaces would not have compiled regexes when the server is restarted and the cluster config is pulled from a raft snapshot. A call to MatchSeries would then reset the regex for the shard space. BAAAAAD.
pauldix added a commit that referenced this issue Sep 8, 2014
This will help users recover from #886. It's dangerous functionality because it only changes the metadata. Will document and tell people to use with caution.
@pauldix
Copy link
Member

pauldix commented Sep 8, 2014

Fixed in 0.8.2 with this commit 64eedb7

@pauldix pauldix closed this as completed Sep 8, 2014
pauldix added a commit that referenced this issue Sep 8, 2014
Fixes #886. Shard spaces would not have compiled regexes when the server is restarted and the cluster config is pulled from a raft snapshot. A call to MatchSeries would then reset the regex for the shard space. BAAAAAD.
pauldix added a commit that referenced this issue Sep 8, 2014
This will help users recover from #886. It's dangerous functionality because it only changes the metadata. Will document and tell people to use with caution.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants