client.WriteSeries returns: Server returned (400): IO error: /opt/influxdb/shared/data/db/shard_db_v2/00190/MANIFEST-000006: No such file or directory #985
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
this might also be useful |
|
when i manually retry the same write later, it works fine. so maybe it uses another dir then, or it was a race condition between creating the dir and trying to use it? |
|
today i got another, bit more exotic variant of this: nothing particular in /var/log/messages, dmesg, and plenty of space and inodes available. again, resuming my program where it left off (i.e. doing the same write that failed) seems to work fine |
|
Similar to #1009 and #1013. This is caused by the concurrent closing of a shard and opening it at the same time. This operation needs to be go routine safe. I'm not sure why the shards are being dropped though. @Dieterbe are you trying to write points in the past or the data collection is lagging behind ? |
|
yes, this is an import of old data, with timestamps anywhere between 2y ago and now. |
|
What's the retention and duration of those shards |
|
not sure, have recreated the db a couple of times in the meantime. I think i've usually kept shard duration 7d, retention was probably 365 or 730 days. (it's possible that some of the points being written have timestamps older than what the shard cares about) |
|
Cool, just wanted to make sure my guess makes sense. |
* shard_datastore.go(Deleteshard): Check the reference count of the shard and mark it for deletion if there are still more references out there. Otherwise, delete the shard immediately. Also refactor the deletion code in deleteShard(), see below. * shard_datastore.go(ReturnShard): Check to see if the shard is marked for deletion. * shard_datastore.go(deleteShard): Refactor the code that used to be in Deleteshard in its own method. Use `closeShard` instead of doing the cleanup ourselves.
|
Looks good to me. |
|
lgtm |
jvshahid
added a commit
that referenced
this pull request
Oct 24, 2014
client.WriteSeries returns: Server returned (400): IO error: /opt/influxdb/shared/data/db/shard_db_v2/00190/MANIFEST-000006: No such file or directory
jvshahid
added a commit
that referenced
this pull request
Oct 31, 2014
Background of the bug: Prior to this patch we actually tried writing points that were older than the retention period of the shard. This caused race condition when it came to writing points to a shard that's being dropped, which will happen frequently if the user is loading old data (by accident). This is demonstrated in the test in this commit.This bug was previously addressed in #985. It turns the fix for #985 wasn't enough. A user reported in #1078 that some shards are left behind and not deleted. It turns out that while the shard is being dropped more write requests could come in and end up on line `cluster/shard.go:195` which will cause the datastore to create a shard on disk that isn't tracked anywhere in the metadata. This shard will live forever and never get deleted. This fix address this issue by not writing old points in, but there are still some edge cases with the current implementation, at least not as bad as current master.
jvshahid
added a commit
that referenced
this pull request
Nov 3, 2014
Background of the bug: Prior to this patch we actually tried writing points that were older than the retention period of the shard. This caused race condition when it came to writing points to a shard that's being dropped, which will happen frequently if the user is loading old data (by accident). This is demonstrated in the test in this commit.This bug was previously addressed in #985. It turns the fix for #985 wasn't enough. A user reported in #1078 that some shards are left behind and not deleted. It turns out that while the shard is being dropped more write requests could come in and end up on line `cluster/shard.go:195` which will cause the datastore to create a shard on disk that isn't tracked anywhere in the metadata. This shard will live forever and never get deleted. This fix address this issue by not writing old points in, but there are still some edge cases with the current implementation, at least not as bad as current master. Close #1078
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
using https://github.com/vimeo/whisper-to-influxdb/ which invokes
influxClient.WriteSeriesWithTimePrecision(toCommit, client.Second)to write a series called "servers.dfvimeostatsd1.diskspace.root.inodes_free" with 60643 records of (time, sequence_number, value) format, to my graphite database, which i recreated from scratch yesterday after i upgraded.
got this response:
my influxdb is 0.8.3, has debug logging enabled,
but the log only contains messages matching
(GraphiteServer committing|Executing leader loop|Dumping the cluster config|Testing if we should|Checking for shards to drop), no other messages.I also checked dmesg, no errors there. ditto for /var/log/messages, nothing useful there.