Improved compaction scheduling #8886
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ghost
assigned 
e-dard
suggested changes
Oct 3, 2017
tsdb/engine/tsm1/writer.go
Outdated
| return nil | ||
| } | ||
|
|
||
| if err := d.fd.Close(); err != nil { |
There was a problem hiding this comment.
So if there is a problem closing the file we're not going to clean up, but we're not going to return an error to the caller either? This scenario seems unusual enough that we don't want to ignore it or not log it.
There was a problem hiding this comment.
That should be returning the err here. Will fix.
tsdb/engine/tsm1/writer.go
Outdated
| } | ||
|
|
||
| if f, ok := t.wrapped.(*os.File); ok { | ||
| f.Close() |
There was a problem hiding this comment.
In this case we're going to ignore a Close error (fine) but then still try and remove the file. This seems at odds with how the directIndex is handled.
There was a problem hiding this comment.
I'll fix this up too.
jwilder
changed the title
jwilder
changed the title
Doing this for the WAL reduces throughput quite a bit.
This changes the compaction scheduling to better utilize the available cores that are free. Previously, a level was planned in its own goroutine and would kick off a number of compactions groups. The problem with this model was that if there were 4 groups, and 3 completed quickly, the planning would be blocked for that level until the last group finished. If the compactions at the prior level are running more quickly, a large backlog could accumlate. This now moves the planning to a single goroutine that plans each level in succession and starts as many groups as it can. When one group finishes, the planning will start the next group for the level.
Shows up under high cardinality compactions.
This switches the thresholds that are used for writing snapshots concurrently. This scales better than the prior model.
The chunked slice is unnecessary and we can re-use k.blocks throughout the compaction.
This gives an indication as to whether compactions are backed up or not.
With higher cardinality or larger series keys, the files can roll over early which causes them to take longer to be compacted by higher levels. This causes larger disk usage and higher numbers of tsm files at times.
One shard might be able to run a compaction, but could fail to limits being hit. This loop would continue indefinitely as the same task would continue to be rescheduled.
This check doesn't make sense for high cardinality data as the files typically get big and sparse very quickly. This causes a lot of extra disk space to be used which is taken up by large indexes and sparse data.
e-dard
approved these changes
Oct 3, 2017
If there is a backlog of level 3 and 4 compacitons, but few level 1 and 2 compactions, allow them to use some excess capacity.
Some files seem to get orphan behind higher levels. This causes the compactions to get blocked as the lowere level files will not get picked up by their lower level planners. This allows the full plan to identify them and pull them into their plans.
4 tasks
This was referenced Mar 6, 2018
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Required for all non-trivial PRs
This fixes some regressions and new issues due to recent changes from #8872 and #8856.
O_SYNCflag for the wal for now.max-concurrent-compactions).There are also two bug fixes where the temp TSM index file could be left around when running out of disk space as well as fixing the cache mem bytes stat to include the size of the series key.