Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

tsm compaction does not remove .tmp on error (e.g. disk full) #8123

Closed
hpbieker opened this issue Mar 12, 2017 · 2 comments
Closed

tsm compaction does not remove .tmp on error (e.g. disk full) #8123

hpbieker opened this issue Mar 12, 2017 · 2 comments
Assignees
Milestone

Comments

@hpbieker
Copy link
Contributor

System info: InfluxDB 1.2.1, Linux

Actual behavior:
When tsm compaction failes because of disk full, it reports:
Mar 12 07:58:20 PPS-UbuntuLTS influxd[11583]: [I] 2017-03-12T06:58:20Z error compacting TSM files: write /media/db_disk/data/XXXX/autogen/84/000000294-000000003.tsm.tmp: no space left on device engine=tsm1

But the file is not removed (three hours later):
$ sudo ls -l /media/db_disk/data/XXXX/autogen/84/000000294-000000003.tsm.tmp
-rw-r--r-- 1 influxdb influxdb 125829120 Mar 12 10:59 /media/db_disk/data/history/autogen/84/000000294-000000003.tsm.tmp

Desired behaviour:
The .tmp file from the compaction should be removed. (It is removed when Influx is restarted.) This is especially a problem because InfluxDB tries to compact many directories at the same time, which eats up the disk space.

Looking at the comment line 734-737 (

// We hit an error but didn't finish the compaction. Remove the temp file and abort.
) in tsdb/engine/tsm1/compact.go it looks like it is supposed to remove the file, but the actual implementation does not.
// We hit an error but didn't finish the compaction. Remove the temp file and abort.
if err != nil {
return nil, err
}

@hpbieker hpbieker changed the title tsm compaction does not remove .tmp on error tsm compaction does not remove .tmp on error (e.g. disk full) Mar 12, 2017
@hpbieker
Copy link
Contributor Author

I guess the files in the "files" list should be deleted in addition to the file in "fileName".

@rbetts rbetts assigned stuartcarnie and jwilder and unassigned stuartcarnie Apr 25, 2017
@jwilder jwilder mentioned this issue May 2, 2017
4 tasks
@jwilder jwilder added this to the 1.3.0 milestone May 4, 2017
@jwilder
Copy link
Contributor

jwilder commented May 4, 2017

Fixed via #8348

@jwilder jwilder closed this as completed May 4, 2017
@ghost ghost removed the proposed label May 4, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

5 participants