Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

DELETE query is very slow #9015

Closed
hpbieker opened this issue Oct 26, 2017 · 8 comments
Closed

DELETE query is very slow #9015

hpbieker opened this issue Oct 26, 2017 · 8 comments
Assignees
@jwilder
Labels
area/storage

Comments

@hpbieker
Copy link
Contributor

Bug report

System info:
Influx 1.3.2

Steps to reproduce:
Run DELETE FROM measurement WHERE ...

Expected behavior:
The query below should be fast.

DELETE FROM "measurement" WHERE  (time >= 'XXX' AND time < 'XXX')

Actual behavior:
It is slow, and actually I get a few of these in my log:

Detected slow query: DELETE FROM "measurement" WHERE (time >= '2015-12-09 08:00:00' AND time < '2015-12-09 11:50:00') (qid: 155309, database: dbname+, threshold: 10s) service=query
@hpbieker
Copy link
Contributor Author

hpbieker commented Oct 26, 2017 •
edited

iostat.txt
profiles.tar.gz

@hpbieker
Copy link
Contributor Author

Sorry, I did not mean to close the issue.

@hpbieker hpbieker reopened this Oct 26, 2017
@hpbieker
Copy link
Contributor Author

I am not sure why deleting is very slow, but shouldn't it be possible handle it by just storing that the values in a given period is deleted. So when querying the values are just skipped? And then when the compaction is ran, the values are finally removed.

More than 10 seconds to delete four hours is a lot...

@hpbieker
Copy link
Contributor Author

I see that I the following code is blocking (block.txt) my deletes. Could the blocks be prevented somehow?

9597643013880 32251 @ 0x406455 0x9efa14 0x9effa2 0x9c05e4 0x820ea1 0x835695 0x834ca8 0x45b2c1
#	0x406454	runtime.chanrecv1+0x34									/usr/local/go/src/runtime/chan.go:395
#	0x9efa13	github.com/influxdata/influxdb/tsdb/engine/tsm1.(*WAL).writeToLog+0x323			/root/go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/wal.go:447
#	0x9effa1	github.com/influxdata/influxdb/tsdb/engine/tsm1.(*WAL).DeleteRange+0xa1			/root/go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/wal.go:508
#	0x9c05e3	github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).DeleteSeriesRange+0x383	/root/go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:993
#	0x820ea0	github.com/influxdata/influxdb/tsdb.(*Shard).DeleteSeriesRange+0x90			/root/go/src/github.com/influxdata/influxdb/tsdb/shard.go:486
#	0x835694	github.com/influxdata/influxdb/tsdb.(*Store).DeleteSeries.func1+0x4b4			/root/go/src/github.com/influxdata/influxdb/tsdb/store.go:884
#	0x834ca7	github.com/influxdata/influxdb/tsdb.(*Store).walkShards.func1+0x47			/root/go/src/github.com/influxdata/influxdb/tsdb/store.go:630


2163533033226 1682 @ 0x406455 0x9d2f7f 0x9cffb7 0x9c0459 0x820ea1 0x835695 0x834ca8 0x45b2c1
#	0x406454	runtime.chanrecv1+0x34									/usr/local/go/src/runtime/chan.go:395
#	0x9d2f7e	github.com/influxdata/influxdb/tsdb/engine/tsm1.(*FileStore).walkFiles+0x1ae		/root/go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/file_store.go:714
#	0x9cffb6	github.com/influxdata/influxdb/tsdb/engine/tsm1.(*FileStore).DeleteRange+0x96		/root/go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/file_store.go:325
#	0x9c0458	github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).DeleteSeriesRange+0x1f8	/root/go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:969
#	0x820ea0	github.com/influxdata/influxdb/tsdb.(*Shard).DeleteSeriesRange+0x90			/root/go/src/github.com/influxdata/influxdb/tsdb/shard.go:486
#	0x835694	github.com/influxdata/influxdb/tsdb.(*Store).DeleteSeries.func1+0x4b4			/root/go/src/github.com/influxdata/influxdb/tsdb/store.go:884
#	0x834ca7	github.com/influxdata/influxdb/tsdb.(*Store).walkShards.func1+0x47			/root/go/src/github.com/influxdata/influxdb/tsdb/store.go:630

1292844309252 347 @ 0x47040a 0x9ba6f4 0x9c0301 0x820ea1 0x835695 0x834ca8 0x45b2c1
#	0x470409	sync.(*WaitGroup).Wait+0x79								/usr/local/go/src/sync/waitgroup.go:131
#	0x9ba6f3	github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).disableLevelCompactions+0x73	/root/go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:274
#	0x9c0300	github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).DeleteSeriesRange+0xa0	/root/go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:944
#	0x820ea0	github.com/influxdata/influxdb/tsdb.(*Shard).DeleteSeriesRange+0x90			/root/go/src/github.com/influxdata/influxdb/tsdb/shard.go:486
#	0x835694	github.com/influxdata/influxdb/tsdb.(*Store).DeleteSeries.func1+0x4b4			/root/go/src/github.com/influxdata/influxdb/tsdb/store.go:884
#	0x834ca7	github.com/influxdata/influxdb/tsdb.(*Store).walkShards.func1+0x47			/root/go/src/github.com/influxdata/influxdb/tsdb/store.go:630

87654592938 605 @ 0x406495 0x9cf9ca 0x9c03f3 0x820ea1 0x835695 0x834ca8 0x45b2c1
#	0x406494	runtime.chanrecv2+0x34									/usr/local/go/src/runtime/chan.go:400
#	0x9cf9c9	github.com/influxdata/influxdb/tsdb/engine/tsm1.(*FileStore).WalkKeys+0x2a9		/root/go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/file_store.go:278
#	0x9c03f2	github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).DeleteSeriesRange+0x192	/root/go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:965
#	0x820ea0	github.com/influxdata/influxdb/tsdb.(*Shard).DeleteSeriesRange+0x90			/root/go/src/github.com/influxdata/influxdb/tsdb/shard.go:486
#	0x835694	github.com/influxdata/influxdb/tsdb.(*Store).DeleteSeries.func1+0x4b4			/root/go/src/github.com/influxdata/influxdb/tsdb/store.go:884
#	0x834ca7	github.com/influxdata/influxdb/tsdb.(*Store).walkShards.func1+0x47			/root/go/src/github.com/influxdata/influxdb/tsdb/store.go:630

71620425369 616 @ 0x406495 0x9cf9ca 0x9c014c 0x9c062f 0x820ea1 0x835695 0x834ca8 0x45b2c1
#	0x406494	runtime.chanrecv2+0x34									/usr/local/go/src/runtime/chan.go:400
#	0x9cf9c9	github.com/influxdata/influxdb/tsdb/engine/tsm1.(*FileStore).WalkKeys+0x2a9		/root/go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/file_store.go:278
#	0x9c014b	github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).containsSeries+0x28b		/root/go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:915
#	0x9c062e	github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).DeleteSeriesRange+0x3ce	/root/go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:999
#	0x820ea0	github.com/influxdata/influxdb/tsdb.(*Shard).DeleteSeriesRange+0x90			/root/go/src/github.com/influxdata/influxdb/tsdb/shard.go:486
#	0x835694	github.com/influxdata/influxdb/tsdb.(*Store).DeleteSeries.func1+0x4b4			/root/go/src/github.com/influxdata/influxdb/tsdb/store.go:884
#	0x834ca7	github.com/influxdata/influxdb/tsdb.(*Store).walkShards.func1+0x47			/root/go/src/github.com/influxdata/influxdb/tsdb/store.go:630

64257055725 319 @ 0x46ffde 0x9ba6b1 0x9c0301 0x820ea1 0x835695 0x834ca8 0x45b2c1
#	0x46ffdd	sync.(*RWMutex).Lock+0x6d								/usr/local/go/src/sync/rwmutex.go:91
#	0x9ba6b0	github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).disableLevelCompactions+0x30	/root/go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:257
#	0x9c0300	github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).DeleteSeriesRange+0xa0	/root/go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:944
#	0x820ea0	github.com/influxdata/influxdb/tsdb.(*Shard).DeleteSeriesRange+0x90			/root/go/src/github.com/influxdata/influxdb/tsdb/shard.go:486
#	0x835694	github.com/influxdata/influxdb/tsdb.(*Store).DeleteSeries.func1+0x4b4			/root/go/src/github.com/influxdata/influxdb/tsdb/store.go:884
#	0x834ca7	github.com/influxdata/influxdb/tsdb.(*Store).walkShards.func1+0x47			/root/go/src/github.com/influxdata/influxdb/tsdb/store.go:630

31502338863 108 @ 0x46ffde 0x9ba48e 0x9c06e5 0x820ea1 0x835695 0x834ca8 0x45b2c1
#	0x46ffdd	sync.(*RWMutex).Lock+0x6d								/usr/local/go/src/sync/rwmutex.go:91
#	0x9ba48d	github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).enableLevelCompactions+0x3d	/root/go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:228
#	0x9c06e4	github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).DeleteSeriesRange+0x484	/root/go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:1012
#	0x820ea0	github.com/influxdata/influxdb/tsdb.(*Shard).DeleteSeriesRange+0x90			/root/go/src/github.com/influxdata/influxdb/tsdb/shard.go:486
#	0x835694	github.com/influxdata/influxdb/tsdb.(*Store).DeleteSeries.func1+0x4b4			/root/go/src/github.com/influxdata/influxdb/tsdb/store.go:884
#	0x834ca7	github.com/influxdata/influxdb/tsdb.(*Store).walkShards.func1+0x47			/root/go/src/github.com/influxdata/influxdb/tsdb/store.go:630

28316278185 64 @ 0x46efed 0x46ff9d 0x9ba6b1 0x9c0301 0x820ea1 0x835695 0x834ca8 0x45b2c1
#	0x46efec	sync.(*Mutex).Lock+0x9c									/usr/local/go/src/sync/mutex.go:87
#	0x46ff9c	sync.(*RWMutex).Lock+0x2c								/usr/local/go/src/sync/rwmutex.go:86
#	0x9ba6b0	github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).disableLevelCompactions+0x30	/root/go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:257
#	0x9c0300	github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).DeleteSeriesRange+0xa0	/root/go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:944
#	0x820ea0	github.com/influxdata/influxdb/tsdb.(*Shard).DeleteSeriesRange+0x90			/root/go/src/github.com/influxdata/influxdb/tsdb/shard.go:486
#	0x835694	github.com/influxdata/influxdb/tsdb.(*Store).DeleteSeries.func1+0x4b4			/root/go/src/github.com/influxdata/influxdb/tsdb/store.go:884
#	0x834ca7	github.com/influxdata/influxdb/tsdb.(*Store).walkShards.func1+0x47			/root/go/src/github.com/influxdata/influxdb/tsdb/store.go:630

21802171593 50 @ 0x46efed 0x46ff9d 0x9ba48e 0x9c06e5 0x820ea1 0x835695 0x834ca8 0x45b2c1
#	0x46efec	sync.(*Mutex).Lock+0x9c									/usr/local/go/src/sync/mutex.go:87
#	0x46ff9c	sync.(*RWMutex).Lock+0x2c								/usr/local/go/src/sync/rwmutex.go:86
#	0x9ba48d	github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).enableLevelCompactions+0x3d	/root/go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:228
#	0x9c06e4	github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).DeleteSeriesRange+0x484	/root/go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:1012
#	0x820ea0	github.com/influxdata/influxdb/tsdb.(*Shard).DeleteSeriesRange+0x90			/root/go/src/github.com/influxdata/influxdb/tsdb/shard.go:486
#	0x835694	github.com/influxdata/influxdb/tsdb.(*Store).DeleteSeries.func1+0x4b4			/root/go/src/github.com/influxdata/influxdb/tsdb/store.go:884
#	0x834ca7	github.com/influxdata/influxdb/tsdb.(*Store).walkShards.func1+0x47			/root/go/src/github.com/influxdata/influxdb/tsdb/store.go:630

2979152928 8 @ 0x46efed 0x46ff9d 0xa0094e 0x9ef93b 0x9effa2 0x9c05e4 0x820ea1 0x835695 0x834ca8 0x45b2c1
#	0x46efec	sync.(*Mutex).Lock+0x9c									/usr/local/go/src/sync/mutex.go:87
#	0x46ff9c	sync.(*RWMutex).Lock+0x2c								/usr/local/go/src/sync/rwmutex.go:86
#	0xa0094d	github.com/influxdata/influxdb/tsdb/engine/tsm1.(*WAL).writeToLog.func1+0x5d		/root/go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/wal.go:404
#	0x9ef93a	github.com/influxdata/influxdb/tsdb/engine/tsm1.(*WAL).writeToLog+0x24a			/root/go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/wal.go:438
#	0x9effa1	github.com/influxdata/influxdb/tsdb/engine/tsm1.(*WAL).DeleteRange+0xa1			/root/go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/wal.go:508
#	0x9c05e3	github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).DeleteSeriesRange+0x383	/root/go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:993
#	0x820ea0	github.com/influxdata/influxdb/tsdb.(*Shard).DeleteSeriesRange+0x90			/root/go/src/github.com/influxdata/influxdb/tsdb/shard.go:486
#	0x835694	github.com/influxdata/influxdb/tsdb.(*Store).DeleteSeries.func1+0x4b4			/root/go/src/github.com/influxdata/influxdb/tsdb/store.go:884
#	0x834ca7	github.com/influxdata/influxdb/tsdb.(*Store).walkShards.func1+0x47			/root/go/src/github.com/influxdata/influxdb/tsdb/store.go:630

2522426802 1 @ 0x46efed 0x46ff9d 0x9cffda 0x9c0459 0x820ea1 0x835695 0x834ca8 0x45b2c1
#	0x46efec	sync.(*Mutex).Lock+0x9c									/usr/local/go/src/sync/mutex.go:87
#	0x46ff9c	sync.(*RWMutex).Lock+0x2c								/usr/local/go/src/sync/rwmutex.go:86
#	0x9cffd9	github.com/influxdata/influxdb/tsdb/engine/tsm1.(*FileStore).DeleteRange+0xb9		/root/go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/file_store.go:329
#	0x9c0458	github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).DeleteSeriesRange+0x1f8	/root/go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:969
#	0x820ea0	github.com/influxdata/influxdb/tsdb.(*Shard).DeleteSeriesRange+0x90			/root/go/src/github.com/influxdata/influxdb/tsdb/shard.go:486
#	0x835694	github.com/influxdata/influxdb/tsdb.(*Store).DeleteSeries.func1+0x4b4			/root/go/src/github.com/influxdata/influxdb/tsdb/store.go:884
#	0x834ca7	github.com/influxdata/influxdb/tsdb.(*Store).walkShards.func1+0x47			/root/go/src/github.com/influxdata/influxdb/tsdb/store.go:630

2284633347 1 @ 0x46ffde 0x9cffda 0x9c0459 0x820ea1 0x835695 0x834ca8 0x45b2c1
#	0x46ffdd	sync.(*RWMutex).Lock+0x6d								/usr/local/go/src/sync/rwmutex.go:91
#	0x9cffd9	github.com/influxdata/influxdb/tsdb/engine/tsm1.(*FileStore).DeleteRange+0xb9		/root/go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/file_store.go:329
#	0x9c0458	github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).DeleteSeriesRange+0x1f8	/root/go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:969
#	0x820ea0	github.com/influxdata/influxdb/tsdb.(*Shard).DeleteSeriesRange+0x90			/root/go/src/github.com/influxdata/influxdb/tsdb/shard.go:486
#	0x835694	github.com/influxdata/influxdb/tsdb.(*Store).DeleteSeries.func1+0x4b4			/root/go/src/github.com/influxdata/influxdb/tsdb/store.go:884
#	0x834ca7	github.com/influxdata/influxdb/tsdb.(*Store).walkShards.func1+0x47			/root/go/src/github.com/influxdata/influxdb/tsdb/store.go:630

775087032 5 @ 0x46feb9 0x99f98a 0x9c0542 0x820ea1 0x835695 0x834ca8 0x45b2c1
#	0x46feb8	sync.(*RWMutex).RLock+0x48								/usr/local/go/src/sync/rwmutex.go:43
#	0x99f989	github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Cache).ApplyEntryFn+0x39		/root/go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/cache.go:576
#	0x9c0541	github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).DeleteSeriesRange+0x2e1	/root/go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:988
#	0x820ea0	github.com/influxdata/influxdb/tsdb.(*Shard).DeleteSeriesRange+0x90			/root/go/src/github.com/influxdata/influxdb/tsdb/shard.go:486
#	0x835694	github.com/influxdata/influxdb/tsdb.(*Store).DeleteSeries.func1+0x4b4			/root/go/src/github.com/influxdata/influxdb/tsdb/store.go:884
#	0x834ca7	github.com/influxdata/influxdb/tsdb.(*Store).walkShards.func1+0x47			/root/go/src/github.com/influxdata/influxdb/tsdb/store.go:630

756727782 7 @ 0x46efed 0x46ff9d 0x99f5aa 0x9c0593 0x820ea1 0x835695 0x834ca8 0x45b2c1
#	0x46efec	sync.(*Mutex).Lock+0x9c									/usr/local/go/src/sync/mutex.go:87
#	0x46ff9c	sync.(*RWMutex).Lock+0x2c								/usr/local/go/src/sync/rwmutex.go:86
#	0x99f5a9	github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Cache).DeleteRange+0x39		/root/go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/cache.go:522
#	0x9c0592	github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).DeleteSeriesRange+0x332	/root/go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:990
#	0x820ea0	github.com/influxdata/influxdb/tsdb.(*Shard).DeleteSeriesRange+0x90			/root/go/src/github.com/influxdata/influxdb/tsdb/shard.go:486
#	0x835694	github.com/influxdata/influxdb/tsdb.(*Store).DeleteSeries.func1+0x4b4			/root/go/src/github.com/influxdata/influxdb/tsdb/store.go:884
#	0x834ca7	github.com/influxdata/influxdb/tsdb.(*Store).walkShards.func1+0x47			/root/go/src/github.com/influxdata/influxdb/tsdb/store.go:630

557750058 1 @ 0x46efed 0x46ff9d 0x9504ad 0x946549 0x9c0730 0x820ea1 0x835695 0x834ca8 0x45b2c1
#	0x46efec	sync.(*Mutex).Lock+0x9c									/usr/local/go/src/sync/mutex.go:87
#	0x46ff9c	sync.(*RWMutex).Lock+0x2c								/usr/local/go/src/sync/rwmutex.go:86
#	0x9504ac	github.com/influxdata/influxdb/tsdb/index/inmem.(*Series).UnassignShard+0x2c		/root/go/src/github.com/influxdata/influxdb/tsdb/index/inmem/meta.go:1146
#	0x946548	github.com/influxdata/influxdb/tsdb/index/inmem.(*Index).UnassignShard+0xf8		/root/go/src/github.com/influxdata/influxdb/tsdb/index/inmem/inmem.go:759
#	0x9c072f	github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).DeleteSeriesRange+0x4cf	/root/go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:1006
#	0x820ea0	github.com/influxdata/influxdb/tsdb.(*Shard).DeleteSeriesRange+0x90			/root/go/src/github.com/influxdata/influxdb/tsdb/shard.go:486
#	0x835694	github.com/influxdata/influxdb/tsdb.(*Store).DeleteSeries.func1+0x4b4			/root/go/src/github.com/influxdata/influxdb/tsdb/store.go:884
#	0x834ca7	github.com/influxdata/influxdb/tsdb.(*Store).walkShards.func1+0x47			/root/go/src/github.com/influxdata/influxdb/tsdb/store.go:630

56192685 5 @ 0x46feb9 0x99ef2a 0x9bfff3 0x9c062f 0x820ea1 0x835695 0x834ca8 0x45b2c1
#	0x46feb8	sync.(*RWMutex).RLock+0x48								/usr/local/go/src/sync/rwmutex.go:43
#	0x99ef29	github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Cache).unsortedKeys+0x39		/root/go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/cache.go:449
#	0x9bfff2	github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).containsSeries+0x132		/root/go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:904
#	0x9c062e	github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).DeleteSeriesRange+0x3ce	/root/go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:999
#	0x820ea0	github.com/influxdata/influxdb/tsdb.(*Shard).DeleteSeriesRange+0x90			/root/go/src/github.com/influxdata/influxdb/tsdb/shard.go:486
#	0x835694	github.com/influxdata/influxdb/tsdb.(*Store).DeleteSeries.func1+0x4b4			/root/go/src/github.com/influxdata/influxdb/tsdb/store.go:884
#	0x834ca7	github.com/influxdata/influxdb/tsdb.(*Store).walkShards.func1+0x47			/root/go/src/github.com/influxdata/influxdb/tsdb/store.go:630

@hpbieker
Copy link
Contributor Author

hpbieker commented Nov 2, 2017

@e-dard -- is it possible to get this into the milestone for 1.4 or 1.4.1?

@e-dard
Copy link
Contributor

e-dard commented Nov 2, 2017

@hpbieker we have made some significant changes recently to how deletes are handled, but it's getting late into the day to push that into 1.4.

However, if the issue you're experiencing is a regression of some kind then it could be fixed and put into a patch release on either the 1.3 or 1.4 releases. @jwilder might have more insight into the issue you're experiencing.

@hpbieker
Copy link
Contributor Author

hpbieker commented Jan 5, 2018

Hi again @e-dard and @jwilder,

After digging a bit in the source code, it looks like the problem (at least one part) is that Engine.disableLevelCompactions() has to run before a delete is allowed. This will wait for all (!?!) compaction jobs on the engine to finish and prevent new ones to start.

If we assume that a full compact of a shard is running while we try to delete a few samples in a different shard, the delete job will just wait for the compaction job to finish (such a job may take 15 minutes here) before the samples can be removed.

Wouldn't it be better if:

  • The delete actually aborted the compaction job, or
  • Deleting samples in one shard was not affected by compaction in a second shard, or
  • Thumbstoning was done by appending a special deletetimerange block in the tsm files, and this delete was then merged when reading data and on next compaction?

It might be that I missed a few points here?

@jwilder
Copy link
Contributor

jwilder commented Jan 8, 2018

@hpbieker Deletes were significantly reworked in #9084 which also improved performance. This will be a part of 1.5.

  • The delete actually aborted the compaction job, or

Engine.disableLevelCompactions does abort running compactions. It calls Compactor.DisableCompactions which closes the compactionsInterrupt channel. Running compactions periodically check the compactionInterrupt channel and abort when it's closed. This was initially added in #8629 which was part of 1.4.

  • Deleting samples in one shard was not affected by compaction in a second shard, or

Shards are independent. There is an engine per shard so deleting samples in one shard is not affected by compaction in others. The systems resources (CPU/Disk/Memory) are shared so there is some throttling and limits to prevent many concurrent deletes across shards from adversely affecting the system.

  • Thumbstoning was done by appending a special deletetimerange block in the tsm files, and this delete was then merged when reading data and on next compaction?

That is essentially what we do. TSM files are immutable by design so we append a tombstone record to a tombstone file instead and the delete is merged when reading and removed on the next compaction that that file is a part of.

@jwilder jwilder closed this as completed Jan 8, 2018
@hpbieker hpbieker mentioned this issue Aug 22, 2018
Closed
@hpbieker hpbieker mentioned this issue Nov 20, 2018
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jwilder jwilder

Labels
area/storage
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@jwilder @e-dard @hpbieker