influxdb 2.x - very slow delete statement

[matrog]

Steps to reproduce:

run the following statement to delete a non existing metric in 1 hour timeframe (it happens also with actual metrics)

[root@influxdb ~]# time influx delete --http-debug --bucket prometheus --org myorg -p '_measurement="not_existing_metric"' --start '2022-09-19T15:00:00Z' --stop '2022-09-19T16:00:00Z'
2022/09/21 07:25:41
POST /api/v2/delete?bucket=prometheus&org=myorg HTTP/1.1
Host: localhost:8086
User-Agent: influx/2.2.1 (linux) Sha/31ac783 Date/2021-11-09T21:24:22Z
Transfer-Encoding: chunked
Accept: application/json
Authorization: Token xxxxx
Content-Type: application/json
Accept-Encoding: gzip

72
{"predicate":"_measurement=\"not_existing_metric\"","start":"2022-09-19T15:00:00Z","stop":"2022-09-19T16:00:00Z"}

0


2022/09/21 07:33:39
HTTP/1.1 204 No Content
Content-Length: 0
Date: Wed, 21 Sep 2022 07:33:39 GMT
X-Influxdb-Build: OSS
X-Influxdb-Version: v2.4.0



real	7m57.226s
user	0m0.004s
sys	0m0.011s

Expected behavior:
delete statement being fast considering 1 hour timeframe and non existing metric

Actual behavior:
it takes 8 minutes

Environment info:

• System info: Linux 4.18.0-372.26.1.el8_6.x86_64 x86_64
• InfluxDB version: InfluxDB v2.4.0 (git: de247bab08) build_date: 2022-08-18T19:41:15Z

VM: 16CPU, 64GB of Ram, disk usage

/dev/mapper/influx_data 1618795736 1018890836 528798796 66% /influx_data
storage perfomance

/dev/mapper/influx_data:
 Timing buffered disk reads: 596 MB in  3.01 seconds = 198.10 MB/sec

Logs:
nothing useful in logs

Performance:

iostat.txt
profiles.tar.gz

[xPaw]

I can also confirm it's very slow. I was just trying to delete a measurement with less than 100k values and it didn't finish deleting after 20 minutes.

Furthermore, while it was executing this, the whole bucket became unavailable.

[jeffreyssmith2nd]

Are these consistently reproducible? If so, what is your usage pattern at the time it happens (lots of simultaneous deletes, heavy query or write load, etc)

[xPaw]

In my case it's read heavy, but not on the measurement that I was deleting.

[matrog]

i'm running it on a not busy instance, few writes, few reads

real 8m37.806s
user 0m0.005s
sys 0m0.011s

[jeffreyssmith2nd]

Do you have lot of measurements in the same bucket you're deleting from? What about a lot of series?

[matrog]

here's some numbers

number of series (not considering labels): 9362
number of series (considering labels): 883367
number of measurement: 1353758350
timespan: 8 months
retention: infinite

[xPaw]

This issue was closed as fixed, but I currently tried to delete some data on 2.7.5, and it's still very slow.

influx delete -o o --bucket b --predicate '_measurement="m"' --start 2015-01-01T00:00:00Z --stop 2024-04-04T00:00:00Z