latency spike and wave pattern in disk usage - v19.2.4

[ghost]

Describe the problem

Please describe the issue you observed, and any steps we can take to reproduce it:
latency problem; production system.

To Reproduce

What did you do? Describe in your own words.
added a whole lot of BLOBs (~1TB; 50 mil. rows)

Additional data / screenshots
1 week graph

Environment:

single node
started with v19.2.2 and when noticing the problem, updated to v19.2.4
Linux 4.15.0-74-generic 84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Jira issue: CRDB-8005

[petermattis]

The wave pattern in disk usage is really strange. Even with RocksDB compactions we never see fluctuations that dramatic. Can you share the RocksDB SSTables and RocksDB Flushes/Compactions graphs (on the Storage dashboard)?

What did you do? Describe in your own words.
added a whole lot of BLOBs (~1TB; 50 mil. rows)

It would be helpful for you to precisely describe what your workload is doing, the types of queries, the size of the cluster, including the number of nodes and the types of machines. Even better if you can share source code from the workload program.

[ghost]

There are two drops, these are caused by me restarting the node.
The reason its now less violent is (I guess), because I set gc.ttlseconds = 1000000.

It would be helpful for you to precisely describe what your workload is doing, the types of queries, the size of the cluster, including the number of nodes and the types of machines. Even better if you can share source code from the workload program.

Single node; 48 cores; 512 GB ram; 6 disks NVMe RAID10.
As said, this is a production system, not a stress test system.
All types of queries, with an avg. of 2-3k q/s

[ghost]

I yesterday ran into that problem again (latency spikes) and talked with @dt about how to change the soft and hard limit for compaction bytes and now that I think of it, I dropped about a week before the problems started a column in a table that was maybe 400-500 GB-ish JSONB
So maybe this is related to #24029 and #26693

cockroach-rocksdb.db.root.2020-03-07T07_37_58Z.043406.log

[mwang1026]

19.2 has been EOL for a while. If you repro this in a higher version feel free to reopen