poor write performance with high measurement cardinality

[cfz]

when writing to a newly created database, with a high measurement cardinality, e.g. 100k, data set, the performance is poor.
after some digging, i found that the MeasurementFieldSet would synchronously overwrite and fsync the 'fields.idx' file within a mutex with each measurement/field update, which lead to all writing requests to queue up.
one solution is to involve some delay sync techniq like the one used in WAL

[kumud-pe]

@cfz Do you think #13319 is related to this?

[cfz]

@kumud-pe it happens when you write large amount of new measurements/fields in different requests in a very short period. does it sound familiar with your case?

[kumud-pe]

Yes. We were trying to push Kafka metrics through a StatsD server. There will be a bunch of new measurements. But, it's not a delay in our case, the new measurements are not being written at all. Now, even if I try to write one new dummy measurement (manually), it's not getting written.

[cfz]

i saw a 204 successful response in your log. i'm afraid it's not my case. in my case, the write request will hung for a long time until http timeout or an error.

[kumud-pe]

Oh. Ok. This could be a different issue then. Did you check disk I/O stats? That could be becoming a bottleneck.

[cfz]

yep, i will submit a PR later for this

[dgnorton]

@cfz @kumud-pe what version are you running?

[kumud-pe]

@dgnorton Mine is a different issue (I was using 1.7.4 on K8s using tick-charts). I have closed #13319

[cfz]

i'm using 1.6.4, but i believe 1.7.x has the same behavior

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically closed because it has not had recent activity. Please reopen if this issue is still important to you. Thank you for your contributions.