max-series-per-database limit doesn't work when max-values-per-tag is also set

[maazkhan]

We are running a single Influx 1.6.3 node with the below settings:

max-series-per-database = 2000000 // 2 million
max-values-per-tag = 100000 // 100 thousand

What we are noticing that certain databases cardinality goes beyond 2 million. Ideally some couple of thousand above the limit is fine but today one database cardinality went to 11 million.

The thing to note here is that the tags limit didn't reached on this particular database for any of the tags which somewhat indicates that as long as max-values-per-tag is not hit the cardinality of the database keeps on increasing even if max-series-per-database is set.

[dgnorton]

@maazkhan what is index-version set to? If it's set to tsi1, then those limits are not used.

[maazkhan]

@dgnorton Yes the index is set to "tsi1" Is there any way we can enforce these limits using tsi1?

The problem is that if these limits are not in place we might get unwanted metrics cardinality for some dev tenants and eventually this leads to huge startup time when we need to restart influx. Yesterday only we saw roughly a 35 mins startup time with this high cardinality.

Any ideas suggestion on this?

[dgnorton]

@maazkhan index-version = "tsi1" is designed for high cardinality (10s or 100s of millions of series) use cases. index-version = "inmem" should be fine for <= 2M series.

[maazkhan]

@dgnorton I understand "tsi1" is disk based index and supports high cardinality but our system support 100's of tenants and each of them can have around 2 million cardinality so we are looking at 100+ million cardinality overall for which I presume inmem would not be an ideal option.

At the same time we also don't want One tenant to abuse the system and generate infinite cardinality. Having hard limits on series/tags would definitely help.

[dgnorton]

@maazkhan could you run a separate instance of influxd for the dev tenants and keep well behaved prod tenants on the original instance?

[maazkhan]

@dgnorton I think we are a bit deviating from the underlying issue here. Not having cardinality control feature for "tsi1" is not suitable from a product point of view. Any production tenant has possibility to misbehave and create havoc from cardinality point of view.

Not sure why this feature is turned off for "tsi1" but I would highly suggest/recommend to bring it back. We have been running Influx for more over 3 years in production and seen first hand issues created by uncontrolled cardinality.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically closed because it has not had recent activity. Please reopen if this issue is still important to you. Thank you for your contributions.