Flush memtable after timeout since got dirty (1st write) instead of idle (last write)

[aliszka]

What's being changed:

Memtable is flushed to segment file when its size exceeds given threshold or time has passed since it is being written to.

Previously idle timeout (time since last write) was used to trigger flush. In case of multiple tenants having little data pushed to memtable very often, both flush conditions (exceeded size or idle timeout) could not be met and a lot of data had to be stored in memory before it was eventually saved to file.

This PR changes flush trigger to dirty timeout (time since first write). That way memtable is flushed even if very little data is written within configured timeframe, preventing data being kept in memory for too long.

Notes:

• new env var is introduced PERSISTENCE_MEMTABLES_FLUSH_DIRTY_AFTER_SECONDS replacing existing PERSISTENCE_MEMTABLES_FLUSH_IDLE_AFTER_SECONDS. If first one is not set, latter one is used as a fallback.
• default value of PERSISTENCE_MEMTABLES_FLUSH_DIRTY_AFTER_SECONDS is 60s (same as PERSISTENCE_MEMTABLES_FLUSH_IDLE_AFTER_SECONDS used to be)

Review checklist

• Documentation has been updated, if necessary. Link to changed documentation:
• Chaos pipeline run or not necessary. Link to pipeline: [TEST] memtables dirty flush weaviate-chaos-engineering#185
• All new code is covered by tests where it is reasonable.
• Performance tests have been run or not necessary.

[amourao]

The changes look great, I just think we should do "standard" bm25 benchmarking suite to see how/if this affects performance and also run the chaos pipeline, to see if there are any changes in the tests.

[sonarcloud]

Quality Gate failed

Failed conditions
36.2% Duplication on New Code (required ≤ 3%)

See analysis details on SonarCloud