[DocDB] Intents are populated back after restart and not cleaned up

[suranjan]

Jira Link: DB-5931

Description

On restart, we replay the transactions that were not flushed and also try to clean them up if we find that they were already applied or if we applied it again. While cleaning the intent, cleanup task might fail due to race conditions. Basically, we initialise the thread pool for clean up tasks after bootstrap.

So, the transaction metadata is cleaned up but the intents do remain in IntentDB and they only get cleaned up when compaction takes place as we have some optimisations where we delete the intent files as well.
I think this could have 'intermittent' performance issue

create table sample(a int, primary key(a asc));
insert into sample select generate_series(1,100000);
./build/latest/bin/yb-ts-cli count_intents
0

restart the cluster

./build/latest/bin/yb-ts-cli count_intents
20776

This also causes perf issue.

If we flush all tablets and then restart the intent count goes back to 0.
./build/latest/bin/yb-ts-cli flush_all_tablets

Restart the cluster

./build/latest/bin/yb-ts-cli count_intents
0

Warning: Please confirm that this issue does not contain any sensitive information

• I confirm this issue does not contain any sensitive information.