[DocDB][LRU] Tserver continues to access SST files that were deleted as a result they are not cleaned from the system[2.8.8.0-b2]

[kripasreenivasan]

Jira Link: DB-3337

Description

• Create a universe in 2.8.8.0-b2 with the following gflags

• Trigger a YCQL- Cassandra batch time series workload from a load VM with a large no. of keys such that it will execute over 5-6 days without interruption.
• After the workload ends, connect to the leader node of the universe and drop the YCQL table created by the above workload.
  Issue:

1. Navigate to the UI and reload the overview and nodes tab of this universe. You will find that though the table was successfully dropped and disappears from the tables listing tab, there is no difference to the disk usage metrics, SST file size before and after deleetion. On navigating to the path corresponding to the table in question in any of the nodes, you will find the tablets are not deleted: /mnt/d0/yb-data/tserver/data/rocksdb/table-c07537a0504c463e92b80e1e0c277841/tablet-1a1a2c6d7d6c4af48265a33194d2d20c
2. There are lots of open files for the tserver process-

[yugabyte@ip-172-151-27-132 master]$ ps aux|grep yb-tserver
yugabyte  7803  217 34.1 18932688 2629692 ?    Sl   Aug25 21158:29 /home/yugabyte/tserver/bin/yb-tserver --flagfile /home/yugabyte/tserver/conf/server.conf
yugabyte 18297  0.0  0.0 112636   984 pts/0    S+   04:57   0:00 grep --color=auto yb-tserver

3. Looks like the SST files were deleted (unlinked on filesystem), but for some reason tserver is still accessing them, thus they can’t actually be cleaned from the filesystem yet. We should not keep them open when we delete them from File system.

[yugabyte@ip-172-151-27-132 fd]$ ls -lha | grep sst | grep deleted | wc -l
312

A rolling restart fixes this however, we should handle this gracefully without a restart.

This was observed on long running universe repeatedly: http://portal.aws-stable.dev.yugabyte.com/universes/309cdc4f-a785-4013-86e6-3d6555eab86a/overview

Slack thread: https://yugabyte.slack.com/archives/C01CB38CZHU/p1661420772861299

CC: @bmatican , @ragh