CASSANDRA-8132: Save or stream hints to a safe place in node replacement

[ShaunakDas88]

The JIRA.

This seems to behave as expected. We are injecting writes to all nodes in the cluster when we kill one node:

INFO  [GossipStage:1] 2021-07-30 00:49:25,579 Gossiper.java:1290 - InetAddress /<DOWNED NODE IP> is now DOWN

We see hints are getting stored on other nodes, and when we run nodetool transferhints -d:

• the command blocks for a bit during the transfer
• no more hints on disk at the end
• hinted handoff is disabled at the end

$ ls /mnt/cass_data_disks/data1/hints/
a7c7f39d-bcaf-4b51-811a-5b00e708b7f5-1627606160115-2.crc32  a7c7f39d-bcaf-4b51-811a-5b00e708b7f5-1627606160115-2.hints  a7c7f39d-bcaf-4b51-811a-5b00e708b7f5-1627606170111-2.hints
$
$ nodetool statushandoff
Hinted handoff is running
$
$ date -u; nodetool transferhints -d; date -u
Fri Jul 30 00:59:10 UTC 2021
Fri Jul 30 00:59:18 UTC 2021
$
$ nodetool statushandoff
Hinted handoff is not running

The above timestamps for the hint transfer window are confirmed both in the system logs:

INFO  [HintsDispatcher:19] 2021-07-30 00:59:12,396 HintsDispatchExecutor.java:158 - Transferring all hints to <RECEIVING NODE IP>:7000: 15737a33-a0a6-4331-97bf-32f8af2534c1

INFO  [HintsDispatcher:19] 2021-07-30 00:59:12,532 HintsDispatchExecutor.java:282 - Finished hinted handoff of file a7c7f39d-bcaf-4b51-811a-5b00e708b7f5-1627606160115-2.hints to endpoint <RECEIVING NODE IP>:7000: 15737a33-a0a6-4331-97bf-32f8af2534c1
INFO  [HintsDispatcher:19] 2021-07-30 00:59:18,371 HintsDispatchExecutor.java:282 - Finished hinted handoff of file a7c7f39d-bcaf-4b51-811a-5b00e708b7f5-1627606170111-2.hints to endpoint <RECEIVING NODE IP>:7000: 15737a33-a0a6-4331-97bf-32f8af2534c1

DEBUG [RMI TCP Connection(10)-<SENDING NODE IP>] 2021-07-30 00:59:18,372 StorageService.java:4536 - Hint transfer complete

and the metric for total hints flat-lining: