ActivityTimeout event handling behavior not consistent in Deli

[iamanuradha]

Describe the bug

LambdaCloseType.ActivityTimeout event is received and processed successfully by Scribe, but Deli receives and/or processes this event intermittently, because of this sometimes the session isSessionActive and isSessionAlive flag set to false, but sometimes the event is either not received or not processed by Deli properly.

To Reproduce

Steps to reproduce the behavior:

1. Create a container. Does not have matter how many clients we connect to this, issue can be reproduced even with one client.
2. Do some operations on the shared dds used in conatinerSchema
3. Wait for 15 mins
4. close the container
5. Check for the isSessionAlive and isSessionActive flag and/or check log for "Session terminated due to inactivity". This appears in Scribe but not in Deli

Expected behavior

1. Deli's acitivtyTimeout and close event handling needs to be consistent.

Logs

Below are the snippet from splunk logs
{"line":{"durationInMs":798823.436247,"eventName":"ScribeSessionResult","id":"3d3aef78-4621-46d1-8985-f43306c2f495","label":"winston","level":"info","message":"Session terminated due to inactivity","properties":"{"tenantId":"fluid","documentId":"1eca0c47-0313-451f-96dd-8d2f846adbde","sessionEndReason":"ActivityTimeout","sessionState":"end","sequenceNumber":12,"lastSummarySequenceNumber":7}","successful":true,
"timestamp":"2023-07-05T08:48:17.255Z","type":"Metric"},"source":"stdout","tag":"731817eed348","attrs":{"com.amazonaws.ecs.task-arn":"arn:aws:ecs:us-west-2:401049420436:task/CEDIT-C-UW2/731817eed3484b8da98c894c07a4a2ab","docker.image":"401049420436.dkr.ecr.us-west-2.amazonaws.com/autodeskcloud/concurrent-editing-deploy-scribe:HDSEnvVarClenup-daf064b6af-5","obs.mnkr":"cedit-c-uw2","spinnaker.servergroup":"cedit-c_uw2-scribe_app-v276"}}

{"line":{"eventName":"LogMessage:lambda.js:Promise.all.then.errorMsg","id":"2e678ecb-bcc5-42f2-ad1d-36bf0b3543c4","label":"winston","level":"info","message":"Writing checkpoint. Reason: EveryMessage","properties":"{"tenantId":"fluid","documentId":"1eca0c47-0313-451f-96dd-8d2f846adbde","checkpointReason":"EveryMessage"}","timestamp":"2023-07-05T08:50:55.360Z","type":"Log"},"source":"stdout","tag":"77d9280d44cb","attrs":{"com.amazonaws.ecs.task-arn":"arn:aws:ecs:us-west-2:401049420436:task/CEDIT-C-UW2/77d9280d44cb4471a4a72f5d30849933","docker.image":"401049420436.dkr.ecr.us-west-2.amazonaws.com/autodeskcloud/concurrent-editing-deploy-deli:master-fcc52cd28c-315","obs.mnkr":"cedit-c-uw2","spinnaker.servergroup":"cedit-c_uw2-deli_app-v272"}}

[iamanuradha]

On analysing further found that this issue occurs when rebalance take place

[microsoft-github-policy-service]

This issue has been automatically marked as stale because it has had no activity for 180 days. It will be closed if no further activity occurs within 8 days of this comment. Thank you for your contributions to Fluid Framework!