[Epic] ODSP Scalability

[vladsud]

This EPIC will track all of the issues identified as result of analyzing scalability runs.

All critical issues should be duplicated in VSO:
https://onedrive.visualstudio.com/SPIN/_queries/query/e214a6b1-472f-49f6-aace-0a785ab3a8be/

Issues that are blocking progress / scalability or add too much noise.
Roughly in priority order, based on impact I think these issues will have on Production usage, not scalability run workloads:

• SPO: Reduce number of storage calls when ops can be retrieved from PUSH #7145
• 0x07b, 0x07d /* "Unexpected op" */ assert in TaskManager in ODSP scalability tests #6689
• We still hit "Failed to retrieve ops from storage: too many retries" in prod & stress tests #7313
• fluid:telemetry:OpPerf:OpRoundtripTime errors are hit in prod #7406

Not blocking issue / low priority / forward looking

• Hitting 10K unsummarized ops results in client getting into infinite loop of reconnects #7137
• "Container was closed while load()" errors in ODSP scalability runs #6911
• Unhandled exception when fetching snapshot times out #6996
• It would be great for fluid:telemetry:Container:ContainerWarning event to have "source" property #7138
• Too many fluid:telemetry:Container:connectedStateRejected events with source = timeout #8235
• Change default connection mode from "write" to "read" #7772

Issues that has been addressed

• fluid:telemetry:DeltaManager:NoJoinOp is hit in production and ODSP scalability tests #7312
• Improve summary failure logging #6649
• ODSP: Add client correlation ID on Ops fetch requests  #7000
• fluid:telemetry:InitialElectedClientNotFound errors in telemetry #6948
• Add handling of summary nacks error codes and retryAfter #6596
• MissingAudienceMember event is hit rather often #6910
• ODSP Scaling: fluid:telemetry:Container:connectedStateRejected #7016
• Early container closure errors in ODSP stress tests #6690
• VSO 1156752 - Summary nack message is very generic (Failed to summarize the document), makes it hard to diagnose issues.
• VSO 1156694 - Clients not sending ops
• Summarizer election may result in summarizer not being started #7311
• Unify single-summary locking and advancement of summaryGenTag property in logging #7274
• "Error 412 (Precondition Failed)" should not be retryable error when uploading summaries #7158
• Follow ups to fluid:telemetry:OdspDriver:GetDeltas_cancel events #7040
• Need reliable way for ops to reach SPO for summaries to succeed (single-stage summary commit) #7021
• Clients get stuck (with a bunch of downstream failures including summarization stalling) when fetch() takes 1 min+ to complete / fail #6997
• Summarizer keep going when summarizer container is closed #6896
• Early container closure errors in ODSP stress tests #6690
• Design & implement system that reduces storage pressure and provides client feedback when PUSH is throttled #6685
• Summarizer telemetry to better reflect summarization attempt #6648
• Container closes without proper error. #6027
• FetchMissingDeltas Incorrectly closing containers on some paths #6026

Other references:
#6555 - Analyze latest ODSP stress test run
#6335 - Change elected summarizer when not summarizing for too long (unrelated, but has a chance to make things worse, so we should be very careful about enabling it with a safe velocity in mind)

Client side logging from these runs is stored here: Logs
Latest TAP40 run: customDimensions.testUid = 980e2dc0-f8ef-4661-9d5d

[vladsud]

Most of the items here are either fixed, or tracked in other EPICs, or likely need better home. So closing this EPIC