RUM-1837 Update logic to send N batches sequentially in each cycle

[maciejburda]

What and why?

Modifies the core logic to get a list of batches instead of single batch. It's done through a new configuration called BatchProcessingLevel that allows controlling the amount of batches processed sequentially without a delay within one reading/uploading cycle. Currently it exposed 3 levels: low, medium and high that translate to 1, 10 and 100 of batches processed. By default it's taking up to 10 batches in a cycle.

This logic improves the data upload when batch back pressure occurs.

How?

Configuration is delivered to DataUploadWorker which implementation was modified to read up to X batches instead of single batch.

Review checklist

• Feature or bugfix MUST have appropriate tests (unit, integration)
• Make sure each commit and the PR mention the Issue number or JIRA reference
• Add CHANGELOG entry for user facing changes

Custom CI job configuration (optional)

• Run unit tests for Core, RUM, Trace, Logs, CR and WVT
• Run unit tests for Session Replay
• Run integration tests
• Run smoke tests
• Run tests for tools/

[datadog-datadog-prod-us1]

Datadog Report

Branch report: maciey/RUM-1837-batch-uploading-modification
Commit report: 6f1660e

❌ dd-sdk-ios: 1 Failed (0 Known Flaky), 0 New Flaky, 2705 Passed, 0 Skipped, 25m 13.94s Wall Time

❌ Failed Tests (1)

• testRUMStopSessionScenario - RUMStopSessionScenarioTests

[ncreated]

How will this improvement be measured? During backlog grooming we discussed adding "batch processing level" to "Batch Deleted" telemetry, so we can correlate batch delivery with certain configuration. This part of definition of done seems lacking.

[datadog-datadog-prod-us1]

Datadog Report

Branch report: maciey/RUM-1837-batch-uploading-modification
Commit report: 49400c8

✅ dd-sdk-ios: 0 Failed, 0 New Flaky, 2708 Passed, 0 Skipped, 27m 2.58s Wall Time

[maxep]

Looks great! I see room for improvement in perfs: our DataUploader uses a semaphore which will be blocking the thread. So I think we should balance the load instead of uploading in a loop. I left a suggestion in comment, LMKWYT!

[maciejburda]

@maxep @ncreated
I've made RAM tests using log pressure example from:
#185

10 batches:

develop:

Since files are symlinks turned into data just before upload synchronously - we just see extended allocation, but it's not a spike. In my tests it was way faster to upload everything when using 10-batches - which is a goal of the PR.

Using multiple DispatchWorkItem will be cumbersome as we rely on plenty of different states. I actually explored this solution before jumping into a loop, and I would advice strongly against it. Blocking the thread is in my opinion fine. DataUploadWorker was designed in a way where it's in control of it's own work and since DataUploader is using semaphore it's playing nicely together in a synchronous fashion. Note that we only schedule next DispatchWorkItem after we finish current one.

I think that there is a potential in refactoring this into a OperationQueue, but it will become quite a big change. I think it's better iterate on existing setup first.

[maciejburda]

@maxep @ncreated I refactored the code to use multiple DispatchWorkItem maintaining all the designed behaviours. I added ability to cancel, which can come in handy when we want to restart the flow under certain conditions (i.e. when app goes into background).

Please take another look 🙏

[datadog-datadog-prod-us1]

Datadog Report

Branch report: maciey/RUM-1837-batch-uploading-modification
Commit report: af5ac06

❄️ dd-sdk-ios: 1 Failed (0 Known Flaky), 1 New Flaky, 2355 Passed, 0 Skipped, 4m 56.32s Wall Time

❌ Failed Tests (1)

• testWhenCPUUnderHeavyLoadItMeasuresHigherCPUTicks - VitalCPUReaderTest - Details

  Expand for error

   Assertion Failure at VitalCPUReaderTests.swift:24: XCTAssertGreaterThan failed: ("188.20427311010408") is not greater than ("206.39051796067693")
  

New Flaky Tests (1)

• testWhenDataIsBeingUploaded_itPrintsUploadProgressInformation - DataUploadWorkerTests - Last Failure

  Expand for error

   
   Assertion Failure at DataUploadWorkerTests.swift:375: XCTAssertEqual failed: ("3") is not equal to ("2")
   Assertion Failure at DataUploadWorkerTests.swift:377: XCTAssertEqual failed: ("NTP time synchronization completed.
   Server time will be used for signing events (0.015s difference with device time).") is not equal to ("⏳ (DlKtZ0nALW) Uploading batches...") - Batch start information should be printed to \`userLogger\`. All captured logs:
   [(level: DatadogInternal.CoreLoggerLevel.debug, message: "NTP time synchronization completed.\nServer time will be used for signing events (0.015s difference with device time).", error: nil), (level: DatadogInternal.CoreLoggerLevel.debug, message: "⏳ (DlKtZ0nALW) Uploading batches...", error: nil), (level: DatadogInternal.CoreLoggerLevel.debug, message: "   → (DlKtZ0nALW) not delivered, will be retransmitted: xeKlmUsPWx", error: nil)]
   Assertion Failure at DataUploadWorkerTests.swift:383: XCTAssertEqual failed: ("⏳ (DlKtZ0nALW) Uploading batches...") is not equal to ("   → (DlKtZ0nALW) not delivered, will be retransmitted: xeKlmUsPWx") - Batch completion information should be printed to \`userLogger\`. All captured logs:
   [(level: DatadogInternal.CoreLoggerLevel.debug, message: "NTP time synchronization completed.\nServer time will be used for signing events (0.015s difference with device time).", error: nil), (level: DatadogInternal.CoreLoggerLevel.debug, message: "⏳ (DlKtZ0nALW) Uploading batches...", error: nil), (level: DatadogInternal.CoreLoggerLevel.debug, message: "   → (DlKtZ0nALW) not delivered, will be retransmitted: xeKlmUsPWx", error: nil)]
  

[maxep]

Great work @maciejburda 👏
I've left one minor comment, but it looks great!

PTO

[maxep]

LGTM, thanks for the updates 🙏