ENG-55392-Googlestackdriver collector: Handle the throttle error for pagination scenario

[rakhimundhada15]

Problem Description

During throttling, we increase the poll interval and decrease the time duration. However, when the collector is collecting data through pagination, it already sets the filter timestamp (timestamp >= since and timestamp < until). The actual until value changes in the state during throttling. Once pagination completes and the collector creates a new state, it starts again from the same timestamp, resulting in duplicate data collection.

ex. In below state example until value(i.e 2024-05-25T10:39:49.000Z) in filter and actual(i.e 2024-05-24T10:39:50.000Z) is different.
{"priv_collector_state":{"since":"2024-05-24T10:39:49.000Z","nextPage":{"filter":"timestamp >= \"2024-05-24T10:39:49.000Z\"\ntimestamp < \"2024-05-25T10:39:49.000Z\"","pageSize":1000,"resourceNames":["projects/prj-mstr-prod"],"pageToken":"EAA44cC8vNq0xP5RSq "},"stream":"projects/prj-mstr-prod","until":"2024-05-24T10:39:50.000Z","poll_interval_sec":420},"retry_count":0}

Solution Description

Now we just reduce the page size during pagination and not the until value.

[al-cibot]

CI Service Job Number 985 Logs - SUCCESS

alps ci job-logs --organization alertlogic --repository paws-collector --job-num 985 -g

Stages Run:

• PR Test

[rakhimundhada15]

build-collector-googlestackdriver

[al-cibot]

CI Service Job Number 986 Logs - SUCCESS

alps ci job-logs --organization alertlogic --repository paws-collector --job-num 986 -g

Stages Run:

• build-collector-googlestackdriver
• file_packager_36834a