[Google Workspace] Cursor is not set correctly in httpjson.yml.hbs

[tlee-elastic]

Hi Team,

The cursor is not set correctly in the httpjson.yml.hbs.

cursor:
  last_execution_datetime:
    value: "[[formatDate now]]"

When the agent restarts, it will continue from the current time instead of progressing from the last API response time. This causes an outage as the agent cannot automatically backfill the data from the last successful document.

[elasticmachine]

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

[andrewkroh]

@marc-gr Do you recall why "execution time" was used as opposed to a timestamp taken from the last received event?

integrations/packages/google_workspace/data_stream/admin/agent/stream/httpjson.yml.hbs

Lines 28 to 30 in 8bebafa

	cursor:
	last_execution_datetime:
	value: "[[formatDate now]]"

[andrewkroh]

When the agent restarts, it will continue from the current time instead of progressing from the last API response time.

@tlee-elastic It should be resuming by using the execution time of the last request (as was persisted to disk). If that state was lost (or it was the first time the integration runs) then by default it will use now - 24h. Does your Agent container have persistent storage mounted such that on restart it retains the persisted state?

[tlee-elastic]

@andrewkroh from what I can see in the logs, when the google workspace integration paginates, it still updates the cursor with the execution time. If there is an agent restart/disruption during this time. When it resumes, it continues from the cursor time and results in missing data. Logs below:

cursor.last_execution_datetime stored with 2022-10-24T01:32:59Z

last received page: &httpjson.response{page:15, url:url.URL{Scheme:"https", Opaque:"", User:(*url.Userinfo)(nil), Host:"www.googleapis.com", Path:"/admin/reports/v1/activity/users/all/applications/drive", RawPath:"", ForceQuery:false, RawQuery:"pageToken=<REDACTED>&startTime=2022-10-24T01%3A03%3A49Z", Fragment:"", RawFragment:""}

You can see that the start time in the pagination is still 2022-10-24T01:03:49Z but the cursor has already been updated to 2022-10-24T01:32:59Z

[marc-gr]

When the agent restarts, it will continue from the current time instead of progressing from the last API response time.

@tlee-elastic It should be resuming by using the execution time of the last request (as was persisted to disk). If that state was lost (or it was the first time the integration runs) then by default it will use now - 24h. Does your Agent container have persistent storage mounted such that on restart it retains the persisted state?

I do not remember any specific reason, and indeed it seems like using the provided time in the events would result in a more robust solution.

[marc-gr]

When the agent restarts, it will continue from the current time instead of progressing from the last API response time.

@tlee-elastic It should be resuming by using the execution time of the last request (as was persisted to disk). If that state was lost (or it was the first time the integration runs) then by default it will use now - 24h. Does your Agent container have persistent storage mounted such that on restart it retains the persisted state?

I do not remember any specific reason, and indeed it seems like using the provided time in the events would result in a more robust solution.

Answering to myself. Going through the integration I remember why I did not use the id.time field. I saw different date formats for the time from several samples, and hence supporting these all in the httpjson config would be a bit brittle and cumbersome. I can explore it again and amend it, as I indeed think would be better under a failing scenario like the proposed one.

[botelastic]

Hi! We just realized that we haven't looked into this issue in a while. We're sorry! We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!

[geekpete]

Still relevant 👍

[kcreddy]

@geekpete The PR #4982 from @marc-gr was supposed to fix this problem using new cursor state.
Can you give more context into the issue you are facing? Which version of the integration and datastream you are using and also if it is the same issue (i.e., incorrect cursor state leading to data loss)?

[andrewkroh]

This seems like it was a duplicate of #4796 and it should have been closed along with it.

[kcreddy]

Closed as duplicate.