Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP] Implement repeat requests on GSuite #153

Closed
wants to merge 7 commits into from
Closed

Conversation

skiptomyliu
Copy link
Member

@skiptomyliu skiptomyliu commented Sep 14, 2019

  • We make a lot of requests to GSuite. This makes it more resilient in case of a Google server blip, which we've seen.
  • Refactor the transform objects method to be more general to prevent copy-pasta.

@skiptomyliu skiptomyliu changed the title retry Implement retry requests on GSuite Sep 14, 2019
@skiptomyliu skiptomyliu changed the title Implement retry requests on GSuite Implement repeat requests on GSuite Sep 14, 2019
@achantavy
Copy link
Contributor

@skiptomyliu @tayasteere Can you please provide more information on the failures? Which data is missing and on which function calls? Do we have stack traces available for these failures? Can we repro?

Would like to more thoroughly understand and document the failure before adding retries.

@skiptomyliu
Copy link
Member Author

We receive 5xx Service Unavailable from Google's API call. Scrubbed stack trace:

{"ts": "2019-09-12T21:35:26.334Z", "uuid": "2fdb2fbf-d6d7-4d47-8472-76e12f0ede90", "app": "infraintelgraph", "name": "cartography.intel.gsuite.api", "lvlname": "WARNING", "lvlno": 30, "pid": 31966, "call_site": "srv/venvs/service/trusty/service_venv_python3.6/lib/python3.6/site-packages/cartography/intel/gsuite/api.py:266", "revision": "e6105806c2", "msg": "HttpError occurred in api.get_all_groups(), returning empty list. Details: <HttpError 503 when requesting https://www.googleapis.com/admin/directory/v1/groups?customer=my_customer&maxResults=20&orderBy=email&alt=json&pageToken=..."}
{"ts": "2019-09-12T21:35:26.337Z", "uuid": "6f25e5d0-30fc-4d36-bb89-11ed6578ec44", "app": "infraintelgraph", "name": "cartography.intel.gsuite.api", "lvlname": "INFO", "lvlno": 20, "pid": 31966, "call_site": "srv/venvs/service/trusty/service_venv_python3.6/lib/python3.6/site-packages/cartography/intel/gsuite/api.py:268", "revision": "e6105806c2", "msg": "Ingesting 0 gsuite groups"}

As soon as that occurs, we break out of the loop and return an empty list. The cleanup script is then run which removes the old entries. I think to make this more resilient we can:

  1. Add retries (what this PR does) because we make a lot of API calls due to the pagination (i'd estimate perhaps multiple hundreds of calls largely due to fetching group membership)
  2. Don't run the cleanup script if empty lists are returned

@ecdavis ecdavis added keep fresh Disables stalebot from closing an issue enhancement labels Sep 20, 2019
@skiptomyliu skiptomyliu changed the title Implement repeat requests on GSuite [WIP] Implement repeat requests on GSuite Sep 26, 2019
@skiptomyliu skiptomyliu added the WIP work in progress label Sep 26, 2019
@skiptomyliu
Copy link
Member Author

skiptomyliu commented Oct 1, 2019

Redoing PR with: #169

CLA signing is borked when I re-configured my new laptop and didn't setup my github config yet.

@skiptomyliu skiptomyliu closed this Oct 1, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
keep fresh Disables stalebot from closing an issue WIP work in progress
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants