Skip to content

Need to recalculate remaining if there's an error #365

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
michael-richey merged 6 commits into from
Aug 15, 2025
Merged

Need to recalculate remaining if there's an error #365

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
aa0efcd
Need to recalculate remaining if there's an error
michael-richey Aug 13, 2025
72db19b
Need to handle no during an error
michael-richey Aug 14, 2025
de150da
Black
michael-richey Aug 14, 2025
89c835a
Handle a 404 or other errors
michael-richey Aug 14, 2025
9a83b91
Public API changed
michael-richey Aug 15, 2025
2c08167
Cassettes
michael-richey Aug 15, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
1 change: 1 addition & 0 deletions datadog_sync/model/users.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,7 @@ class Users(BaseResource):
"attributes.icon",
"attributes.modified_at",
"attributes.mfa_enabled",
"attributes.allowed_login_methods",
"relationships.org",
],
)
Expand Down
8 changes: 7 additions & 1 deletion datadog_sync/utils/custom_client.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -134,14 +134,16 @@ async def wrapper(*args, **kwargs):
resources_attempted = 0
saved_idx = idx
save_idx = True

while remaining > 0:
remaining = 0
log.debug(
f"fetching {args[0]} "
f"{pagination_config.page_number_param}: {page_number} "
f"{pagination_config.page_size_param}: {page_size} "
f"idx: {idx} "
f"remaining: {remaining}"
)
remaining = 0
params = {
pagination_config.page_size_param: page_size,
pagination_config.page_number_param: page_number,
Expand Down Expand Up @@ -214,6 +216,10 @@ async def wrapper(*args, **kwargs):
page_size = new_page_size
idx = resources_attempted // page_size - 1
page_number = pagination_config.page_number_func(idx, page_size, page_number)
remaining = 1 # after this error we need to keep processing
else:
log.error(f"Error during paginated request: {err}")
break

# made it through the try/except no increase the page number and idx
page_number = pagination_config.page_number_func(idx, page_size, page_number)
Expand Down
2 changes: 1 addition & 1 deletion tests/integration/cassettes/test_cli/TestCli.test_cleanup.frozen
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1 +1 @@
2025-08-13T09:51:22.867088-04:00
2025-08-15T11:49:50.182909-04:00
1,827 changes: 987 additions & 840 deletions tests/integration/cassettes/test_cli/TestCli.test_cleanup.yaml
View file
Open in desktop

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion tests/integration/cassettes/test_cli/TestCli.test_import.frozen
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1 +1 @@
2025-08-13T09:50:47.140755-04:00
2025-08-15T11:49:13.708113-04:00
4,527 changes: 2,353 additions & 2,174 deletions tests/integration/cassettes/test_cli/TestCli.test_import.yaml
View file
Open in desktop

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion tests/integration/cassettes/test_cli/TestCli.test_import_without_verify_ddr_status.frozen
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1 +1 @@
2025-08-13T09:51:00.290151-04:00
2025-08-15T11:49:29.683592-04:00
12,200 changes: 6,740 additions & 5,460 deletions tests/integration/cassettes/test_cli/TestCli.test_import_without_verify_ddr_status.yaml
View file
Open in desktop

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion tests/integration/cassettes/test_cli/TestCli.test_migrate.frozen
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1 +1 @@
2025-08-13T09:51:15.824945-04:00
2025-08-15T11:49:43.286023-04:00
8,001 changes: 4,847 additions & 3,154 deletions tests/integration/cassettes/test_cli/TestCli.test_migrate.yaml
View file
Open in desktop

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion tests/integration/cassettes/test_cli/TestCli.test_migrate_without_verify_ddr_status.frozen
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1 +1 @@
2025-08-13T09:51:07.594752-04:00
2025-08-15T11:49:36.497240-04:00
7,704 changes: 4,709 additions & 2,995 deletions tests/integration/cassettes/test_cli/TestCli.test_migrate_without_verify_ddr_status.yaml
View file
Open in desktop

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion tests/integration/cassettes/test_cli/TestCli.test_sync.frozen
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1 +1 @@
2025-08-13T09:50:51.388071-04:00
2025-08-15T11:49:19.251650-04:00
6,951 changes: 3,712 additions & 3,239 deletions tests/integration/cassettes/test_cli/TestCli.test_sync.yaml
View file
Open in desktop

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion tests/integration/cassettes/test_cli/TestCli.test_sync_without_verify_ddr_status.frozen
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1 +1 @@
2025-08-13T09:51:01.877455-04:00
2025-08-15T11:49:31.324206-04:00
4,225 changes: 2,352 additions & 1,873 deletions tests/integration/cassettes/test_cli/TestCli.test_sync_without_verify_ddr_status.yaml
View file
Open in desktop

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion tests/integration/cassettes/test_cli/TestCli.test_verify_ddr_status_failure.frozen
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1 +1 @@
2025-08-13T09:50:58.514402-04:00
2025-08-15T11:49:27.972074-04:00
2 changes: 1 addition & 1 deletion ...es/cassettes/test_authn_mappings/TestAuthNMappingsResources.test_no_resource_diffs.frozen
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1 +1 @@
2025-08-13T09:35:09.214220-04:00
2025-08-15T11:31:18.286732-04:00
978 changes: 543 additions & 435 deletions ...rces/cassettes/test_authn_mappings/TestAuthNMappingsResources.test_no_resource_diffs.yaml
View file
Open in desktop

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion ...ces/cassettes/test_authn_mappings/TestAuthNMappingsResources.test_resource_cleanup.frozen
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1 +1 @@
2025-08-13T09:35:10.584520-04:00
2025-08-15T11:31:21.412490-04:00
1,956 changes: 1,086 additions & 870 deletions ...urces/cassettes/test_authn_mappings/TestAuthNMappingsResources.test_resource_cleanup.yaml
View file
Open in desktop

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion ...rces/cassettes/test_authn_mappings/TestAuthNMappingsResources.test_resource_import.frozen
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1 +1 @@
2025-08-13T09:34:59.978351-04:00
2025-08-15T11:31:09.784117-04:00
1,118 changes: 635 additions & 483 deletions ...ources/cassettes/test_authn_mappings/TestAuthNMappingsResources.test_resource_import.yaml
View file
Open in desktop

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion ...ettes/test_authn_mappings/TestAuthNMappingsResources.test_resource_import_per_file.frozen
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1 +1 @@
2025-08-13T09:35:13.414541-04:00
2025-08-15T11:31:24.415026-04:00
978 changes: 543 additions & 435 deletions ...ssettes/test_authn_mappings/TestAuthNMappingsResources.test_resource_import_per_file.yaml
View file
Open in desktop

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion ...ources/cassettes/test_authn_mappings/TestAuthNMappingsResources.test_resource_sync.frozen
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1 +1 @@
2025-08-13T09:35:02.098024-04:00
2025-08-15T11:31:11.861120-04:00
1,123 changes: 630 additions & 493 deletions ...esources/cassettes/test_authn_mappings/TestAuthNMappingsResources.test_resource_sync.yaml
View file
Open in desktop

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion ...ssettes/test_authn_mappings/TestAuthNMappingsResources.test_resource_sync_per_file.frozen
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1 +1 @@
2025-08-13T09:35:15.905710-04:00
2025-08-15T11:31:26.943486-04:00
1,123 changes: 630 additions & 493 deletions ...cassettes/test_authn_mappings/TestAuthNMappingsResources.test_resource_sync_per_file.yaml
View file
Open in desktop

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion ...cassettes/test_authn_mappings/TestAuthNMappingsResources.test_resource_update_sync.frozen
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1 +1 @@
2025-08-13T09:35:05.882204-04:00
2025-08-15T11:31:14.972556-04:00
1,133 changes: 635 additions & 498 deletions ...s/cassettes/test_authn_mappings/TestAuthNMappingsResources.test_resource_update_sync.yaml
View file
Open in desktop

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion .../test_authn_mappings/TestAuthNMappingsResources.test_resource_update_sync_per_file.frozen
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1 +1 @@
2025-08-13T09:35:19.135410-04:00
2025-08-15T11:31:30.236403-04:00
1,133 changes: 635 additions & 498 deletions ...es/test_authn_mappings/TestAuthNMappingsResources.test_resource_update_sync_per_file.yaml
View file
Open in desktop

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion .../cassettes/test_dashboard_lists/TestDashboardListsResources.test_no_resource_diffs.frozen
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1 +1 @@
2025-08-13T09:35:33.059195-04:00
2025-08-15T11:31:42.851361-04:00
723 changes: 393 additions & 330 deletions ...es/cassettes/test_dashboard_lists/TestDashboardListsResources.test_no_resource_diffs.yaml
View file
Open in desktop

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion ...s/cassettes/test_dashboard_lists/TestDashboardListsResources.test_resource_cleanup.frozen
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1 +1 @@
2025-08-13T09:35:34.082313-04:00
2025-08-15T11:31:43.871857-04:00
1,446 changes: 786 additions & 660 deletions ...ces/cassettes/test_dashboard_lists/TestDashboardListsResources.test_resource_cleanup.yaml
View file
Open in desktop

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion ...es/cassettes/test_dashboard_lists/TestDashboardListsResources.test_resource_import.frozen
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1 +1 @@
2025-08-13T09:35:23.579871-04:00
2025-08-15T11:31:34.613605-04:00
723 changes: 393 additions & 330 deletions ...rces/cassettes/test_dashboard_lists/TestDashboardListsResources.test_resource_import.yaml
View file
Open in desktop

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion ...tes/test_dashboard_lists/TestDashboardListsResources.test_resource_import_per_file.frozen
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1 +1 @@
2025-08-13T09:35:36.363527-04:00
2025-08-15T11:31:45.851200-04:00
723 changes: 393 additions & 330 deletions ...ettes/test_dashboard_lists/TestDashboardListsResources.test_resource_import_per_file.yaml
View file
Open in desktop

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion ...rces/cassettes/test_dashboard_lists/TestDashboardListsResources.test_resource_sync.frozen
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1 +1 @@
2025-08-13T09:35:25.014597-04:00
2025-08-15T11:31:36.104212-04:00
Loading
Loading