Source Google Analytics (GA4): Fix Pagination #26126

mpetrykin · 2023-05-16T16:59:32Z

What

Fix the pagination mechanism. Problem is described in 26066 issue.

How

Add next_page_offset property to report config, that increase in condition current next_page_offset or first offset less than rowCount from response.
Add offset and limit fields to request body parameters that allowed users define them (it is possible from documentation).

…request body

…_pagination Source google analytics data api: fix pagination

github-actions · 2023-05-16T16:59:55Z

Before Merging a Connector Pull Request

Wow! What a great pull request you have here! 🎉

To merge this PR, ensure the following has been done/considered for each connector added or updated:

PR name follows PR naming conventions
Breaking changes are considered. If a Breaking Change is being introduced, ensure an Airbyte engineer has created a Breaking Change Plan and you've followed all steps in the Breaking Changes Checklist
Connector version has been incremented in the Dockerfile and metadata.yaml according to our Semantic Versioning for Connectors guidelines
Secrets in the connector's spec are annotated with airbyte_secret
All documentation files are up to date. (README.md, bootstrap.md, docs.md, etc...)
Changelog updated in docs/integrations/<source or destination>/<name>.md with an entry for the new version. See changelog example
You, or an Airbyter, have run /test successfully on this PR - or on a non-forked branch
You, or an Airbyter, have run /publish successfully on this PR - or on a non-forked branch
You've updated the connector's metadata.yaml file new!

If the checklist is complete, but the CI check is failing,

Check for hidden checklists in your PR description
Toggle the github label checklist-action-run on/off to re-run the checklist CI.

...ogle-analytics-data-api/source_google_analytics_data_api/defaults/custom_reports_schema.json

docs/integrations/sources/google-analytics-data-api.md

bazarnov

Another change is needed, we should not have the multi-types in JSON Schema, as this will cause issues during data normalization, please refer to the comments bellow.

...ogle-analytics-data-api/source_google_analytics_data_api/defaults/custom_reports_schema.json

airbyte-integrations/connectors/source-google-analytics-data-api/unit_tests/test_streams.py

bazarnov · 2023-05-18T11:51:27Z

/test connector=connectors/source-google-analytics-data-api

🕑 connectors/source-google-analytics-data-api https://github.com/airbytehq/airbyte/actions/runs/5013631854
❌ connectors/source-google-analytics-data-api https://github.com/airbytehq/airbyte/actions/runs/5013631854
🐛 https://gradle.com/s/zatfaci7dtc3w

Build Failed

Test summary info:

=========================== short test summary info ============================
FAILED test_core.py::TestConnection::test_check[inputs0] - docker.errors.Cont...
FAILED test_core.py::TestBasicRead::test_read[inputs0] - docker.errors.Contai...
FAILED test_full_refresh.py::TestFullRefresh::test_sequential_reads[inputs0]
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_core.py:100: The previous and actual specifications are identical.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_core.py:578: The previous and actual discovered catalogs are identical.
======== 3 failed, 35 passed, 2 skipped, 1 warning in 828.97s (0:13:48) ========

bazarnov · 2023-05-18T11:52:58Z

@marcosmarxm I think we're good to go with this contribution, once CAT is passed.

...tions/connectors/source-google-analytics-data-api/source_google_analytics_data_api/source.py

marcosmarxm

Added some comments @mpetrykin

...tions/connectors/source-google-analytics-data-api/source_google_analytics_data_api/source.py

airbyte-integrations/connectors/source-google-analytics-data-api/unit_tests/test_streams.py

marcosmarxm · 2023-05-23T15:09:21Z

/test connector=connectors/source-google-analytics-data-api

🕑 connectors/source-google-analytics-data-api https://github.com/airbytehq/airbyte/actions/runs/5059002651
❌ connectors/source-google-analytics-data-api https://github.com/airbytehq/airbyte/actions/runs/5059002651
🐛 https://gradle.com/s/yw6txj3sn5kfk

Build Failed

Test summary info:

=========================== short test summary info ============================
FAILED test_core.py::TestConnection::test_check[inputs0] - docker.errors.Cont...
FAILED test_core.py::TestBasicRead::test_read[inputs0] - docker.errors.Contai...
FAILED test_full_refresh.py::TestFullRefresh::test_sequential_reads[inputs0]
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_core.py:100: The previous and actual specifications are identical.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_core.py:578: The previous and actual discovered catalogs are identical.
======== 3 failed, 35 passed, 2 skipped, 1 warning in 968.73s (0:16:08) ========

mpetrykin · 2023-05-24T08:42:20Z

/test connector=connectors/source-google-analytics-data-api

marcosmarxm

Some nit comments. I'll run tests locally and check what is not working

...tions/connectors/source-google-analytics-data-api/source_google_analytics_data_api/source.py

airbyte-integrations/connectors/source-google-analytics-data-api/unit_tests/test_streams.py

mpetrykin · 2023-05-29T19:48:44Z

@marcosmarxm hi, sorry to ask, did you manage to run the tests locally and find the problem or haven't had time yet?

marcosmarxm · 2023-05-30T11:47:43Z

Not yet @mpetrykin I'm going to test and run today.

sh4sh · 2023-05-31T16:59:44Z

/test connector=connectors/source-google-analytics-data-api

🕑 connectors/source-google-analytics-data-api https://github.com/airbytehq/airbyte/actions/runs/5135617899
✅ connectors/source-google-analytics-data-api https://github.com/airbytehq/airbyte/actions/runs/5135617899
Python tests coverage:

Name                                                Stmts   Miss  Cover
-----------------------------------------------------------------------
source_google_analytics_data_api/__init__.py            2      0   100%
source_google_analytics_data_api/authenticator.py      44      2    95%
source_google_analytics_data_api/utils.py              26      2    92%
source_google_analytics_data_api/source.py            247     31    87%
source_google_analytics_data_api/api_quota.py          94     12    87%
-----------------------------------------------------------------------
TOTAL                                                 413     47    89%

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_core.py:100: The previous and actual specifications are identical.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_core.py:594: The previous and actual discovered catalogs are identical.
============ 39 passed, 2 skipped, 1 warning in 1102.76s (0:18:22) =============

changes resolved

* Fix paggination and add offset and limit to acceptable parameters in request body * Change next_page_token and add tests * Update dockerImageTag * Update PR version * Remove minimum, maximum, pattern fields * Remove pattern limit and offset from test_source.py * Remove offset and limit string type * Remove offset and limit string type * Increase limit number to 100000 and remove limit and offset from parameters * Change return type value of next_page_token from int to dict * Change return type value of next_page_token from int to dict * Change page_size to offset and add constant PAGE_SIZE equals 100000 * Add comment to PAGE_SIZE constant and add constant to unit tests * Remove offset and limit from PivotReport * Import PAGE_SIZE in unit_tests from source.py --------- Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com> Co-authored-by: sh4sh <6833405+sh4sh@users.noreply.github.com>

mpetrykin and others added 4 commits May 16, 2023 19:43

Fix paggination and add offset and limit to acceptable parameters in …

a3e5e09

…request body

Change next_page_token and add tests

cc7c532

Update dockerImageTag

5e7c567

Merge pull request #3 from mpetrykin/source-google-analytics-data-api…

ab04903

…_pagination Source google analytics data api: fix pagination

octavia-squidington-iii added area/connectors Connector related issues area/documentation Improvements or additions to documentation bounty community connectors/source/google-analytics-data-api labels May 16, 2023

Merge branch 'master' into master

0390ba4

marcosmarxm requested a review from artem1205 May 16, 2023 17:22

airbytehq deleted a comment from mpetrykin May 16, 2023

marcosmarxm added the team/connectors-python label May 17, 2023

bazarnov requested changes May 17, 2023

View reviewed changes

...ogle-analytics-data-api/source_google_analytics_data_api/defaults/custom_reports_schema.json Outdated Show resolved Hide resolved

docs/integrations/sources/google-analytics-data-api.md Outdated Show resolved Hide resolved

mpetrykin added 3 commits May 17, 2023 22:42

Update PR version

7b66859

Remove minimum, maximum, pattern fields

42c5061

Remove pattern limit and offset from test_source.py

e321d3b

mpetrykin requested review from bazarnov May 17, 2023 19:51

bazarnov requested changes May 17, 2023

View reviewed changes

mpetrykin added 2 commits May 17, 2023 23:14

Remove offset and limit string type

d106d4b

Remove offset and limit string type

7fd5623

bazarnov assigned marcosmarxm, bazarnov and mpetrykin May 18, 2023

bazarnov approved these changes May 18, 2023

View reviewed changes

grubberr previously requested changes May 21, 2023

View reviewed changes

...tions/connectors/source-google-analytics-data-api/source_google_analytics_data_api/source.py Outdated Show resolved Hide resolved

...tions/connectors/source-google-analytics-data-api/source_google_analytics_data_api/source.py Outdated Show resolved Hide resolved

mpetrykin added 2 commits May 22, 2023 17:20

Change return type value of next_page_token from int to dict

a845ee8

Change return type value of next_page_token from int to dict

2252ac8

marcosmarxm requested changes May 23, 2023

View reviewed changes

...tions/connectors/source-google-analytics-data-api/source_google_analytics_data_api/source.py Outdated Show resolved Hide resolved

Change page_size to offset and add constant PAGE_SIZE equals 100000

58513f6

mpetrykin requested a review from marcosmarxm May 23, 2023 14:22

marcosmarxm requested changes May 23, 2023

View reviewed changes

Add comment to PAGE_SIZE constant and add constant to unit tests

c24d400

mpetrykin requested a review from marcosmarxm May 23, 2023 15:36

Remove offset and limit from PivotReport

90f21f5

bazarnov removed their assignment May 24, 2023

marcosmarxm previously requested changes May 24, 2023

View reviewed changes

...tions/connectors/source-google-analytics-data-api/source_google_analytics_data_api/source.py Outdated Show resolved Hide resolved

airbyte-integrations/connectors/source-google-analytics-data-api/unit_tests/test_streams.py Outdated Show resolved Hide resolved

Import PAGE_SIZE in unit_tests from source.py

b21308b

mpetrykin requested a review from marcosmarxm May 24, 2023 15:28

Merge branch 'master' into master

1ba4dc0

jrolom removed the bounty label May 29, 2023

sh4sh added 2 commits May 31, 2023 09:19

Merge branch 'master' into master

566c25f

Merge branch 'master' into master

1811337

sh4sh mentioned this pull request May 31, 2023

26126/checks #26857

Closed

sh4sh self-requested a review May 31, 2023 17:57

sh4sh approved these changes May 31, 2023

View reviewed changes

sh4sh merged commit 746ccb4 into airbytehq:master May 31, 2023
52 of 68 checks passed

mpetrykin mentioned this pull request Jun 1, 2023

Source Google Analytics (GA4): Fix Pagination #26066

Closed

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Source Google Analytics (GA4): Fix Pagination #26126

Source Google Analytics (GA4): Fix Pagination #26126

mpetrykin commented May 16, 2023

github-actions bot commented May 16, 2023 •

edited by sh4sh

Loading

bazarnov left a comment

bazarnov commented May 18, 2023 •

edited by github-actions bot

Loading

bazarnov commented May 18, 2023

marcosmarxm left a comment

marcosmarxm commented May 23, 2023 •

edited by github-actions bot

Loading

mpetrykin commented May 24, 2023

marcosmarxm left a comment

mpetrykin commented May 29, 2023

marcosmarxm commented May 30, 2023

sh4sh commented May 31, 2023 •

edited by github-actions bot

Loading

Source Google Analytics (GA4): Fix Pagination #26126

Source Google Analytics (GA4): Fix Pagination #26126

Conversation

mpetrykin commented May 16, 2023

What

How

github-actions bot commented May 16, 2023 • edited by sh4sh Loading

Before Merging a Connector Pull Request

bazarnov left a comment

Choose a reason for hiding this comment

bazarnov commented May 18, 2023 • edited by github-actions bot Loading

Build Failed

bazarnov commented May 18, 2023

marcosmarxm left a comment

Choose a reason for hiding this comment

marcosmarxm commented May 23, 2023 • edited by github-actions bot Loading

Build Failed

mpetrykin commented May 24, 2023

marcosmarxm left a comment

Choose a reason for hiding this comment

mpetrykin commented May 29, 2023

marcosmarxm commented May 30, 2023

sh4sh commented May 31, 2023 • edited by github-actions bot Loading

Build Passed

github-actions bot commented May 16, 2023 •

edited by sh4sh

Loading

bazarnov commented May 18, 2023 •

edited by github-actions bot

Loading

marcosmarxm commented May 23, 2023 •

edited by github-actions bot

Loading

sh4sh commented May 31, 2023 •

edited by github-actions bot

Loading