Fix check connection for earlier date in Google Analytics #8087

augan-rymkhan · 2021-11-18T09:26:14Z

What

This error happens If a user enters start_date before the date when Google analytics account had the first data.
In check_connection method the current implementation tries to get data for the first date-slice to check the reading permissions.

For example:

Google Analytics account is created on 2021-02-10
User enters 2021-01-01 as start_date, and windows_in_date = 30
3. check_connection method sends a request only for the first slice (2021-01-01 - 2021-01-31)
read_check list is empty because there is no data for that slice. Also there no exception is raised, so check_connection methods returns None
self.check could not unpack None

How

In check_connection read records using test stream TestStreamConnection, which has pag_size = 1 and overriden request_body_json method where we configure request to fetch records from start_date up to now for testing case.
If it returns some valid records, we assume a user has the right permission to read, otherwise we return validation error message.

🚨 User Impact 🚨

Are there any breaking changes? If yes, please make sure to include it here and in any changelogs with the 🚨🚨 emoji
What is the end result perceived by the user?

Pre-merge Checklist

Expand the relevant checklist and delete the others.

New Connector

Community member or Airbyter

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
Credentials added to Github CI. Instructions.
/test connector=connectors/<name> command is passing.
New Connector version released on Dockerhub by running the /publish command described here
After the connector is published, connector added to connector index as described here
Seed specs have been re-generated by building the platform and committing the changes to the seed spec files, as described here

Updating a connector

Community member or Airbyter

Grant edit access to maintainers (instructions)
Secrets in the connector's spec are annotated with airbyte_secret
Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
Code reviews completed
Documentation updated
- Connector's README.md
- Connector's bootstrap.md. See description and examples
- Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
Credentials added to Github CI. Instructions.
/test connector=connectors/<name> command is passing.
New Connector version released on Dockerhub by running the /publish command described here
After the new connector version is published, connector version bumped in the seed directory as described here
Seed specs have been re-generated by building the platform and committing the changes to the seed spec files, as described here

Connector Generator

Issue acceptance criteria met
PR name follows PR naming conventions
If adding a new generator, add it to the list of scaffold modules being tested
The generator test modules (all connectors with -scaffold in their name) have been updated with the latest scaffold by running ./gradlew :airbyte-integrations:connector-templates:generator:testScaffoldTemplates then checking in your changes
Documentation which references the generator is updated as needed.

…ions in `check_connection`

avida

Instead of guessing which slice contains records you could just modify request to specify whole date range from start_date up to now and set pageSize parameter to 1 (instead of 100000). Please read docs for details

augan-rymkhan · 2021-11-18T17:23:01Z

@keu @sergei-solonitcyn @avida
It's not related to this issue. Source Google Analytics v4 fails two acceptance tests in master branch.

1st failed test.
When running acceptance test, abnormal state is start_date = 2050-05-01
end_date = current day
We set the earliest date of them to start_date, it will be equal to end_date, i.e. current day
So for that slice we get some records, but the test case assumes that it should not produce records for future date, 2050-05-01.

assert not records, "The sync should produce no records when run with the state with abnormally large values"
E AssertionError: The sync should produce no records when run with the state with abnormally large values

2nd failed test

E AssertionError: Second incremental sync should produce records older or equal to cursor value from the state. Stream: monthly_active_users
E assert '20211118' >= '20211212'

keu · 2021-11-19T10:42:49Z

Instead of guessing which slice contains records you could just modify request to specify whole date range from start_date up to now and set pageSize parameter to 1 (instead of 100000). Please read docs for details

but this could be a lot of data and has a high chance of timing out...

airbyte-integrations/connectors/source-google-analytics-v4/source_google_analytics_v4/source.py

…r-earlier-date-ga' into arymkhan/fix-check-connection-for-earlier-date-ga

augan-rymkhan · 2021-11-19T11:47:43Z

Instead of guessing which slice contains records you could just modify request to specify whole date range from start_date up to now and set pageSize parameter to 1 (instead of 100000). Please read docs for details

but this could be a lot of data and has a high chance of timing out...

We set page_size = 1 in TestStreamConnection

@keu maybe, we can try this approach?907795d

eliziario

lgtm

airbyte-integrations/connectors/source-google-analytics-v4/source_google_analytics_v4/source.py

sherifnada

Nicely done with adding the extra unit tests!

airbyte-integrations/connectors/source-google-analytics-v4/source_google_analytics_v4/source.py

augan-rymkhan · 2021-11-24T05:15:00Z

/publish connector=connectors/source-google-analytics-v4

🕑 connectors/source-google-analytics-v4 https://github.com/airbytehq/airbyte/actions/runs/1497946039
❌ connectors/source-google-analytics-v4 https://github.com/airbytehq/airbyte/actions/runs/1497946039

augan-rymkhan · 2021-11-24T06:18:48Z

/publish connector=connectors/source-google-analytics-v4

🕑 connectors/source-google-analytics-v4 https://github.com/airbytehq/airbyte/actions/runs/1498096139
❌ connectors/source-google-analytics-v4 https://github.com/airbytehq/airbyte/actions/runs/1498096139
🕑 connectors/source-google-analytics-v4 https://github.com/airbytehq/airbyte/actions/runs/1498096139
❌ connectors/source-google-analytics-v4 https://github.com/airbytehq/airbyte/actions/runs/1498096139

augan-rymkhan · 2021-11-24T08:17:31Z

/publish connector=connectors/source-google-analytics-v4 --run-tests=false

Error: Unexpected inputs provided: ["--run-tests"]

augan-rymkhan · 2021-11-24T08:20:43Z

/publish connector=connectors/source-google-analytics-v4 run-tests=false

🕑 connectors/source-google-analytics-v4 https://github.com/airbytehq/airbyte/actions/runs/1498432378
✅ connectors/source-google-analytics-v4 https://github.com/airbytehq/airbyte/actions/runs/1498432378

…8087) * Changed tests for `check_connection` method * Start reading from the latest date-slice to check the reading permissions in `check_connection` * Fetch records from start_date up to now for testing case * Add minimal valid records into response_with_records.json * Changed version of the connector * formatted codes * Override stream_slices in Test stream * return list of date slices inline * replace valiation msg to "Cannot retrieve data from that view ID" * formatted code * fixed error msg in unit tests * changed version in seed/ yaml files Co-authored-by: Auganbay <auganenu@gmail.com>

Augan93 added 2 commits November 18, 2021 15:20

Changed tests for check_connection method

93ae516

Start reading from the latest date-slice to check the reading permiss…

907795d

…ions in `check_connection`

github-actions bot added the area/connectors Connector related issues label Nov 18, 2021

avida suggested changes Nov 18, 2021

View reviewed changes

Fetch records from start_date up to now for testing case

4c3ea6c

augan-rymkhan temporarily deployed to more-secrets November 19, 2021 05:33 Inactive

Add minimal valid records into response_with_records.json

4b54fd7

augan-rymkhan temporarily deployed to more-secrets November 19, 2021 06:02 Inactive

augan-rymkhan requested review from sergei-solonitcyn, keu, eliziario, vitaliizazmic and avida November 19, 2021 06:08

augan-rymkhan self-assigned this Nov 19, 2021

augan-rymkhan linked an issue Nov 19, 2021 that may be closed by this pull request

Entering start date before data in Google analytics gives error #7901

Closed

augan-rymkhan changed the title ~~Arymkhan/fix check connection for earlier date ga~~ Fix check connection for earlier date in Google Analytics Nov 19, 2021

Changed version of the connector

0b99441

github-actions bot added the area/documentation Improvements or additions to documentation label Nov 19, 2021

augan-rymkhan temporarily deployed to more-secrets November 19, 2021 08:47 Inactive

formatted codes

b28ed71

augan-rymkhan temporarily deployed to more-secrets November 19, 2021 10:49 Inactive

keu reviewed Nov 19, 2021

View reviewed changes

airbyte-integrations/connectors/source-google-analytics-v4/source_google_analytics_v4/source.py Show resolved Hide resolved

Augan93 added 2 commits November 19, 2021 17:41

Override stream_slices in Test stream

d9c29b8

Merge remote-tracking branch 'origin/arymkhan/fix-check-connection-fo…

5463346

…r-earlier-date-ga' into arymkhan/fix-check-connection-for-earlier-date-ga

augan-rymkhan temporarily deployed to more-secrets November 19, 2021 11:44 Inactive

augan-rymkhan requested a review from keu November 19, 2021 12:01

avida approved these changes Nov 19, 2021

View reviewed changes

eliziario approved these changes Nov 22, 2021

View reviewed changes

airbyte-integrations/connectors/source-google-analytics-v4/source_google_analytics_v4/source.py Outdated Show resolved Hide resolved

return list of date slices inline

3db0f17

augan-rymkhan temporarily deployed to more-secrets November 23, 2021 06:06 Inactive

augan-rymkhan requested review from sherifnada and Phlair and removed request for Phlair November 23, 2021 11:51

sherifnada approved these changes Nov 23, 2021

View reviewed changes

airbyte-integrations/connectors/source-google-analytics-v4/source_google_analytics_v4/source.py Outdated Show resolved Hide resolved

replace valiation msg to "Cannot retrieve data from that view ID"

2f2f7dd

augan-rymkhan temporarily deployed to more-secrets November 24, 2021 04:21 Inactive

formatted code

b7fe11e

augan-rymkhan temporarily deployed to more-secrets November 24, 2021 04:40 Inactive

jrhizor temporarily deployed to more-secrets November 24, 2021 05:17 Inactive

fixed error msg in unit tests

b758838

augan-rymkhan temporarily deployed to more-secrets November 24, 2021 05:48 Inactive

jrhizor temporarily deployed to more-secrets November 24, 2021 06:21 Inactive

jrhizor temporarily deployed to more-secrets November 24, 2021 06:36 Inactive

jrhizor temporarily deployed to more-secrets November 24, 2021 08:23 Inactive

changed version in seed/ yaml files

fd74573

augan-rymkhan temporarily deployed to more-secrets November 24, 2021 08:38 Inactive

augan-rymkhan merged commit 82775d6 into master Nov 24, 2021

augan-rymkhan deleted the arymkhan/fix-check-connection-for-earlier-date-ga branch November 24, 2021 09:07

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix check connection for earlier date in Google Analytics #8087

Fix check connection for earlier date in Google Analytics #8087

augan-rymkhan commented Nov 18, 2021 •

edited

Loading

avida left a comment •

edited

Loading

augan-rymkhan commented Nov 18, 2021 •

edited

Loading

keu commented Nov 19, 2021 •

edited

Loading

augan-rymkhan commented Nov 19, 2021 •

edited

Loading

eliziario left a comment

sherifnada left a comment

augan-rymkhan commented Nov 24, 2021 •

edited by github-actions bot

Loading

augan-rymkhan commented Nov 24, 2021 •

edited by github-actions bot

Loading

augan-rymkhan commented Nov 24, 2021 •

edited

Loading

augan-rymkhan commented Nov 24, 2021 •

edited by github-actions bot

Loading

Fix check connection for earlier date in Google Analytics #8087

Fix check connection for earlier date in Google Analytics #8087

Conversation

augan-rymkhan commented Nov 18, 2021 • edited Loading

What

How

Recommended reading order

🚨 User Impact 🚨

Pre-merge Checklist

Community member or Airbyter

Airbyter

Community member or Airbyter

Airbyter

avida left a comment • edited Loading

Choose a reason for hiding this comment

augan-rymkhan commented Nov 18, 2021 • edited Loading

keu commented Nov 19, 2021 • edited Loading

augan-rymkhan commented Nov 19, 2021 • edited Loading

eliziario left a comment

Choose a reason for hiding this comment

sherifnada left a comment

Choose a reason for hiding this comment

augan-rymkhan commented Nov 24, 2021 • edited by github-actions bot Loading

augan-rymkhan commented Nov 24, 2021 • edited by github-actions bot Loading

augan-rymkhan commented Nov 24, 2021 • edited Loading

augan-rymkhan commented Nov 24, 2021 • edited by github-actions bot Loading

augan-rymkhan commented Nov 18, 2021 •

edited

Loading

avida left a comment •

edited

Loading

augan-rymkhan commented Nov 18, 2021 •

edited

Loading

keu commented Nov 19, 2021 •

edited

Loading

augan-rymkhan commented Nov 19, 2021 •

edited

Loading

augan-rymkhan commented Nov 24, 2021 •

edited by github-actions bot

Loading

augan-rymkhan commented Nov 24, 2021 •

edited by github-actions bot

Loading

augan-rymkhan commented Nov 24, 2021 •

edited

Loading

augan-rymkhan commented Nov 24, 2021 •

edited by github-actions bot

Loading