🐛 Source Delighted: Fix pagination for `survey_responses`, `bounces` and `unsubscribes` streams #9275

Zirochkaa · 2022-01-04T10:37:37Z

Bug-fix Request

Reason

Right now pagination handling for survey_responses, bounces and unsubscribes streams works incorrectly.
Current implementation is the following: if we got Link response header then we extract page_info parameter from it and send it in the next request (like, page_info=f54r54r4r). If there is no Link response header then all data was read. This behavior works only for people stream.

Confirmation

Was reproduced locally

How does the code change in the PR fix the issue?

For survey_responses, bounces and unsubscribes streams pagination handling should be following: send page parameter with number of page in it (like, page=1, page=10 etc).

Zirochkaa · 2022-01-04T11:09:24Z

/test connector=source-delighted

🕑 source-delighted https://github.com/airbytehq/airbyte/actions/runs/1653330935
✅ source-delighted https://github.com/airbytehq/airbyte/actions/runs/1653330935
Python tests coverage:

	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                                 Stmts   Miss  Cover
	 ------------------------------------------------------------------------
	 source_acceptance_test/__init__.py                       2      0   100%
	 source_acceptance_test/base.py                          10      4    60%
	 source_acceptance_test/config.py                        74      6    92%
	 source_acceptance_test/conftest.py                     109    109     0%
	 source_acceptance_test/plugin.py                        47     47     0%
	 source_acceptance_test/tests/__init__.py                 4      0   100%
	 source_acceptance_test/tests/test_core.py              242     96    60%
	 source_acceptance_test/tests/test_full_refresh.py       38      0   100%
	 source_acceptance_test/tests/test_incremental.py        69     38    45%
	 source_acceptance_test/utils/__init__.py                 6      0   100%
	 source_acceptance_test/utils/asserts.py                 37      2    95%
	 source_acceptance_test/utils/common.py                  54     17    69%
	 source_acceptance_test/utils/compare.py                 62     23    63%
	 source_acceptance_test/utils/connector_runner.py       110     48    56%
	 source_acceptance_test/utils/json_schema_helper.py     115     14    88%
	 ------------------------------------------------------------------------
	 TOTAL                                                  979    404    59%
	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                       Stmts   Miss  Cover
	 --------------------------------------------------------------
	 base_python/__init__.py                       13      0   100%
	 base_python/catalog_helpers.py                10      6    40%
	 base_python/cdk/__init__.py                    0      0   100%
	 base_python/cdk/abstract_source.py            89     64    28%
	 base_python/cdk/streams/__init__.py            0      0   100%
	 base_python/cdk/streams/auth/__init__.py       0      0   100%
	 base_python/cdk/streams/auth/core.py           8      1    88%
	 base_python/cdk/streams/auth/jwt.py            5      5     0%
	 base_python/cdk/streams/auth/oauth.py         37     26    30%
	 base_python/cdk/streams/auth/token.py          9      4    56%
	 base_python/cdk/streams/core.py               63     32    49%
	 base_python/cdk/streams/exceptions.py         10      2    80%
	 base_python/cdk/streams/http.py               67     33    51%
	 base_python/cdk/streams/rate_limiting.py      30     14    53%
	 base_python/cdk/utils/__init__.py              0      0   100%
	 base_python/cdk/utils/casing.py                4      0   100%
	 base_python/cdk/utils/event_timing.py         47      3    94%
	 base_python/client.py                         56     33    41%
	 base_python/entrypoint.py                     70     56    20%
	 base_python/integration.py                    52     25    52%
	 base_python/logger.py                         33     15    55%
	 base_python/schema_helpers.py                 56     41    27%
	 base_python/source.py                         51     34    33%
	 main_dev.py                                    3      3     0%
	 --------------------------------------------------------------
	 TOTAL                                        713    397    44%
	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                           Stmts   Miss  Cover
	 --------------------------------------------------
	 source_delighted/__init__.py       2      2     0%
	 source_delighted/source.py        92     92     0%
	 --------------------------------------------------
	 TOTAL                             94     94     0%

airbyte-integrations/connectors/source-delighted/source_delighted/schemas/survey_responses.json

docs/integrations/sources/delighted.md

airbyte-integrations/connectors/source-delighted/source_delighted/source.py

yevhenii-ldv · 2022-01-04T12:54:43Z

airbyte-integrations/connectors/source-delighted/source_delighted/source.py

-        else:
-            return None
+        response_data = response.json()
+        if response_data and len(response_data) == self.limit:


Can response_data be not an array?

No, it'll always an array.

yevhenii-ldv

A little changes

Zirochkaa · 2022-01-04T13:57:30Z

/test connector=source-delighted

🕑 source-delighted https://github.com/airbytehq/airbyte/actions/runs/1653927250
✅ source-delighted https://github.com/airbytehq/airbyte/actions/runs/1653927250
Python tests coverage:

	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                                 Stmts   Miss  Cover
	 ------------------------------------------------------------------------
	 source_acceptance_test/__init__.py                       2      0   100%
	 source_acceptance_test/base.py                          10      4    60%
	 source_acceptance_test/config.py                        74      6    92%
	 source_acceptance_test/conftest.py                     109    109     0%
	 source_acceptance_test/plugin.py                        47     47     0%
	 source_acceptance_test/tests/__init__.py                 4      0   100%
	 source_acceptance_test/tests/test_core.py              242     96    60%
	 source_acceptance_test/tests/test_full_refresh.py       38      0   100%
	 source_acceptance_test/tests/test_incremental.py        69     38    45%
	 source_acceptance_test/utils/__init__.py                 6      0   100%
	 source_acceptance_test/utils/asserts.py                 37      2    95%
	 source_acceptance_test/utils/common.py                  54     17    69%
	 source_acceptance_test/utils/compare.py                 62     23    63%
	 source_acceptance_test/utils/connector_runner.py       110     48    56%
	 source_acceptance_test/utils/json_schema_helper.py     115     14    88%
	 ------------------------------------------------------------------------
	 TOTAL                                                  979    404    59%
	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                       Stmts   Miss  Cover
	 --------------------------------------------------------------
	 base_python/__init__.py                       13      0   100%
	 base_python/catalog_helpers.py                10      6    40%
	 base_python/cdk/__init__.py                    0      0   100%
	 base_python/cdk/abstract_source.py            89     64    28%
	 base_python/cdk/streams/__init__.py            0      0   100%
	 base_python/cdk/streams/auth/__init__.py       0      0   100%
	 base_python/cdk/streams/auth/core.py           8      1    88%
	 base_python/cdk/streams/auth/jwt.py            5      5     0%
	 base_python/cdk/streams/auth/oauth.py         37     26    30%
	 base_python/cdk/streams/auth/token.py          9      4    56%
	 base_python/cdk/streams/core.py               63     32    49%
	 base_python/cdk/streams/exceptions.py         10      2    80%
	 base_python/cdk/streams/http.py               67     33    51%
	 base_python/cdk/streams/rate_limiting.py      30     14    53%
	 base_python/cdk/utils/__init__.py              0      0   100%
	 base_python/cdk/utils/casing.py                4      0   100%
	 base_python/cdk/utils/event_timing.py         47      3    94%
	 base_python/client.py                         56     33    41%
	 base_python/entrypoint.py                     70     56    20%
	 base_python/integration.py                    52     25    52%
	 base_python/logger.py                         33     15    55%
	 base_python/schema_helpers.py                 56     41    27%
	 base_python/source.py                         51     34    33%
	 main_dev.py                                    3      3     0%
	 --------------------------------------------------------------
	 TOTAL                                        713    397    44%
	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------/actions-runner/_work/airbyte/airbyte/airbyte-integrations/connectors/source-delighted/.venv/lib/python3.8/site-packages/coverage/control.py:768: CoverageWarning: No data was collected. (no-data-collected)
	 Name                           Stmts   Miss  Cover
	 --------------------------------------------------
	 source_delighted/__init__.py       2      2     0%
	 source_delighted/source.py        91     91     0%
	   self._warn("No data was collected.", slug="no-data-collected")
	 --------------------------------------------------
	 TOTAL                             93     93     0%

Zirochkaa · 2022-01-04T14:13:02Z

/test connector=source-delighted

🕑 source-delighted https://github.com/airbytehq/airbyte/actions/runs/1653991801
✅ source-delighted https://github.com/airbytehq/airbyte/actions/runs/1653991801
Python tests coverage:

	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                                 Stmts   Miss  Cover
	 ------------------------------------------------------------------------
	 source_acceptance_test/__init__.py                       2      0   100%
	 source_acceptance_test/base.py                          10      4    60%
	 source_acceptance_test/config.py                        74      6    92%
	 source_acceptance_test/conftest.py                     109    109     0%
	 source_acceptance_test/plugin.py                        47     47     0%
	 source_acceptance_test/tests/__init__.py                 4      0   100%
	 source_acceptance_test/tests/test_core.py              242     96    60%
	 source_acceptance_test/tests/test_full_refresh.py       38      0   100%
	 source_acceptance_test/tests/test_incremental.py        69     38    45%
	 source_acceptance_test/utils/__init__.py                 6      0   100%
	 source_acceptance_test/utils/asserts.py                 37      2    95%
	 source_acceptance_test/utils/common.py                  54     17    69%
	 source_acceptance_test/utils/compare.py                 62     23    63%
	 source_acceptance_test/utils/connector_runner.py       110     48    56%
	 source_acceptance_test/utils/json_schema_helper.py     115     14    88%
	 ------------------------------------------------------------------------
	 TOTAL                                                  979    404    59%
	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                       Stmts   Miss  Cover
	 --------------------------------------------------------------
	 base_python/__init__.py                       13      0   100%
	 base_python/catalog_helpers.py                10      6    40%
	 base_python/cdk/__init__.py                    0      0   100%
	 base_python/cdk/abstract_source.py            89     64    28%
	 base_python/cdk/streams/__init__.py            0      0   100%
	 base_python/cdk/streams/auth/__init__.py       0      0   100%
	 base_python/cdk/streams/auth/core.py           8      1    88%
	 base_python/cdk/streams/auth/jwt.py            5      5     0%
	 base_python/cdk/streams/auth/oauth.py         37     26    30%
	 base_python/cdk/streams/auth/token.py          9      4    56%
	 base_python/cdk/streams/core.py               63     32    49%
	 base_python/cdk/streams/exceptions.py         10      2    80%
	 base_python/cdk/streams/http.py               67     33    51%
	 base_python/cdk/streams/rate_limiting.py      30     14    53%
	 base_python/cdk/utils/__init__.py              0      0   100%
	 base_python/cdk/utils/casing.py                4      0   100%
	 base_python/cdk/utils/event_timing.py         47      3    94%
	 base_python/client.py                         56     33    41%
	 base_python/entrypoint.py                     70     56    20%
	 base_python/integration.py                    52     25    52%
	 base_python/logger.py                         33     15    55%
	 base_python/schema_helpers.py                 56     41    27%
	 base_python/source.py                         51     34    33%
	 main_dev.py                                    3      3     0%
	 --------------------------------------------------------------
	 TOTAL                                        713    397    44%
	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                           Stmts   Miss  Cover
	 --------------------------------------------------
	 source_delighted/__init__.py       2      2     0%
	 --------------------------------------------------/actions-runner/_work/airbyte/airbyte/airbyte-integrations/connectors/source-delighted/.venv/lib/python3.8/site-packages/coverage/control.py:768: CoverageWarning: No data was collected. (no-data-collected)
	 source_delighted/source.py        91     91     0%
	 --------------------------------------------------
	 TOTAL                             93     93     0%

Zirochkaa · 2022-01-04T14:21:49Z

/publish connector=connectors/source-delighted

🕑 connectors/source-delighted https://github.com/airbytehq/airbyte/actions/runs/1654027959
✅ connectors/source-delighted https://github.com/airbytehq/airbyte/actions/runs/1654027959

9271 Fix pagination for delighted source

98dc5cd

Zirochkaa self-assigned this Jan 4, 2022

github-actions bot added area/connectors Connector related issues area/documentation Improvements or additions to documentation labels Jan 4, 2022

Zirochkaa temporarily deployed to more-secrets January 4, 2022 10:38 Inactive

jrhizor temporarily deployed to more-secrets January 4, 2022 11:15 Inactive

Zirochkaa requested review from htrueman, sergei-solonitcyn and yevhenii-ldv January 4, 2022 11:20

yevhenii-ldv reviewed Jan 4, 2022

View reviewed changes

airbyte-integrations/connectors/source-delighted/source_delighted/schemas/survey_responses.json Outdated Show resolved Hide resolved

yevhenii-ldv reviewed Jan 4, 2022

View reviewed changes

docs/integrations/sources/delighted.md Outdated Show resolved Hide resolved

yevhenii-ldv reviewed Jan 4, 2022

View reviewed changes

airbyte-integrations/connectors/source-delighted/source_delighted/source.py Outdated Show resolved Hide resolved

yevhenii-ldv reviewed Jan 4, 2022

View reviewed changes

yevhenii-ldv requested changes Jan 4, 2022

View reviewed changes

9271 Implement change request

f88d6ac

Zirochkaa temporarily deployed to more-secrets January 4, 2022 13:44 Inactive

jrhizor temporarily deployed to more-secrets January 4, 2022 13:59 Inactive

Zirochkaa requested a review from yevhenii-ldv January 4, 2022 14:05

9271 Implement change request

beae128

Zirochkaa temporarily deployed to more-secrets January 4, 2022 14:14 Inactive

jrhizor temporarily deployed to more-secrets January 4, 2022 14:15 Inactive

yevhenii-ldv approved these changes Jan 4, 2022

View reviewed changes

jrhizor temporarily deployed to more-secrets January 4, 2022 14:23 Inactive

9271 Bump connector's version

e96c8c0

Zirochkaa merged commit 103e003 into master Jan 4, 2022

Zirochkaa deleted the oleh/9271-fix-pagination-delighted-source branch January 4, 2022 14:32

Zirochkaa temporarily deployed to more-secrets January 4, 2022 14:33 Inactive

karinakuz added connectors/source/delighted connectors/sources-api labels Jan 5, 2022

jrhizor mentioned this pull request Jan 7, 2022

Bump Airbyte version from 0.35.3-alpha to 0.35.4-alpha #9353

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🐛 Source Delighted: Fix pagination for `survey_responses`, `bounces` and `unsubscribes` streams #9275

🐛 Source Delighted: Fix pagination for `survey_responses`, `bounces` and `unsubscribes` streams #9275

Zirochkaa commented Jan 4, 2022 •

edited

Loading

Zirochkaa commented Jan 4, 2022 •

edited by github-actions bot

Loading

yevhenii-ldv Jan 4, 2022

Zirochkaa Jan 4, 2022

yevhenii-ldv left a comment

Zirochkaa commented Jan 4, 2022 •

edited by github-actions bot

Loading

Zirochkaa commented Jan 4, 2022 •

edited by github-actions bot

Loading

Zirochkaa commented Jan 4, 2022 •

edited by github-actions bot

Loading

🐛 Source Delighted: Fix pagination for survey_responses, bounces and unsubscribes streams #9275

🐛 Source Delighted: Fix pagination for survey_responses, bounces and unsubscribes streams #9275

Conversation

Zirochkaa commented Jan 4, 2022 • edited Loading

Bug-fix Request

Reason

Confirmation

How does the code change in the PR fix the issue?

Zirochkaa commented Jan 4, 2022 • edited by github-actions bot Loading

yevhenii-ldv Jan 4, 2022

Choose a reason for hiding this comment

Zirochkaa Jan 4, 2022

Choose a reason for hiding this comment

yevhenii-ldv left a comment

Choose a reason for hiding this comment

Zirochkaa commented Jan 4, 2022 • edited by github-actions bot Loading

Zirochkaa commented Jan 4, 2022 • edited by github-actions bot Loading

Zirochkaa commented Jan 4, 2022 • edited by github-actions bot Loading

🐛 Source Delighted: Fix pagination for `survey_responses`, `bounces` and `unsubscribes` streams #9275

🐛 Source Delighted: Fix pagination for `survey_responses`, `bounces` and `unsubscribes` streams #9275

Zirochkaa commented Jan 4, 2022 •

edited

Loading

Zirochkaa commented Jan 4, 2022 •

edited by github-actions bot

Loading

Zirochkaa commented Jan 4, 2022 •

edited by github-actions bot

Loading

Zirochkaa commented Jan 4, 2022 •

edited by github-actions bot

Loading

Zirochkaa commented Jan 4, 2022 •

edited by github-actions bot

Loading