🐛 Source Delighted: output only records in which cursor field is greater than the value in state for incremental streams #9550

Zirochkaa · 2022-01-17T13:15:14Z

Bug-fix Request

Reason

Link to message with details is here.
The Delighted API supports since or updated_since parameters and they require Unix timestamp in order to restrict responses to those updated on or after this time. So, for example, if we send since=1641289583 this means that records which were created at 1641289583 or later will be returned.

Confirmation

Was reproduced locally

How does the code change in the PR fix the issue?

Filter records in python code and output records in which cursor field value is greater then value in state.

Zirochkaa · 2022-01-17T13:15:46Z

/test connector=source-delighted

🕑 source-delighted https://github.com/airbytehq/airbyte/actions/runs/1708126897
✅ source-delighted https://github.com/airbytehq/airbyte/actions/runs/1708126897
Python tests coverage:

	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                                 Stmts   Miss  Cover
	 ------------------------------------------------------------------------
	 source_acceptance_test/__init__.py                       2      0   100%
	 source_acceptance_test/base.py                          10      4    60%
	 source_acceptance_test/config.py                        74      6    92%
	 source_acceptance_test/conftest.py                     109    109     0%
	 source_acceptance_test/plugin.py                        47     47     0%
	 source_acceptance_test/tests/__init__.py                 4      0   100%
	 source_acceptance_test/tests/test_core.py              242     96    60%
	 source_acceptance_test/tests/test_full_refresh.py       38      0   100%
	 source_acceptance_test/tests/test_incremental.py        69     38    45%
	 source_acceptance_test/utils/__init__.py                 6      0   100%
	 source_acceptance_test/utils/asserts.py                 37      2    95%
	 source_acceptance_test/utils/common.py                  54     17    69%
	 source_acceptance_test/utils/compare.py                 62     23    63%
	 source_acceptance_test/utils/connector_runner.py       110     48    56%
	 source_acceptance_test/utils/json_schema_helper.py     115     14    88%
	 ------------------------------------------------------------------------
	 TOTAL                                                  979    404    59%
	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                       Stmts   Miss  Cover
	 --------------------------------------------------------------
	 base_python/__init__.py                       13      0   100%
	 base_python/catalog_helpers.py                10      6    40%
	 base_python/cdk/__init__.py                    0      0   100%
	 base_python/cdk/abstract_source.py            89     64    28%
	 base_python/cdk/streams/__init__.py            0      0   100%
	 base_python/cdk/streams/auth/__init__.py       0      0   100%
	 base_python/cdk/streams/auth/core.py           8      1    88%
	 base_python/cdk/streams/auth/jwt.py            5      5     0%
	 base_python/cdk/streams/auth/oauth.py         37     26    30%
	 base_python/cdk/streams/auth/token.py          9      4    56%
	 base_python/cdk/streams/core.py               63     32    49%
	 base_python/cdk/streams/exceptions.py         10      2    80%
	 base_python/cdk/streams/http.py               67     33    51%
	 base_python/cdk/streams/rate_limiting.py      30     14    53%
	 base_python/cdk/utils/__init__.py              0      0   100%
	 base_python/cdk/utils/casing.py                4      0   100%
	 base_python/cdk/utils/event_timing.py         47      3    94%
	 base_python/client.py                         56     33    41%
	 base_python/entrypoint.py                     70     56    20%
	 base_python/integration.py                    52     25    52%
	 base_python/logger.py                         33     15    55%
	 base_python/schema_helpers.py                 56     41    27%
	 base_python/source.py                         51     34    33%
	 main_dev.py                                    3      3     0%
	 --------------------------------------------------------------
	 TOTAL                                        713    397    44%
	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                           Stmts   Miss  Cover
	 --------------------------------------------------
	 --------------------------------------------------/actions-runner/_work/airbyte/airbyte/airbyte-integrations/connectors/source-delighted/.venv/lib/python3.8/site-packages/coverage/control.py:768: CoverageWarning: No data was collected. (no-data-collected)
	 source_delighted/__init__.py       2      2     0%
	   self._warn("No data was collected.", slug="no-data-collected")
	 source_delighted/source.py        95     95     0%
	 --------------------------------------------------
	 TOTAL                             97     97     0%

airbyte-integrations/connectors/source-delighted/source_delighted/source.py

Zirochkaa · 2022-01-18T08:49:47Z

/test connector=source-delighted

🕑 source-delighted https://github.com/airbytehq/airbyte/actions/runs/1711807130
✅ source-delighted https://github.com/airbytehq/airbyte/actions/runs/1711807130
Python tests coverage:

	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                                 Stmts   Miss  Cover
	 ------------------------------------------------------------------------
	 source_acceptance_test/__init__.py                       2      0   100%
	 source_acceptance_test/base.py                          10      4    60%
	 source_acceptance_test/config.py                        74      6    92%
	 source_acceptance_test/conftest.py                     109    109     0%
	 source_acceptance_test/plugin.py                        47     47     0%
	 source_acceptance_test/tests/__init__.py                 4      0   100%
	 source_acceptance_test/tests/test_core.py              242     96    60%
	 source_acceptance_test/tests/test_full_refresh.py       38      0   100%
	 source_acceptance_test/tests/test_incremental.py        69     38    45%
	 source_acceptance_test/utils/__init__.py                 6      0   100%
	 source_acceptance_test/utils/asserts.py                 37      2    95%
	 source_acceptance_test/utils/common.py                  54     17    69%
	 source_acceptance_test/utils/compare.py                 62     23    63%
	 source_acceptance_test/utils/connector_runner.py       110     48    56%
	 source_acceptance_test/utils/json_schema_helper.py     115     14    88%
	 ------------------------------------------------------------------------
	 TOTAL                                                  979    404    59%
	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                       Stmts   Miss  Cover
	 --------------------------------------------------------------
	 base_python/__init__.py                       13      0   100%
	 base_python/catalog_helpers.py                10      6    40%
	 base_python/cdk/__init__.py                    0      0   100%
	 base_python/cdk/abstract_source.py            89     64    28%
	 base_python/cdk/streams/__init__.py            0      0   100%
	 base_python/cdk/streams/auth/__init__.py       0      0   100%
	 base_python/cdk/streams/auth/core.py           8      1    88%
	 base_python/cdk/streams/auth/jwt.py            5      5     0%
	 base_python/cdk/streams/auth/oauth.py         37     26    30%
	 base_python/cdk/streams/auth/token.py          9      4    56%
	 base_python/cdk/streams/core.py               63     32    49%
	 base_python/cdk/streams/exceptions.py         10      2    80%
	 base_python/cdk/streams/http.py               67     33    51%
	 base_python/cdk/streams/rate_limiting.py      30     14    53%
	 base_python/cdk/utils/__init__.py              0      0   100%
	 base_python/cdk/utils/casing.py                4      0   100%
	 base_python/cdk/utils/event_timing.py         47      3    94%
	 base_python/client.py                         56     33    41%
	 base_python/entrypoint.py                     70     56    20%
	 base_python/integration.py                    52     25    52%
	 base_python/logger.py                         33     15    55%
	 base_python/schema_helpers.py                 56     41    27%
	 base_python/source.py                         51     34    33%
	 main_dev.py                                    3      3     0%
	 --------------------------------------------------------------
	 TOTAL                                        713    397    44%
	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                           Stmts   Miss  Cover
	 --------------------------------------------------
	 source_delighted/__init__.py       2      2     0%
	 --------------------------------------------------/actions-runner/_work/airbyte/airbyte/airbyte-integrations/connectors/source-delighted/.venv/lib/python3.8/site-packages/coverage/control.py:768: CoverageWarning: No data was collected. (no-data-collected)
	 source_delighted/source.py        97     97     0%
	   self._warn("No data was collected.", slug="no-data-collected")
	 --------------------------------------------------
	 TOTAL                             99     99     0%

Zirochkaa · 2022-01-28T15:18:45Z

/test connector=source-delighted

🕑 source-delighted https://github.com/airbytehq/airbyte/actions/runs/1762232774
✅ source-delighted https://github.com/airbytehq/airbyte/actions/runs/1762232774
Python tests coverage:

	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                                 Stmts   Miss  Cover
	 ------------------------------------------------------------------------
	 source_acceptance_test/__init__.py                       2      0   100%
	 source_acceptance_test/base.py                          10      4    60%
	 source_acceptance_test/config.py                        74      6    92%
	 source_acceptance_test/conftest.py                     109    109     0%
	 source_acceptance_test/plugin.py                        47     47     0%
	 source_acceptance_test/tests/__init__.py                 4      0   100%
	 source_acceptance_test/tests/test_core.py              270    123    54%
	 source_acceptance_test/tests/test_full_refresh.py       52      2    96%
	 source_acceptance_test/tests/test_incremental.py        69     38    45%
	 source_acceptance_test/utils/__init__.py                 6      0   100%
	 source_acceptance_test/utils/asserts.py                 37      2    95%
	 source_acceptance_test/utils/common.py                  54     17    69%
	 source_acceptance_test/utils/compare.py                 62     23    63%
	 source_acceptance_test/utils/connector_runner.py       110     48    56%
	 source_acceptance_test/utils/json_schema_helper.py      98     12    88%
	 ------------------------------------------------------------------------
	 TOTAL                                                 1004    431    57%
	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                       Stmts   Miss  Cover
	 --------------------------------------------------------------
	 base_python/__init__.py                       13      0   100%
	 base_python/catalog_helpers.py                10      6    40%
	 base_python/cdk/__init__.py                    0      0   100%
	 base_python/cdk/abstract_source.py            89     64    28%
	 base_python/cdk/streams/__init__.py            0      0   100%
	 base_python/cdk/streams/auth/__init__.py       0      0   100%
	 base_python/cdk/streams/auth/core.py           8      1    88%
	 base_python/cdk/streams/auth/jwt.py            5      5     0%
	 base_python/cdk/streams/auth/oauth.py         37     26    30%
	 base_python/cdk/streams/auth/token.py          9      4    56%
	 base_python/cdk/streams/core.py               63     32    49%
	 base_python/cdk/streams/exceptions.py         10      2    80%
	 base_python/cdk/streams/http.py               67     33    51%
	 base_python/cdk/streams/rate_limiting.py      30     14    53%
	 base_python/cdk/utils/__init__.py              0      0   100%
	 base_python/cdk/utils/casing.py                4      0   100%
	 base_python/cdk/utils/event_timing.py         47      3    94%
	 base_python/client.py                         56     33    41%
	 base_python/entrypoint.py                     70     56    20%
	 base_python/integration.py                    52     25    52%
	 base_python/logger.py                         33     15    55%
	 base_python/schema_helpers.py                 56     41    27%
	 base_python/source.py                         51     34    33%
	 main_dev.py                                    3      3     0%
	 --------------------------------------------------------------
	 TOTAL                                        713    397    44%
	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                           Stmts   Miss  Cover
	 --------------------------------------------------
	 source_delighted/__init__.py       2      0   100%
	 source_delighted/source.py        97     18    81%
	 --------------------------------------------------
	 TOTAL                             99     18    82%

codecov · 2022-01-28T15:21:06Z

Codecov Report

❗ No coverage uploaded for pull request base (master@c873898). Click here to learn what that means.
The diff coverage is n/a.

❗ Current head f008a20 differs from pull request most recent head 4569705. Consider uploading reports for the commit 4569705 to get more accurate results

@@            Coverage Diff            @@
##             master    #9550   +/-   ##
=========================================
  Coverage          ?   81.63%           
=========================================
  Files             ?        2           
  Lines             ?       98           
  Branches          ?        0           
=========================================
  Hits              ?       80           
  Misses            ?       18           
  Partials          ?        0

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c873898...4569705. Read the comment docs.

Zirochkaa · 2022-01-31T07:07:06Z

/publish connector=connectors/source-delighted

🕑 connectors/source-delighted https://github.com/airbytehq/airbyte/actions/runs/1771274598
❌ connectors/source-delighted https://github.com/airbytehq/airbyte/actions/runs/1771274598

Zirochkaa · 2022-01-31T07:18:42Z

/publish connector=connectors/source-delighted

🕑 connectors/source-delighted https://github.com/airbytehq/airbyte/actions/runs/1771313576
❌ connectors/source-delighted https://github.com/airbytehq/airbyte/actions/runs/1771313576

…lue in state for incremental streams

Zirochkaa · 2022-01-31T07:57:23Z

/test connector=source-delighted

🕑 source-delighted https://github.com/airbytehq/airbyte/actions/runs/1771427889
✅ source-delighted https://github.com/airbytehq/airbyte/actions/runs/1771427889
Python tests coverage:

	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                                 Stmts   Miss  Cover
	 ------------------------------------------------------------------------
	 source_acceptance_test/__init__.py                       2      0   100%
	 source_acceptance_test/base.py                          10      4    60%
	 source_acceptance_test/config.py                        74      6    92%
	 source_acceptance_test/conftest.py                     109    109     0%
	 source_acceptance_test/plugin.py                        47     47     0%
	 source_acceptance_test/tests/__init__.py                 4      0   100%
	 source_acceptance_test/tests/test_core.py              275    106    61%
	 source_acceptance_test/tests/test_full_refresh.py       52      2    96%
	 source_acceptance_test/tests/test_incremental.py        69     38    45%
	 source_acceptance_test/utils/__init__.py                 6      0   100%
	 source_acceptance_test/utils/asserts.py                 37      2    95%
	 source_acceptance_test/utils/common.py                  70     17    76%
	 source_acceptance_test/utils/compare.py                 62     23    63%
	 source_acceptance_test/utils/connector_runner.py       110     48    56%
	 source_acceptance_test/utils/json_schema_helper.py     105     13    88%
	 ------------------------------------------------------------------------
	 TOTAL                                                 1032    415    60%
	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                       Stmts   Miss  Cover
	 --------------------------------------------------------------
	 base_python/__init__.py                       13      0   100%
	 base_python/catalog_helpers.py                10      6    40%
	 base_python/cdk/__init__.py                    0      0   100%
	 base_python/cdk/abstract_source.py            89     64    28%
	 base_python/cdk/streams/__init__.py            0      0   100%
	 base_python/cdk/streams/auth/__init__.py       0      0   100%
	 base_python/cdk/streams/auth/core.py           8      1    88%
	 base_python/cdk/streams/auth/jwt.py            5      5     0%
	 base_python/cdk/streams/auth/oauth.py         37     26    30%
	 base_python/cdk/streams/auth/token.py          9      4    56%
	 base_python/cdk/streams/core.py               63     32    49%
	 base_python/cdk/streams/exceptions.py         10      2    80%
	 base_python/cdk/streams/http.py               67     33    51%
	 base_python/cdk/streams/rate_limiting.py      30     14    53%
	 base_python/cdk/utils/__init__.py              0      0   100%
	 base_python/cdk/utils/casing.py                4      0   100%
	 base_python/cdk/utils/event_timing.py         47      3    94%
	 base_python/client.py                         56     33    41%
	 base_python/entrypoint.py                     70     56    20%
	 base_python/integration.py                    52     25    52%
	 base_python/logger.py                         33     15    55%
	 base_python/schema_helpers.py                 56     41    27%
	 base_python/source.py                         51     34    33%
	 main_dev.py                                    3      3     0%
	 --------------------------------------------------------------
	 TOTAL                                        713    397    44%
	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                           Stmts   Miss  Cover
	 --------------------------------------------------
	 source_delighted/__init__.py       2      0   100%
	 source_delighted/source.py        97     18    81%
	 --------------------------------------------------
	 TOTAL                             99     18    82%

Zirochkaa · 2022-01-31T08:10:11Z

/publish connector=connectors/source-delighted

🕑 connectors/source-delighted https://github.com/airbytehq/airbyte/actions/runs/1771476893
✅ connectors/source-delighted https://github.com/airbytehq/airbyte/actions/runs/1771476893

Tests were added.

Zirochkaa self-assigned this Jan 17, 2022

github-actions bot added area/connectors Connector related issues area/documentation Improvements or additions to documentation labels Jan 17, 2022

Zirochkaa temporarily deployed to more-secrets January 17, 2022 13:16 Inactive

octavia-squidington-iii temporarily deployed to more-secrets January 17, 2022 13:18 Inactive

Zirochkaa requested review from htrueman, sergei-solonitcyn and yevhenii-ldv January 17, 2022 13:46

sergei-solonitcyn previously requested changes Jan 18, 2022

View reviewed changes

airbyte-integrations/connectors/source-delighted/source_delighted/source.py Show resolved Hide resolved

Zirochkaa temporarily deployed to more-secrets January 18, 2022 08:51 Inactive

octavia-squidington-iii temporarily deployed to more-secrets January 18, 2022 08:52 Inactive

Zirochkaa force-pushed the oleh/8906-output-only-new-records branch from 4ca1447 to 7bf1860 Compare January 28, 2022 15:18

Zirochkaa temporarily deployed to more-secrets January 28, 2022 15:20 Inactive

octavia-squidington-iii temporarily deployed to more-secrets January 28, 2022 15:21 Inactive

github-actions bot referenced this pull request Jan 28, 2022

8906 Add tests + update docs

7bf1860

Zirochkaa requested a review from sergei-solonitcyn January 28, 2022 15:28

htrueman approved these changes Jan 28, 2022

View reviewed changes

Zirochkaa temporarily deployed to more-secrets January 31, 2022 07:08 Inactive

octavia-squidington-iii temporarily deployed to more-secrets January 31, 2022 07:09 Inactive

octavia-squidington-iii temporarily deployed to more-secrets January 31, 2022 07:21 Inactive

Zirochkaa added 3 commits January 31, 2022 09:56

8906 Output only records in which cursor field is greater than the va…

4082c87

…lue in state for incremental streams

8906 Fix full refresh read for SurveyResponses stream

9cfbbf5

8906 Add tests + update docs

976ed8d

8906 Update docs

425a59f

Zirochkaa force-pushed the oleh/8906-output-only-new-records branch from f008a20 to 425a59f Compare January 31, 2022 07:57

Zirochkaa temporarily deployed to more-secrets January 31, 2022 07:59 Inactive

octavia-squidington-iii temporarily deployed to more-secrets January 31, 2022 07:59 Inactive

octavia-squidington-iii temporarily deployed to more-secrets January 31, 2022 08:12 Inactive

8906 Bump connector's version

4569705

Zirochkaa merged commit 0b1b75b into master Jan 31, 2022

Zirochkaa deleted the oleh/8906-output-only-new-records branch January 31, 2022 08:45

Zirochkaa temporarily deployed to more-secrets January 31, 2022 08:46 Inactive

octavia-squidington-iii mentioned this pull request Jan 31, 2022

Bump Airbyte version from 0.35.12-alpha to 0.35.13-alpha #9911

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🐛 Source Delighted: output only records in which cursor field is greater than the value in state for incremental streams #9550

🐛 Source Delighted: output only records in which cursor field is greater than the value in state for incremental streams #9550

Zirochkaa commented Jan 17, 2022 •

edited

Loading

Zirochkaa commented Jan 17, 2022 •

edited by github-actions bot

Loading

Zirochkaa commented Jan 18, 2022 •

edited by github-actions bot

Loading

Zirochkaa commented Jan 28, 2022 •

edited by github-actions bot

Loading

codecov bot commented Jan 28, 2022 •

edited

Loading

Zirochkaa commented Jan 31, 2022 •

edited by github-actions bot

Loading

Zirochkaa commented Jan 31, 2022 •

edited by github-actions bot

Loading

Zirochkaa commented Jan 31, 2022 •

edited by github-actions bot

Loading

Zirochkaa commented Jan 31, 2022 •

edited by github-actions bot

Loading

🐛 Source Delighted: output only records in which cursor field is greater than the value in state for incremental streams #9550

🐛 Source Delighted: output only records in which cursor field is greater than the value in state for incremental streams #9550

Conversation

Zirochkaa commented Jan 17, 2022 • edited Loading

Bug-fix Request

Reason

Confirmation

How does the code change in the PR fix the issue?

Zirochkaa commented Jan 17, 2022 • edited by github-actions bot Loading

Zirochkaa commented Jan 18, 2022 • edited by github-actions bot Loading

Zirochkaa commented Jan 28, 2022 • edited by github-actions bot Loading

codecov bot commented Jan 28, 2022 • edited Loading

Codecov Report

Zirochkaa commented Jan 31, 2022 • edited by github-actions bot Loading

Zirochkaa commented Jan 31, 2022 • edited by github-actions bot Loading

Zirochkaa commented Jan 31, 2022 • edited by github-actions bot Loading

Zirochkaa commented Jan 31, 2022 • edited by github-actions bot Loading

Zirochkaa commented Jan 17, 2022 •

edited

Loading

Zirochkaa commented Jan 17, 2022 •

edited by github-actions bot

Loading

Zirochkaa commented Jan 18, 2022 •

edited by github-actions bot

Loading

Zirochkaa commented Jan 28, 2022 •

edited by github-actions bot

Loading

codecov bot commented Jan 28, 2022 •

edited

Loading

Zirochkaa commented Jan 31, 2022 •

edited by github-actions bot

Loading

Zirochkaa commented Jan 31, 2022 •

edited by github-actions bot

Loading

Zirochkaa commented Jan 31, 2022 •

edited by github-actions bot

Loading

Zirochkaa commented Jan 31, 2022 •

edited by github-actions bot

Loading