🐛 Source Github: Fix error handling for 404 streams #10878

grubberr · 2022-03-05T06:16:47Z

Signed-off-by: Sergey Chvalyuk grubberr@gmail.com

What

Fix regression for streams which generate 404

requests.exceptions.HTTPError: 410 Client Error: Gone for url: ...

How

We need to get empty list for such streams

🚨 User Impact 🚨

Are there any breaking changes? What is the end result perceived by the user? If yes, please merge this PR with the 🚨🚨 emoji so changelog authors can further highlight this if needed.

Pre-merge Checklist

Expand the relevant checklist and delete the others.

Updating a connector

Community member or Airbyter

Grant edit access to maintainers (instructions)
Secrets in the connector's spec are annotated with airbyte_secret
Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
Code reviews completed
Documentation updated
- Connector's README.md
- Connector's bootstrap.md. See description and examples
- Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
Credentials added to Github CI. Instructions.
/test connector=connectors/<name> command is passing
New Connector version released on Dockerhub by running the /publish command described here
After the new connector version is published, connector version bumped in the seed directory as described here
Seed specs have been re-generated by building the platform and committing the changes to the seed spec files, as described here

Tests

Unit

Put your unit tests output here.

Integration

Put your integration tests output here.

Acceptance

Put your acceptance tests output here.

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

codecov · 2022-03-05T06:26:12Z

Codecov Report

❗ No coverage uploaded for pull request base (master@af1a503). Click here to learn what that means.
The diff coverage is n/a.

❗ Current head d8eb6be differs from pull request most recent head 5e2d89f. Consider uploading reports for the commit 5e2d89f to get more accurate results

@@            Coverage Diff            @@
##             master   #10878   +/-   ##
=========================================
  Coverage          ?   52.68%           
=========================================
  Files             ?        3           
  Lines             ?      558           
  Branches          ?        0           
=========================================
  Hits              ?      294           
  Misses            ?      264           
  Partials          ?        0

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update af1a503...5e2d89f. Read the comment docs.

grubberr · 2022-03-05T06:27:48Z

/test connector=connectors/source-github

🕑 connectors/source-github https://github.com/airbytehq/airbyte/actions/runs/1937336079
✅ connectors/source-github https://github.com/airbytehq/airbyte/actions/runs/1937336079
Python tests coverage:

Name                                                 Stmts   Miss  Cover
------------------------------------------------------------------------
source_acceptance_test/utils/__init__.py                 6      0   100%
source_acceptance_test/tests/__init__.py                 4      0   100%
source_acceptance_test/__init__.py                       2      0   100%
source_acceptance_test/tests/test_full_refresh.py       52      2    96%
source_acceptance_test/utils/asserts.py                 37      2    95%
source_acceptance_test/config.py                        74      6    92%
source_acceptance_test/utils/json_schema_helper.py     105     13    88%
source_acceptance_test/utils/common.py                  70     17    76%
source_acceptance_test/utils/compare.py                 62     23    63%
source_acceptance_test/tests/test_core.py              275    106    61%
source_acceptance_test/base.py                          10      4    60%
source_acceptance_test/utils/connector_runner.py       110     48    56%
source_acceptance_test/tests/test_incremental.py        69     38    45%
------------------------------------------------------------------------
TOTAL                                                  876    259    70%
Name                        Stmts   Miss  Cover
-----------------------------------------------
source_github/__init__.py       2      0   100%
source_github/streams.py      434    194    55%
source_github/source.py        88     45    49%
-----------------------------------------------
TOTAL                         524    239    54%

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

grubberr · 2022-03-06T06:09:41Z

/test connector=connectors/source-github

🕑 connectors/source-github https://github.com/airbytehq/airbyte/actions/runs/1940349317
❌ connectors/source-github https://github.com/airbytehq/airbyte/actions/runs/1940349317
🐛 https://gradle.com/s/4u2qoscfhrrj4

bazarnov

LGTM. Please fix the unit_tests. Let's make the description in changelog more informative.

docs/integrations/sources/github.md

grubberr · 2022-03-06T13:33:48Z

/test connector=connectors/source-github

🕑 connectors/source-github https://github.com/airbytehq/airbyte/actions/runs/1941308166
❌ connectors/source-github https://github.com/airbytehq/airbyte/actions/runs/1941308166
🐛 https://gradle.com/s/egcuainp64a2s

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

grubberr · 2022-03-10T09:08:28Z

/test connector=connectors/source-github

🕑 connectors/source-github https://github.com/airbytehq/airbyte/actions/runs/1962163330
✅ connectors/source-github https://github.com/airbytehq/airbyte/actions/runs/1962163330
Python tests coverage:

Name                                                 Stmts   Miss  Cover
------------------------------------------------------------------------
source_acceptance_test/utils/__init__.py                 6      0   100%
source_acceptance_test/tests/__init__.py                 4      0   100%
source_acceptance_test/__init__.py                       2      0   100%
source_acceptance_test/tests/test_full_refresh.py       52      2    96%
source_acceptance_test/utils/asserts.py                 37      2    95%
source_acceptance_test/config.py                        74      6    92%
source_acceptance_test/utils/json_schema_helper.py     105     13    88%
source_acceptance_test/utils/common.py                  70     17    76%
source_acceptance_test/utils/compare.py                 62     23    63%
source_acceptance_test/tests/test_core.py              275    106    61%
source_acceptance_test/base.py                          10      4    60%
source_acceptance_test/utils/connector_runner.py       110     48    56%
source_acceptance_test/tests/test_incremental.py        69     38    45%
------------------------------------------------------------------------
TOTAL                                                  876    259    70%
Name                        Stmts   Miss  Cover
-----------------------------------------------
source_github/__init__.py       2      0   100%
source_github/streams.py      466    217    53%
source_github/source.py        89     46    48%
-----------------------------------------------
TOTAL                         557    263    53%

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

grubberr · 2022-03-10T13:01:46Z

/test connector=connectors/source-github

grubberr · 2022-03-10T13:44:14Z

/test connector=connectors/source-github

🕑 connectors/source-github https://github.com/airbytehq/airbyte/actions/runs/1963428672
✅ connectors/source-github https://github.com/airbytehq/airbyte/actions/runs/1963428672
Python tests coverage:

Name                                                 Stmts   Miss  Cover
------------------------------------------------------------------------
source_acceptance_test/utils/__init__.py                 6      0   100%
source_acceptance_test/tests/__init__.py                 4      0   100%
source_acceptance_test/__init__.py                       2      0   100%
source_acceptance_test/tests/test_full_refresh.py       52      2    96%
source_acceptance_test/utils/asserts.py                 37      2    95%
source_acceptance_test/config.py                        74      6    92%
source_acceptance_test/utils/json_schema_helper.py     105     13    88%
source_acceptance_test/utils/common.py                  70     17    76%
source_acceptance_test/utils/compare.py                 62     23    63%
source_acceptance_test/tests/test_core.py              275    106    61%
source_acceptance_test/base.py                          10      4    60%
source_acceptance_test/utils/connector_runner.py       110     48    56%
source_acceptance_test/tests/test_incremental.py        69     38    45%
------------------------------------------------------------------------
TOTAL                                                  876    259    70%
Name                        Stmts   Miss  Cover
-----------------------------------------------
source_github/__init__.py       2      0   100%
source_github/streams.py      468    218    53%
source_github/source.py        89     46    48%
-----------------------------------------------
TOTAL                         559    264    53%

grubberr · 2022-03-10T13:53:20Z

@sherifnada

if customer provided next config:

{
    "repository": "user/repository"
}

but API only supports organization/repository ( not user/repository)
we can get 404 for such cases

Because we have a some number of such issues with this 404 problem
I have re-implemented to catch 404 for ALL streams.

grubberr · 2022-03-10T14:25:51Z

/test connector=connectors/source-github

🕑 connectors/source-github https://github.com/airbytehq/airbyte/actions/runs/1963638287
✅ connectors/source-github https://github.com/airbytehq/airbyte/actions/runs/1963638287
Python tests coverage:

Name                                                 Stmts   Miss  Cover
------------------------------------------------------------------------
source_acceptance_test/utils/__init__.py                 6      0   100%
source_acceptance_test/tests/__init__.py                 4      0   100%
source_acceptance_test/__init__.py                       2      0   100%
source_acceptance_test/tests/test_full_refresh.py       52      2    96%
source_acceptance_test/utils/asserts.py                 37      2    95%
source_acceptance_test/config.py                        74      6    92%
source_acceptance_test/utils/json_schema_helper.py     105     13    88%
source_acceptance_test/utils/common.py                  70     17    76%
source_acceptance_test/utils/compare.py                 62     23    63%
source_acceptance_test/tests/test_core.py              275    106    61%
source_acceptance_test/base.py                          10      4    60%
source_acceptance_test/utils/connector_runner.py       110     48    56%
source_acceptance_test/tests/test_incremental.py        69     38    45%
------------------------------------------------------------------------
TOTAL                                                  876    259    70%
Name                        Stmts   Miss  Cover
-----------------------------------------------
source_github/__init__.py       2      0   100%
source_github/streams.py      468    218    53%
source_github/source.py        89     46    48%
-----------------------------------------------
TOTAL                         559    264    53%

grubberr · 2022-03-10T15:47:29Z

/publish connector=connectors/source-github

🕑 connectors/source-github https://github.com/airbytehq/airbyte/actions/runs/1964062892
✅ connectors/source-github https://github.com/airbytehq/airbyte/actions/runs/1964062892

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

Fix, no error if "Projects" feature is disabled

7d3378d

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

grubberr self-assigned this Mar 5, 2022

grubberr temporarily deployed to more-secrets March 5, 2022 06:16 Inactive

github-actions bot added area/connectors Connector related issues area/documentation Improvements or additions to documentation labels Mar 5, 2022

github.md updated

24a1023

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

grubberr temporarily deployed to more-secrets March 5, 2022 06:22 Inactive

octavia-squidington-iii temporarily deployed to more-secrets March 5, 2022 06:30 Inactive

grubberr added 2 commits March 5, 2022 16:19

better use isinstance

80f9256

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

improve error handling

97ca8c8

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

grubberr temporarily deployed to more-secrets March 5, 2022 18:13 Inactive

improve code

b1d5a0e

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

grubberr temporarily deployed to more-secrets March 6, 2022 06:02 Inactive

updated github.md

a3dd88e

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

grubberr changed the title ~~🐛 Source Github: Fix, no error if "Projects" feature is disabled~~ 🐛 Source Github: Fix error handling for 404 streams Mar 6, 2022

grubberr temporarily deployed to more-secrets March 6, 2022 06:05 Inactive

grubberr linked an issue Mar 6, 2022 that may be closed by this pull request

GitHub source connector fails if "Projects" feature is disabled #10555

Closed

octavia-squidington-iii temporarily deployed to more-secrets March 6, 2022 06:12 Inactive

grubberr requested review from midavadim and bazarnov March 6, 2022 06:16

bazarnov approved these changes Mar 6, 2022

View reviewed changes

docs/integrations/sources/github.md Outdated Show resolved Hide resolved

bump version 0.2.21 -> 0.2.22

d9818f6

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

grubberr temporarily deployed to more-secrets March 10, 2022 09:09 Inactive

grubberr temporarily deployed to more-secrets March 10, 2022 09:10 Inactive

improve requests.codes.NOT_FOUND

24c04ee

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

grubberr temporarily deployed to more-secrets March 10, 2022 11:20 Inactive

ChristopheDuong approved these changes Mar 10, 2022

View reviewed changes

improve comments

ab13517

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

grubberr temporarily deployed to more-secrets March 10, 2022 13:02 Inactive

grubberr requested a review from sherifnada March 10, 2022 13:50

grubberr linked an issue Mar 10, 2022 that may be closed by this pull request

GitHub source connector fails when /contributors endpoint returns 404 #10594

Closed

Merge branch 'master' into grubberr/10555-source-github

3bae9ba

grubberr temporarily deployed to more-secrets March 10, 2022 14:24 Inactive

sherifnada approved these changes Mar 10, 2022

View reviewed changes

Merge branch 'master' into grubberr/10555-source-github

d8eb6be

grubberr temporarily deployed to more-secrets March 10, 2022 15:48 Inactive

grubberr temporarily deployed to more-secrets March 10, 2022 15:49 Inactive

source_specs.yaml updated

5e2d89f

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

grubberr merged commit d19754c into master Mar 10, 2022

grubberr deleted the grubberr/10555-source-github branch March 10, 2022 16:02

grubberr temporarily deployed to more-secrets March 10, 2022 16:03 Inactive

octavia-squidington-iii mentioned this pull request Mar 10, 2022

Bump Airbyte version from 0.35.49-alpha to 0.35.50-alpha #11050

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🐛 Source Github: Fix error handling for 404 streams #10878

🐛 Source Github: Fix error handling for 404 streams #10878

grubberr commented Mar 5, 2022 •

edited

Loading

codecov bot commented Mar 5, 2022 •

edited

Loading

grubberr commented Mar 5, 2022 •

edited by github-actions bot

Loading

grubberr commented Mar 6, 2022 •

edited by github-actions bot

Loading

bazarnov left a comment

grubberr commented Mar 6, 2022 •

edited by github-actions bot

Loading

grubberr commented Mar 10, 2022 •

edited by github-actions bot

Loading

grubberr commented Mar 10, 2022

grubberr commented Mar 10, 2022 •

edited by github-actions bot

Loading

grubberr commented Mar 10, 2022 •

edited

Loading

grubberr commented Mar 10, 2022 •

edited by github-actions bot

Loading

grubberr commented Mar 10, 2022 •

edited by github-actions bot

Loading

🐛 Source Github: Fix error handling for 404 streams #10878

🐛 Source Github: Fix error handling for 404 streams #10878

Conversation

grubberr commented Mar 5, 2022 • edited Loading

What

How

Recommended reading order

🚨 User Impact 🚨

Pre-merge Checklist

Community member or Airbyter

Airbyter

Tests

codecov bot commented Mar 5, 2022 • edited Loading

Codecov Report

grubberr commented Mar 5, 2022 • edited by github-actions bot Loading

grubberr commented Mar 6, 2022 • edited by github-actions bot Loading

bazarnov left a comment

Choose a reason for hiding this comment

grubberr commented Mar 6, 2022 • edited by github-actions bot Loading

grubberr commented Mar 10, 2022 • edited by github-actions bot Loading

grubberr commented Mar 10, 2022

grubberr commented Mar 10, 2022 • edited by github-actions bot Loading

grubberr commented Mar 10, 2022 • edited Loading

grubberr commented Mar 10, 2022 • edited by github-actions bot Loading

grubberr commented Mar 10, 2022 • edited by github-actions bot Loading

grubberr commented Mar 5, 2022 •

edited

Loading

codecov bot commented Mar 5, 2022 •

edited

Loading

grubberr commented Mar 5, 2022 •

edited by github-actions bot

Loading

grubberr commented Mar 6, 2022 •

edited by github-actions bot

Loading

grubberr commented Mar 6, 2022 •

edited by github-actions bot

Loading

grubberr commented Mar 10, 2022 •

edited by github-actions bot

Loading

grubberr commented Mar 10, 2022 •

edited by github-actions bot

Loading

grubberr commented Mar 10, 2022 •

edited

Loading

grubberr commented Mar 10, 2022 •

edited by github-actions bot

Loading

grubberr commented Mar 10, 2022 •

edited by github-actions bot

Loading