🎉 Source File: separate secure fork #6768

antixar · 2021-10-05T15:15:26Z

What

Local storages can be used for local single environment and can't be available for clould platforms.

How

This new connector reuses all business logic of the native connector "Source File" and removes support of local storages only.

Pre-merge Checklist

New Connector

Community member or Airbyter

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
Credentials added to Github CI. Instructions.
/test connector=connectors/<name> command is passing.
New Connector version released on Dockerhub by running the /publish command described here

…rt-oauth2

…ecure

CLAassistant · 2021-10-05T15:15:30Z

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
1 out of 2 committers have signed the CLA.

✅ antixar
❌ Maksym Pavlenok

Maksym Pavlenok seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

antixar · 2021-10-06T18:35:17Z

/test connector=connectors/source-file-secure

🕑 connectors/source-file-secure https://github.com/airbytehq/airbyte/actions/runs/1313187358
❌ connectors/source-file-secure https://github.com/airbytehq/airbyte/actions/runs/1313187358
🐛 https://gradle.com/s/tzyplu6kthzn2

antixar · 2021-10-06T18:46:10Z

/test connector=connectors/source-file-secure

🕑 connectors/source-file-secure https://github.com/airbytehq/airbyte/actions/runs/1313219312
❌ connectors/source-file-secure https://github.com/airbytehq/airbyte/actions/runs/1313219312
🐛 https://gradle.com/s/majow4njilodw

antixar · 2021-10-06T19:42:45Z

/test connector=connectors/source-file-secure

🕑 connectors/source-file-secure https://github.com/airbytehq/airbyte/actions/runs/1313404077
✅ connectors/source-file-secure https://github.com/airbytehq/airbyte/actions/runs/1313404077
Python tests coverage:

	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                                 Stmts   Miss  Cover
	 ------------------------------------------------------------------------
	 source_acceptance_test/__init__.py                       2      0   100%
	 source_acceptance_test/base.py                          10      4    60%
	 source_acceptance_test/config.py                        74      8    89%
	 source_acceptance_test/conftest.py                     108    108     0%
	 source_acceptance_test/plugin.py                        47     47     0%
	 source_acceptance_test/tests/__init__.py                 4      0   100%
	 source_acceptance_test/tests/test_core.py              200     94    53%
	 source_acceptance_test/tests/test_full_refresh.py       18     11    39%
	 source_acceptance_test/tests/test_incremental.py        69     38    45%
	 source_acceptance_test/utils/__init__.py                 6      0   100%
	 source_acceptance_test/utils/asserts.py                 37      2    95%
	 source_acceptance_test/utils/common.py                  41     24    41%
	 source_acceptance_test/utils/compare.py                 47     20    57%
	 source_acceptance_test/utils/connector_runner.py        82     49    40%
	 source_acceptance_test/utils/json_schema_helper.py     111     11    90%
	 ------------------------------------------------------------------------
	 TOTAL                                                  856    416    51%
	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                       Stmts   Miss  Cover
	 --------------------------------------------------------------
	 base_python/__init__.py                       13      0   100%
	 base_python/catalog_helpers.py                10      6    40%
	 base_python/cdk/__init__.py                    0      0   100%
	 base_python/cdk/abstract_source.py            83     59    29%
	 base_python/cdk/streams/__init__.py            0      0   100%
	 base_python/cdk/streams/auth/__init__.py       0      0   100%
	 base_python/cdk/streams/auth/core.py           8      1    88%
	 base_python/cdk/streams/auth/jwt.py            5      5     0%
	 base_python/cdk/streams/auth/oauth.py         37     26    30%
	 base_python/cdk/streams/auth/token.py          9      4    56%
	 base_python/cdk/streams/core.py               63     32    49%
	 base_python/cdk/streams/exceptions.py         10      2    80%
	 base_python/cdk/streams/http.py               67     33    51%
	 base_python/cdk/streams/rate_limiting.py      30     14    53%
	 base_python/cdk/utils/__init__.py              0      0   100%
	 base_python/cdk/utils/casing.py                4      0   100%
	 base_python/client.py                         56     33    41%
	 base_python/entrypoint.py                     70     56    20%
	 base_python/integration.py                    52     25    52%
	 base_python/logger.py                         33     19    42%
	 base_python/schema_helpers.py                 56     41    27%
	 base_python/source.py                         51     34    33%
	 main_dev.py                                    3      3     0%
	 --------------------------------------------------------------
	 TOTAL                                        660    393    40%
	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                             Stmts   Miss  Cover
	 ----------------------------------------------------
	 main_dev.py                          5      5     0%
	 source_file_secure/__init__.py       2      0   100%
	 source_file_secure/source.py        42      2    95%
	 ----------------------------------------------------
	 TOTAL                               49      7    86%

…zendesk_support/streams.py Co-authored-by: George Claireaux <george@claireaux.co.uk>

docs/integrations/sources/file.md

airbyte-integrations/connectors/source-file-secure/source_file_secure/source.py

sherifnada · 2021-10-20T05:47:49Z

I agree with your reasoning about why this is the correct path forward for the time being, all other "good" alternatives are very heavyweight. Let's go with this for now then.

I think I'm hesitant to introduce an environment variable to solve the platform issue because it couples the connectors more closely with the platform (it is somewhat of a hidden interface too). Let's go with this pattern for now, I think there won't be many connectors that require this hopefully

antixar · 2021-10-20T08:32:12Z

/test connector=connectors/source-file-secure

🕑 connectors/source-file-secure https://github.com/airbytehq/airbyte/actions/runs/1362695477
❌ connectors/source-file-secure https://github.com/airbytehq/airbyte/actions/runs/1362695477
🐛 https://gradle.com/s/fuew7sfcc76a4

antixar · 2021-10-20T09:14:08Z

/test connector=connectors/source-file-secure

🕑 connectors/source-file-secure https://github.com/airbytehq/airbyte/actions/runs/1362847075
✅ connectors/source-file-secure https://github.com/airbytehq/airbyte/actions/runs/1362847075
Python tests coverage:

	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                                 Stmts   Miss  Cover
	 ------------------------------------------------------------------------
	 source_acceptance_test/__init__.py                       2      0   100%
	 source_acceptance_test/base.py                          10      4    60%
	 source_acceptance_test/config.py                        74      8    89%
	 source_acceptance_test/conftest.py                     108    108     0%
	 source_acceptance_test/plugin.py                        47     47     0%
	 source_acceptance_test/tests/__init__.py                 4      0   100%
	 source_acceptance_test/tests/test_core.py              200     94    53%
	 source_acceptance_test/tests/test_full_refresh.py       18     11    39%
	 source_acceptance_test/tests/test_incremental.py        69     38    45%
	 source_acceptance_test/utils/__init__.py                 6      0   100%
	 source_acceptance_test/utils/asserts.py                 37      2    95%
	 source_acceptance_test/utils/common.py                  41     24    41%
	 source_acceptance_test/utils/compare.py                 47     20    57%
	 source_acceptance_test/utils/connector_runner.py        82     49    40%
	 source_acceptance_test/utils/json_schema_helper.py     115     14    88%
	 ------------------------------------------------------------------------
	 TOTAL                                                  860    419    51%
	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                       Stmts   Miss  Cover
	 --------------------------------------------------------------
	 base_python/__init__.py                       13      0   100%
	 base_python/catalog_helpers.py                10      6    40%
	 base_python/cdk/__init__.py                    0      0   100%
	 base_python/cdk/abstract_source.py            83     59    29%
	 base_python/cdk/streams/__init__.py            0      0   100%
	 base_python/cdk/streams/auth/__init__.py       0      0   100%
	 base_python/cdk/streams/auth/core.py           8      1    88%
	 base_python/cdk/streams/auth/jwt.py            5      5     0%
	 base_python/cdk/streams/auth/oauth.py         37     26    30%
	 base_python/cdk/streams/auth/token.py          9      4    56%
	 base_python/cdk/streams/core.py               63     32    49%
	 base_python/cdk/streams/exceptions.py         10      2    80%
	 base_python/cdk/streams/http.py               67     33    51%
	 base_python/cdk/streams/rate_limiting.py      30     14    53%
	 base_python/cdk/utils/__init__.py              0      0   100%
	 base_python/cdk/utils/casing.py                4      0   100%
	 base_python/client.py                         56     33    41%
	 base_python/entrypoint.py                     70     56    20%
	 base_python/integration.py                    52     25    52%
	 base_python/logger.py                         33     19    42%
	 base_python/schema_helpers.py                 56     41    27%
	 base_python/source.py                         51     34    33%
	 main_dev.py                                    3      3     0%
	 --------------------------------------------------------------
	 TOTAL                                        660    393    40%
	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                             Stmts   Miss  Cover
	 ----------------------------------------------------
	 main_dev.py                          5      5     0%
	 source_file_secure/__init__.py       2      0   100%
	 source_file_secure/source.py        45      2    96%
	 ----------------------------------------------------
	 TOTAL                               52      7    87%

antixar · 2021-10-20T09:25:30Z

This PR should be merged without a publish comment because this connector will be used by cloud only and will be published separately.

marcosmarxm · 2021-10-20T16:19:26Z

@antixar this is PR is breaking the /publish command.

##[endgroup***
##[debug***/usr/bin/bash -e /actions-runner/_work/_temp/96a960e6-9513-475f-bf5b-0c71f547bbc8.sh
Creds not set for source-file-secure
##[error***Process completed with exit code 1.
##[debug***Finishing: Write Integration Test Credentials

See: https://github.com/airbytehq/airbyte/runs/3954100006?check_suite_focus=true
From: #7218

* create a fork for the source-file connector * update docs and tests * update docs and tests * add secrets Co-authored-by: George Claireaux <george@claireaux.co.uk> * switching among auth methods * refactoring after reviews * correction of doc * update spec file Co-authored-by: Maksym Pavlenok <maksym.pavlenok@globallogic.com> Co-authored-by: George Claireaux <george@claireaux.co.uk>

Maksym Pavlenok added 7 commits September 28, 2021 19:15

fix 404 responses for the ticket_comments stream

87d02f5

add unit test

a873a6d

add unit test

274a195

Merge remote-tracking branch 'origin' into antixar/6285-zendesk-suppo…

72108cf

…rt-oauth2

add oauth2 access token

d8d6058

create a fork for the source-file connector

92bdcf2

Merge remote-tracking branch 'origin' into antixar/6538-source-file-s…

96d41ea

…ecure

antixar self-assigned this Oct 5, 2021

antixar linked an issue Oct 5, 2021 that may be closed by this pull request

forking the File connector to disallow pulling files from the local filesystem #6538

Closed

4 tasks

antixar temporarily deployed to more-secrets October 5, 2021 15:32 Inactive

github-actions bot added the area/connectors Connector related issues label Oct 5, 2021

update docs and tests

39063bd

github-actions bot added the area/documentation Improvements or additions to documentation label Oct 6, 2021

antixar temporarily deployed to more-secrets October 6, 2021 18:35 Inactive

jrhizor temporarily deployed to more-secrets October 6, 2021 18:38 Inactive

update docs and tests

0502a4b

antixar temporarily deployed to more-secrets October 6, 2021 18:47 Inactive

jrhizor temporarily deployed to more-secrets October 6, 2021 18:48 Inactive

add secrets

195c6d8

antixar temporarily deployed to more-secrets October 6, 2021 19:44 Inactive

jrhizor temporarily deployed to more-secrets October 6, 2021 19:45 Inactive

antixar requested review from bazarnov and midavadim October 6, 2021 19:57

Update airbyte-integrations/connectors/source-zendesk-support/source_…

9f98748

…zendesk_support/streams.py Co-authored-by: George Claireaux <george@claireaux.co.uk>

bazarnov reviewed Oct 8, 2021

View reviewed changes

docs/integrations/sources/file.md Outdated Show resolved Hide resolved

airbyte-integrations/connectors/source-file-secure/source_file_secure/source.py Outdated Show resolved Hide resolved

switching among auth methods

7df812c

antixar requested a review from sherifnada October 19, 2021 21:48

sherifnada approved these changes Oct 20, 2021

View reviewed changes

merge with master

a3a155b

antixar temporarily deployed to more-secrets October 20, 2021 07:54 Inactive

reset changes of other connectors

4a123e2

antixar temporarily deployed to more-secrets October 20, 2021 08:02 Inactive

reset changes of other connectors

7b2f0e8

antixar temporarily deployed to more-secrets October 20, 2021 08:08 Inactive

reset changes of other connectors

6bcd363

antixar temporarily deployed to more-secrets October 20, 2021 08:11 Inactive

reset changes of other connectors

66de336

antixar temporarily deployed to more-secrets October 20, 2021 08:14 Inactive

reset changes of other connectors

f21d96f

antixar temporarily deployed to more-secrets October 20, 2021 08:26 Inactive

antixar added 2 commits October 20, 2021 11:26

reset changes of other connectors

303aaa2

Merge branch 'master' into antixar/6538-source-file-secure

c57a027

antixar temporarily deployed to more-secrets October 20, 2021 08:30 Inactive

jrhizor temporarily deployed to more-secrets October 20, 2021 08:34 Inactive

jrhizor temporarily deployed to more-secrets October 20, 2021 09:16 Inactive

antixar merged commit 059645f into master Oct 20, 2021

antixar deleted the antixar/6538-source-file-secure branch October 20, 2021 09:28

jrhizor mentioned this pull request Oct 21, 2021

Bump Airbyte version from 0.30.21-alpha to 0.30.22-alpha #7264

Merged

ycherniaiev added connectors/source/file connectors/sources-files labels Jan 10, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🎉 Source File: separate secure fork #6768

🎉 Source File: separate secure fork #6768

antixar commented Oct 5, 2021 •

edited

CLAassistant commented Oct 5, 2021 •

edited

antixar commented Oct 6, 2021 •

edited by github-actions bot

antixar commented Oct 6, 2021 •

edited by github-actions bot

antixar commented Oct 6, 2021 •

edited by github-actions bot

sherifnada commented Oct 20, 2021 •

edited

antixar commented Oct 20, 2021 •

edited by github-actions bot

antixar commented Oct 20, 2021 •

edited by github-actions bot

antixar commented Oct 20, 2021

marcosmarxm commented Oct 20, 2021

🎉 Source File: separate secure fork #6768

🎉 Source File: separate secure fork #6768

Conversation

antixar commented Oct 5, 2021 • edited

What

How

Pre-merge Checklist

Community member or Airbyter

Airbyter

CLAassistant commented Oct 5, 2021 • edited

antixar commented Oct 6, 2021 • edited by github-actions bot

antixar commented Oct 6, 2021 • edited by github-actions bot

antixar commented Oct 6, 2021 • edited by github-actions bot

sherifnada commented Oct 20, 2021 • edited

antixar commented Oct 20, 2021 • edited by github-actions bot

antixar commented Oct 20, 2021 • edited by github-actions bot

antixar commented Oct 20, 2021

marcosmarxm commented Oct 20, 2021

antixar commented Oct 5, 2021 •

edited

CLAassistant commented Oct 5, 2021 •

edited

antixar commented Oct 6, 2021 •

edited by github-actions bot

antixar commented Oct 6, 2021 •

edited by github-actions bot

antixar commented Oct 6, 2021 •

edited by github-actions bot

sherifnada commented Oct 20, 2021 •

edited

antixar commented Oct 20, 2021 •

edited by github-actions bot

antixar commented Oct 20, 2021 •

edited by github-actions bot