Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update fields in source-connectors specifications: file, freshdesk, github, google-directory, google-workspace-admin-reports, iterable, kafka #8524

Merged
merged 25 commits into from
Dec 10, 2021

Conversation

igrankova
Copy link
Contributor

@igrankova igrankova commented Dec 6, 2021

What

Add title and description properties to each field in source-connectors specifications:
source-file
source-freshdesk
source-github
source-google-directory
source-google-workspace-admin-reports
source-iterable
source-kafka

How

Describe the solution

Recommended reading order

  1. x.java
  2. y.python

🚨 User Impact 🚨

Are there any breaking changes? What is the end result perceived by the user? If yes, please merge this PR with the 🚨🚨 emoji so changelog authors can further highlight this if needed.

Pre-merge Checklist

Expand the relevant checklist and delete the others.

Updating a connector

Community member or Airbyter

  • Grant edit access to maintainers (instructions)
  • Secrets in the connector's spec are annotated with airbyte_secret
  • Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
  • Code reviews completed
  • Documentation updated
    • Connector's README.md
    • Connector's bootstrap.md. See description and examples
    • Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
  • PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

  • Create a non-forked branch based on this PR and test the below items on it
  • Build is successful
  • Credentials added to Github CI. Instructions.
  • /test connector=connectors/<name> command is passing.
  • New Connector version released on Dockerhub by running the /publish command described here
  • After the new connector version is published, connector version bumped in the seed directory as described here
  • Seed specs have been re-generated by building the platform and committing the changes to the seed spec files, as described here

@igrankova igrankova self-assigned this Dec 6, 2021
@CLAassistant
Copy link

CLAassistant commented Dec 6, 2021

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
2 out of 3 committers have signed the CLA.

✅ igrankova
✅ grubberr
❌ Your Name


Your Name seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
You have signed the CLA already but the status is still pending? Let us recheck it.

@igrankova igrankova temporarily deployed to more-secrets December 6, 2021 08:44 Inactive
@igrankova igrankova temporarily deployed to more-secrets December 6, 2021 08:48 Inactive
@igrankova igrankova temporarily deployed to more-secrets December 6, 2021 08:52 Inactive
@igrankova igrankova temporarily deployed to more-secrets December 6, 2021 11:13 Inactive
@github-actions github-actions bot added the area/documentation Improvements or additions to documentation label Dec 6, 2021
@igrankova igrankova temporarily deployed to more-secrets December 6, 2021 12:42 Inactive
@grubberr grubberr temporarily deployed to more-secrets December 6, 2021 18:14 Inactive
@igrankova igrankova changed the title Update fields in source-connectors specifications: file, file-secure, freshdesk, github, google-directory, google-workspace-admin-reports, instagram, iterable, kafka, klaviyo Update fields in source-connectors specifications: file, file-secure, freshdesk, github, google-directory, google-workspace-admin-reports, instagram, iterable, klaviyo Dec 6, 2021
@igrankova igrankova temporarily deployed to more-secrets December 7, 2021 08:21 Inactive
@igrankova
Copy link
Contributor Author

igrankova commented Dec 7, 2021

/test connector=connectors/source-file

🕑 connectors/source-file https://github.com/airbytehq/airbyte/actions/runs/1548626648
✅ connectors/source-file https://github.com/airbytehq/airbyte/actions/runs/1548626648
Python tests coverage:

	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                                 Stmts   Miss  Cover
	 ------------------------------------------------------------------------
	 source_acceptance_test/__init__.py                       2      0   100%
	 source_acceptance_test/base.py                          10      4    60%
	 source_acceptance_test/config.py                        76      8    89%
	 source_acceptance_test/conftest.py                     109    109     0%
	 source_acceptance_test/plugin.py                        47     47     0%
	 source_acceptance_test/tests/__init__.py                 4      0   100%
	 source_acceptance_test/tests/test_core.py              235     95    60%
	 source_acceptance_test/tests/test_full_refresh.py       38     27    29%
	 source_acceptance_test/tests/test_incremental.py        69     38    45%
	 source_acceptance_test/utils/__init__.py                 6      0   100%
	 source_acceptance_test/utils/asserts.py                 37      2    95%
	 source_acceptance_test/utils/common.py                  54     24    56%
	 source_acceptance_test/utils/compare.py                 62     25    60%
	 source_acceptance_test/utils/connector_runner.py        82     49    40%
	 source_acceptance_test/utils/json_schema_helper.py     115     14    88%
	 ------------------------------------------------------------------------
	 TOTAL                                                  946    442    53%
	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                      Stmts   Miss  Cover
	 ---------------------------------------------
	 source_file/__init__.py       2      0   100%
	 source_file/client.py       237     37    84%
	 source_file/source.py        50     35    30%
	 ---------------------------------------------
	 TOTAL                       289     72    75%

@jrhizor jrhizor temporarily deployed to more-secrets December 7, 2021 09:03 Inactive
@igrankova
Copy link
Contributor Author

igrankova commented Dec 7, 2021

/test connector=connectors/source-file-secure

🕑 connectors/source-file-secure https://github.com/airbytehq/airbyte/actions/runs/1548640571
❌ connectors/source-file-secure https://github.com/airbytehq/airbyte/actions/runs/1548640571
🐛 https://gradle.com/s/z7dta7vtgsqbs

@igrankova
Copy link
Contributor Author

igrankova commented Dec 7, 2021

/test connector=connectors/source-freshdesk

🕑 connectors/source-freshdesk https://github.com/airbytehq/airbyte/actions/runs/1548639945
✅ connectors/source-freshdesk https://github.com/airbytehq/airbyte/actions/runs/1548639945
Python tests coverage:

	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                       Stmts   Miss  Cover
	 --------------------------------------------------------------
	 base_python/__init__.py                       13      0   100%
	 base_python/catalog_helpers.py                10      6    40%
	 base_python/cdk/__init__.py                    0      0   100%
	 base_python/cdk/abstract_source.py            89     64    28%
	 base_python/cdk/streams/__init__.py            0      0   100%
	 base_python/cdk/streams/auth/__init__.py       0      0   100%
	 base_python/cdk/streams/auth/core.py           8      1    88%
	 base_python/cdk/streams/auth/jwt.py            5      5     0%
	 base_python/cdk/streams/auth/oauth.py         37     26    30%
	 base_python/cdk/streams/auth/token.py          9      4    56%
	 base_python/cdk/streams/core.py               63     32    49%
	 base_python/cdk/streams/exceptions.py         10      2    80%
	 base_python/cdk/streams/http.py               67     33    51%
	 base_python/cdk/streams/rate_limiting.py      30     14    53%
	 base_python/cdk/utils/__init__.py              0      0   100%
	 base_python/cdk/utils/casing.py                4      0   100%
	 base_python/cdk/utils/event_timing.py         47      3    94%
	 base_python/client.py                         56     33    41%
	 base_python/entrypoint.py                     70     56    20%
	 base_python/integration.py                    52     25    52%
	 base_python/logger.py                         33     15    55%
	 base_python/schema_helpers.py                 56     41    27%
	 base_python/source.py                         51     34    33%
	 main_dev.py                                    3      3     0%
	 --------------------------------------------------------------
	 TOTAL                                        713    397    44%
	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                           Stmts   Miss  Cover
	 --------------------------------------------------
	 main_dev.py                        6      6     0%
	 source_freshdesk/__init__.py       2      0   100%
	 source_freshdesk/api.py          190     86    55%
	 source_freshdesk/client.py        38     10    74%
	 source_freshdesk/errors.py         8      0   100%
	 source_freshdesk/source.py         4      0   100%
	 source_freshdesk/utils.py         42     23    45%
	 --------------------------------------------------
	 TOTAL                            290    125    57%
	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                           Stmts   Miss  Cover
	 --------------------------------------------------
	 main_dev.py                        6      6     0%
	 source_freshdesk/__init__.py       2      0   100%
	 source_freshdesk/api.py          190     75    61%
	 source_freshdesk/client.py        38     18    53%
	 source_freshdesk/errors.py         8      0   100%
	 source_freshdesk/source.py         4      0   100%
	 source_freshdesk/utils.py         42      2    95%
	 --------------------------------------------------
	 TOTAL                            290    101    65%

@igrankova
Copy link
Contributor Author

igrankova commented Dec 7, 2021

/test connector=connectors/source-github

🕑 connectors/source-github https://github.com/airbytehq/airbyte/actions/runs/1548643244
✅ connectors/source-github https://github.com/airbytehq/airbyte/actions/runs/1548643244
Python tests coverage:

	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                                 Stmts   Miss  Cover
	 ------------------------------------------------------------------------
	 source_acceptance_test/__init__.py                       2      0   100%
	 source_acceptance_test/base.py                          10      4    60%
	 source_acceptance_test/config.py                        76      8    89%
	 source_acceptance_test/conftest.py                     109    109     0%
	 source_acceptance_test/plugin.py                        47     47     0%
	 source_acceptance_test/tests/__init__.py                 4      0   100%
	 source_acceptance_test/tests/test_core.py              235     95    60%
	 source_acceptance_test/tests/test_full_refresh.py       38     27    29%
	 source_acceptance_test/tests/test_incremental.py        69     38    45%
	 source_acceptance_test/utils/__init__.py                 6      0   100%
	 source_acceptance_test/utils/asserts.py                 37      2    95%
	 source_acceptance_test/utils/common.py                  54     24    56%
	 source_acceptance_test/utils/compare.py                 62     25    60%
	 source_acceptance_test/utils/connector_runner.py        82     49    40%
	 source_acceptance_test/utils/json_schema_helper.py     115     14    88%
	 ------------------------------------------------------------------------
	 TOTAL                                                  946    442    53%
	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                        Stmts   Miss  Cover
	 -----------------------------------------------
	 source_github/__init__.py       2      0   100%
	 source_github/source.py        76     34    55%
	 source_github/streams.py      363    176    52%
	 -----------------------------------------------
	 TOTAL                         441    210    52%

@igrankova
Copy link
Contributor Author

igrankova commented Dec 7, 2021

/test connector=connectors/source-google-directory

🕑 connectors/source-google-directory https://github.com/airbytehq/airbyte/actions/runs/1548643838
✅ connectors/source-google-directory https://github.com/airbytehq/airbyte/actions/runs/1548643838
Python tests coverage:

	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                                 Stmts   Miss  Cover
	 ------------------------------------------------------------------------
	 source_acceptance_test/__init__.py                       2      0   100%
	 source_acceptance_test/base.py                          10      4    60%
	 source_acceptance_test/config.py                        76      8    89%
	 source_acceptance_test/conftest.py                     109    109     0%
	 source_acceptance_test/plugin.py                        47     47     0%
	 source_acceptance_test/tests/__init__.py                 4      0   100%
	 source_acceptance_test/tests/test_core.py              235     95    60%
	 source_acceptance_test/tests/test_full_refresh.py       38     27    29%
	 source_acceptance_test/tests/test_incremental.py        69     38    45%
	 source_acceptance_test/utils/__init__.py                 6      0   100%
	 source_acceptance_test/utils/asserts.py                 37      2    95%
	 source_acceptance_test/utils/common.py                  54     24    56%
	 source_acceptance_test/utils/compare.py                 62     25    60%
	 source_acceptance_test/utils/connector_runner.py        82     49    40%
	 source_acceptance_test/utils/json_schema_helper.py     115     14    88%
	 ------------------------------------------------------------------------
	 TOTAL                                                  946    442    53%
	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                  Stmts   Miss  Cover
	 
	 ---------------------------------------------------------
	   self._warn("No data was collected.", slug="no-data-collected")
	 source_google_directory/__init__.py       2      2     0%
	 source_google_directory/api.py           90     90     0%
	 source_google_directory/client.py        26     26     0%
	 source_google_directory/source.py         4      4     0%
	 source_google_directory/utils.py          3      3     0%
	 ---------------------------------------------------------
	 TOTAL                                   125    125     0%

@igrankova
Copy link
Contributor Author

igrankova commented Dec 7, 2021

/test connector=connectors/source-workspace-admin-reports

🕑 connectors/source-workspace-admin-reports https://github.com/airbytehq/airbyte/actions/runs/1548646354
❌ connectors/source-workspace-admin-reports https://github.com/airbytehq/airbyte/actions/runs/1548646354
🐛

@igrankova
Copy link
Contributor Author

igrankova commented Dec 7, 2021

/test connector=connectors/source-iterable

🕑 connectors/source-iterable https://github.com/airbytehq/airbyte/actions/runs/1548648027
✅ connectors/source-iterable https://github.com/airbytehq/airbyte/actions/runs/1548648027
Python tests coverage:

	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                                 Stmts   Miss  Cover
	 ------------------------------------------------------------------------
	 source_acceptance_test/__init__.py                       2      0   100%
	 source_acceptance_test/base.py                          10      4    60%
	 source_acceptance_test/config.py                        76      8    89%
	 source_acceptance_test/conftest.py                     109    109     0%
	 source_acceptance_test/plugin.py                        47     47     0%
	 source_acceptance_test/tests/__init__.py                 4      0   100%
	 source_acceptance_test/tests/test_core.py              235     95    60%
	 source_acceptance_test/tests/test_full_refresh.py       38     27    29%
	 source_acceptance_test/tests/test_incremental.py        69     38    45%
	 source_acceptance_test/utils/__init__.py                 6      0   100%
	 source_acceptance_test/utils/asserts.py                 37      2    95%
	 source_acceptance_test/utils/common.py                  54     24    56%
	 source_acceptance_test/utils/compare.py                 62     25    60%
	 source_acceptance_test/utils/connector_runner.py        82     49    40%
	 source_acceptance_test/utils/json_schema_helper.py     115     14    88%
	 ------------------------------------------------------------------------
	 TOTAL                                                  946    442    53%
	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                  Stmts   Miss  Cover
	 ---------------------------------------------------------
	 source_iterable/__init__.py               2      0   100%
	 source_iterable/api.py                  159     69    57%
	 source_iterable/iterable_streams.py     107     15    86%
	 source_iterable/slice_generators.py      73      0   100%
	 source_iterable/source.py                15      6    60%
	 ---------------------------------------------------------
	 TOTAL                                   356     90    75%

@jrhizor jrhizor temporarily deployed to more-secrets December 10, 2021 22:32 Inactive
@grubberr
Copy link
Contributor

grubberr commented Dec 10, 2021

/publish connector=connectors/source-iterable

🕑 connectors/source-iterable https://github.com/airbytehq/airbyte/actions/runs/1565549485
✅ connectors/source-iterable https://github.com/airbytehq/airbyte/actions/runs/1565549485

@grubberr
Copy link
Contributor

grubberr commented Dec 10, 2021

/publish connector=connectors/source-file

🕑 connectors/source-file https://github.com/airbytehq/airbyte/actions/runs/1565550802
✅ connectors/source-file https://github.com/airbytehq/airbyte/actions/runs/1565550802

@grubberr
Copy link
Contributor

grubberr commented Dec 10, 2021

/publish connector=connectors/source-kafka

🕑 connectors/source-kafka https://github.com/airbytehq/airbyte/actions/runs/1565550993
✅ connectors/source-kafka https://github.com/airbytehq/airbyte/actions/runs/1565550993

@grubberr
Copy link
Contributor

grubberr commented Dec 10, 2021

/publish connector=connectors/source-freshdesk

🕑 connectors/source-freshdesk https://github.com/airbytehq/airbyte/actions/runs/1565551327
✅ connectors/source-freshdesk https://github.com/airbytehq/airbyte/actions/runs/1565551327

@grubberr
Copy link
Contributor

grubberr commented Dec 10, 2021

/publish connector=connectors/source-github

🕑 connectors/source-github https://github.com/airbytehq/airbyte/actions/runs/1565551739
✅ connectors/source-github https://github.com/airbytehq/airbyte/actions/runs/1565551739

@grubberr
Copy link
Contributor

grubberr commented Dec 10, 2021

/publish connector=connectors/source-google-directory

🕑 connectors/source-google-directory https://github.com/airbytehq/airbyte/actions/runs/1565552139
✅ connectors/source-google-directory https://github.com/airbytehq/airbyte/actions/runs/1565552139

@grubberr
Copy link
Contributor

grubberr commented Dec 10, 2021

/publish connector=connectors/source-google-workspace-admin-reports

🕑 connectors/source-google-workspace-admin-reports https://github.com/airbytehq/airbyte/actions/runs/1565552267
✅ connectors/source-google-workspace-admin-reports https://github.com/airbytehq/airbyte/actions/runs/1565552267

@jrhizor jrhizor temporarily deployed to more-secrets December 10, 2021 23:14 Inactive
@jrhizor jrhizor temporarily deployed to more-secrets December 10, 2021 23:14 Inactive
@jrhizor jrhizor temporarily deployed to more-secrets December 10, 2021 23:14 Inactive
@jrhizor jrhizor temporarily deployed to more-secrets December 10, 2021 23:14 Inactive
@jrhizor jrhizor temporarily deployed to more-secrets December 10, 2021 23:14 Inactive
@jrhizor jrhizor temporarily deployed to more-secrets December 10, 2021 23:15 Inactive
@jrhizor jrhizor temporarily deployed to more-secrets December 10, 2021 23:15 Inactive
Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
@grubberr grubberr merged commit fefd575 into master Dec 10, 2021
@grubberr grubberr deleted the igrankova/issue-8366-update-title-description branch December 10, 2021 23:47
@grubberr grubberr temporarily deployed to more-secrets December 10, 2021 23:48 Inactive
jrhizor pushed a commit that referenced this pull request Dec 10, 2021
…ithub, google-directory, google-workspace-admin-reports, iterable (#8524)

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

Co-authored-by: Serhii Chvaliuk <grubberr@gmail.com>
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
jrhizor added a commit that referenced this pull request Dec 11, 2021
* upgrade gradle

* upgrade to Java 17 (and fix a few of the node versioning misses)

* oops

* try to run a different format version

* fix spotless by upgrading / reformatting some files

* fix ci settings

* upgrade mockito to avoid other errors

* undo bad format

* fix "incorrect" sql comments

* fmt

* add debug flag

* remove

* bump

* bump jooq to a version that has a java 17 dist

* fix

* remove logs

* oops

* revert jooq upgrade

* fix

* set up java for connector test

* fix yaml

* generate std source tests

* fail zombie job attempts and add failure reason (#8709)

* fail zombie job attempts and add failure reason

* remove failure reason

* bump gcp dependencies to pick up grpc update (#8713)

* Bump Airbyte version from 0.33.9-alpha to 0.33.10-alpha (#8714)

Co-authored-by: jrhizor <jrhizor@users.noreply.github.com>

* Change CDK "Caching" header to "nested streams & caching"

* Update fields in source-connectors specifications: file, freshdesk, github, google-directory, google-workspace-admin-reports, iterable (#8524)

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

Co-authored-by: Serhii Chvaliuk <grubberr@gmail.com>
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* move S3Config into destination-s3; update dependencies accordingly (#8562)

Co-authored-by: Lake Mossman <lake@airbyte.io>
Co-authored-by: jrhizor <jrhizor@users.noreply.github.com>
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
Co-authored-by: Iryna Grankova <87977540+igrankova@users.noreply.github.com>
Co-authored-by: Serhii Chvaliuk <grubberr@gmail.com>
Co-authored-by: Edward Gao <edward.gao@airbyte.io>
@igrankova igrankova changed the title Update fields in source-connectors specifications: file, freshdesk, github, google-directory, google-workspace-admin-reports, iterable Update fields in source-connectors specifications: file, freshdesk, github, google-directory, google-workspace-admin-reports, iterable, kafka Dec 28, 2021
schlattk pushed a commit to schlattk/airbyte that referenced this pull request Jan 4, 2022
…ithub, google-directory, google-workspace-admin-reports, iterable (airbytehq#8524)

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

Co-authored-by: Serhii Chvaliuk <grubberr@gmail.com>
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
schlattk pushed a commit to schlattk/airbyte that referenced this pull request Jan 4, 2022
* upgrade gradle

* upgrade to Java 17 (and fix a few of the node versioning misses)

* oops

* try to run a different format version

* fix spotless by upgrading / reformatting some files

* fix ci settings

* upgrade mockito to avoid other errors

* undo bad format

* fix "incorrect" sql comments

* fmt

* add debug flag

* remove

* bump

* bump jooq to a version that has a java 17 dist

* fix

* remove logs

* oops

* revert jooq upgrade

* fix

* set up java for connector test

* fix yaml

* generate std source tests

* fail zombie job attempts and add failure reason (airbytehq#8709)

* fail zombie job attempts and add failure reason

* remove failure reason

* bump gcp dependencies to pick up grpc update (airbytehq#8713)

* Bump Airbyte version from 0.33.9-alpha to 0.33.10-alpha (airbytehq#8714)

Co-authored-by: jrhizor <jrhizor@users.noreply.github.com>

* Change CDK "Caching" header to "nested streams & caching"

* Update fields in source-connectors specifications: file, freshdesk, github, google-directory, google-workspace-admin-reports, iterable (airbytehq#8524)

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

Co-authored-by: Serhii Chvaliuk <grubberr@gmail.com>
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

* move S3Config into destination-s3; update dependencies accordingly (airbytehq#8562)

Co-authored-by: Lake Mossman <lake@airbyte.io>
Co-authored-by: jrhizor <jrhizor@users.noreply.github.com>
Co-authored-by: Sherif A. Nada <snadalive@gmail.com>
Co-authored-by: Iryna Grankova <87977540+igrankova@users.noreply.github.com>
Co-authored-by: Serhii Chvaliuk <grubberr@gmail.com>
Co-authored-by: Edward Gao <edward.gao@airbyte.io>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
No open projects
Archived in project
6 participants