Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Prepare Google Sheets, Google Drive, Apify dataset and connector templates for airbyte-lib #33414

Merged
merged 11 commits into from Dec 19, 2023
Merged

Prepare Google Sheets, Google Drive, Apify dataset and connector templates for airbyte-lib #33414

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
1c84bef
prepare for airbyte-lib
Dec 13, 2023
39b92d4
update changelog
Dec 13, 2023
e1b48f2
format
Dec 13, 2023
a16eefb
Merge remote-tracking branch 'origin/master' into flash1293/prepare-c…
Dec 15, 2023
b0e1700
switch bumped connectors to the ones that work
Dec 15, 2023
5bcabe1
revert facebook marketing
Dec 15, 2023
ac60d10
Merge remote-tracking branch 'origin/master' into flash1293/prepare-c…
Dec 15, 2023
a29edb3
convert google drive
Dec 15, 2023
c777042
revert
Dec 15, 2023
8fdac37
revert
Dec 15, 2023
4eeda48
review comment
Dec 18, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
9 changes: 2 additions & 7 deletions airbyte-integrations/connector-templates/source-python-http-api/main.py.hbs
View file
Open in desktop
Expand Up @@ -2,12 +2,7 @@
# Copyright (c) 2023 Airbyte, Inc., all rights reserved.
#


import sys

from airbyte_cdk.entrypoint import launch
from source_{{snakeCase name}} import Source{{properCase name}}
from source_{{snakeCase name}}.run import run

if __name__ == "__main__":
source = Source{{properCase name}}()
launch(source, sys.argv[1:])
run()
5 changes: 5 additions & 0 deletions airbyte-integrations/connector-templates/source-python-http-api/setup.py.hbs
View file
Open in desktop
Expand Up @@ -27,4 +27,9 @@ setup(
extras_require={
"tests": TEST_REQUIREMENTS,
},
entry_points={
"console_scripts": [
"source-{{dashCase name}}=source_{{snakeCase name}}.run:run",
],
},
)
13 changes: 13 additions & 0 deletions ...egrations/connector-templates/source-python-http-api/source_{{snakeCase name}}/run.py.hbs
View file
Open in desktop
@@ -0,0 +1,13 @@
#
# Copyright (c) 2023 Airbyte, Inc., all rights reserved.
#


import sys

from airbyte_cdk.entrypoint import launch
from .source import Source{{properCase name}}

def run():
source = Source{{properCase name}}()
launch(source, sys.argv[1:])
9 changes: 2 additions & 7 deletions airbyte-integrations/connector-templates/source-python/main.py.hbs
View file
Open in desktop
Expand Up @@ -2,12 +2,7 @@
# Copyright (c) 2023 Airbyte, Inc., all rights reserved.
#


import sys

from airbyte_cdk.entrypoint import launch
from source_{{snakeCase name}} import Source{{properCase name}}
from source_{{snakeCase name}}.run import run

if __name__ == "__main__":
source = Source{{properCase name}}()
launch(source, sys.argv[1:])
run()
5 changes: 5 additions & 0 deletions airbyte-integrations/connector-templates/source-python/setup.py.hbs
View file
Open in desktop
Expand Up @@ -27,4 +27,9 @@ setup(
extras_require={
"tests": TEST_REQUIREMENTS,
},
entry_points={
"console_scripts": [
"source-{{dashCase name}}=source_{{snakeCase name}}.run:run",
],
},
)
13 changes: 13 additions & 0 deletions airbyte-integrations/connector-templates/source-python/source_{{snakeCase name}}/run.py.hbs
View file
Open in desktop
@@ -0,0 +1,13 @@
#
# Copyright (c) 2023 Airbyte, Inc., all rights reserved.
#


import sys

from airbyte_cdk.entrypoint import launch
from .source import Source{{properCase name}}

if __name__ == "__main__":
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why is the template file different from the connector's implementation?

I'd expect this file to only contain the run method and for the if __name__ == "__main__" to be in the main.py file

source = Source{{properCase name}}()
launch(source, sys.argv[1:])
2 changes: 1 addition & 1 deletion airbyte-integrations/connectors/source-apify-dataset/Dockerfile
View file
Open in desktop
Expand Up @@ -34,5 +34,5 @@ COPY source_apify_dataset ./source_apify_dataset
ENV AIRBYTE_ENTRYPOINT "python /airbyte/integration_code/main.py"
ENTRYPOINT ["python", "/airbyte/integration_code/main.py"]

LABEL io.airbyte.version=2.1.0
LABEL io.airbyte.version=2.1.1
LABEL io.airbyte.name=airbyte/source-apify-dataset
9 changes: 2 additions & 7 deletions airbyte-integrations/connectors/source-apify-dataset/main.py
View file
Open in desktop
Expand Up @@ -2,12 +2,7 @@
# Copyright (c) 2023 Airbyte, Inc., all rights reserved.
#


import sys

from airbyte_cdk.entrypoint import launch
from source_apify_dataset import SourceApifyDataset
from source_apify_dataset.run import run

if __name__ == "__main__":
source = SourceApifyDataset()
launch(source, sys.argv[1:])
run()
2 changes: 1 addition & 1 deletion airbyte-integrations/connectors/source-apify-dataset/metadata.yaml
View file
Open in desktop
Expand Up @@ -10,7 +10,7 @@ data:
connectorSubtype: api
connectorType: source
definitionId: 47f17145-fe20-4ef5-a548-e29b048adf84
dockerImageTag: 2.1.0
dockerImageTag: 2.1.1
dockerRepository: airbyte/source-apify-dataset
githubIssueLabel: source-apify-dataset
icon: apify.svg
Expand Down
5 changes: 5 additions & 0 deletions airbyte-integrations/connectors/source-apify-dataset/setup.py
View file
Open in desktop
Expand Up @@ -20,4 +20,9 @@
extras_require={
"tests": TEST_REQUIREMENTS,
},
entry_points={
"console_scripts": [
"source-apify-dataset=source_apify_dataset.run:run",
],
},
)
15 changes: 15 additions & 0 deletions airbyte-integrations/connectors/source-apify-dataset/source_apify_dataset/run.py
View file
Open in desktop
@@ -0,0 +1,15 @@
#
# Copyright (c) 2023 Airbyte, Inc., all rights reserved.
#


import sys

from airbyte_cdk.entrypoint import launch

from .source import SourceApifyDataset


def run():
source = SourceApifyDataset()
launch(source, sys.argv[1:])
12 changes: 2 additions & 10 deletions airbyte-integrations/connectors/source-google-drive/main.py
View file
Open in desktop
Expand Up @@ -2,15 +2,7 @@
# Copyright (c) 2023 Airbyte, Inc., all rights reserved.
#


import sys

from airbyte_cdk import AirbyteEntrypoint
from airbyte_cdk.entrypoint import launch
from source_google_drive import SourceGoogleDrive
from source_google_drive.run import run

if __name__ == "__main__":
args = sys.argv[1:]
catalog_path = AirbyteEntrypoint.extract_catalog(args)
source = SourceGoogleDrive(catalog_path)
launch(source, args)
run()
2 changes: 1 addition & 1 deletion airbyte-integrations/connectors/source-google-drive/metadata.yaml
View file
Open in desktop
Expand Up @@ -7,7 +7,7 @@ data:
connectorSubtype: file
connectorType: source
definitionId: 9f8dda77-1048-4368-815b-269bf54ee9b8
dockerImageTag: 0.0.5
dockerImageTag: 0.0.6
dockerRepository: airbyte/source-google-drive
githubIssueLabel: source-google-drive
icon: google-drive.svg
Expand Down
5 changes: 5 additions & 0 deletions airbyte-integrations/connectors/source-google-drive/setup.py
View file
Open in desktop
Expand Up @@ -29,4 +29,9 @@
extras_require={
"tests": TEST_REQUIREMENTS,
},
entry_points={
"console_scripts": [
"source-google-drive=source_google_drive.run:run",
],
},
)
17 changes: 17 additions & 0 deletions airbyte-integrations/connectors/source-google-drive/source_google_drive/run.py
View file
Open in desktop
@@ -0,0 +1,17 @@
#
# Copyright (c) 2023 Airbyte, Inc., all rights reserved.
#


import sys

from airbyte_cdk import AirbyteEntrypoint
from airbyte_cdk.entrypoint import launch
from source_google_drive import SourceGoogleDrive


def run():
args = sys.argv[1:]
catalog_path = AirbyteEntrypoint.extract_catalog(args)
source = SourceGoogleDrive(catalog_path)
launch(source, args)
9 changes: 2 additions & 7 deletions airbyte-integrations/connectors/source-google-sheets/main.py
View file
Open in desktop
Expand Up @@ -2,12 +2,7 @@
# Copyright (c) 2023 Airbyte, Inc., all rights reserved.
#


import sys

from airbyte_cdk.entrypoint import launch
from source_google_sheets import SourceGoogleSheets
from source_google_sheets.run import run

if __name__ == "__main__":
source = SourceGoogleSheets()
launch(source, sys.argv[1:])
run()
2 changes: 1 addition & 1 deletion airbyte-integrations/connectors/source-google-sheets/metadata.yaml
View file
Open in desktop
Expand Up @@ -10,7 +10,7 @@ data:
connectorSubtype: file
connectorType: source
definitionId: 71607ba1-c0ac-4799-8049-7f4b90dd50f7
dockerImageTag: 0.3.11
dockerImageTag: 0.3.12
dockerRepository: airbyte/source-google-sheets
documentationUrl: https://docs.airbyte.com/integrations/sources/google-sheets
githubIssueLabel: source-google-sheets
Expand Down
5 changes: 5 additions & 0 deletions airbyte-integrations/connectors/source-google-sheets/setup.py
View file
Open in desktop
Expand Up @@ -33,4 +33,9 @@
extras_require={
"tests": TEST_REQUIREMENTS,
},
entry_points={
"console_scripts": [
"source-google-sheets=source_google_sheets.run:run",
],
},
)
15 changes: 15 additions & 0 deletions airbyte-integrations/connectors/source-google-sheets/source_google_sheets/run.py
View file
Open in desktop
@@ -0,0 +1,15 @@
#
# Copyright (c) 2023 Airbyte, Inc., all rights reserved.
#


import sys

from airbyte_cdk.entrypoint import launch

from .source import SourceGoogleSheets


def run():
source = SourceGoogleSheets()
launch(source, sys.argv[1:])
1 change: 1 addition & 0 deletions docs/integrations/sources/apify-dataset.md
View file
Open in desktop
Expand Up @@ -69,6 +69,7 @@ The Apify dataset connector uses [Apify Python Client](https://docs.apify.com/ap

| Version | Date | Pull Request | Subject |
| :------ | :--------- | :----------------------------------------------------------- | :-------------------------------------------------------------------------- |
| 2.1.1 | 2023-12-14 | [33414](https://github.com/airbytehq/airbyte/pull/33414) | Prepare for airbyte-lib |
| 2.1.0 | 2023-10-13 | [31333](https://github.com/airbytehq/airbyte/pull/31333) | Add stream for arbitrary datasets |
| 2.0.0 | 2023-09-18 | [30428](https://github.com/airbytehq/airbyte/pull/30428) | Fix broken stream, manifest refactor |
| 1.0.0 | 2023-08-25 | [29859](https://github.com/airbytehq/airbyte/pull/29859) | Migrate to lowcode |
Expand Down
1 change: 1 addition & 0 deletions docs/integrations/sources/google-drive.md
View file
Open in desktop
Expand Up @@ -247,6 +247,7 @@ Before parsing each document, the connector exports Google Document files to Doc

| Version | Date | Pull Request | Subject |
|---------|------------|-----------------------------------------------------------|--------------------------------------------------------------|
| 0.0.6 | 2023-12-16 | [33414](https://github.com/airbytehq/airbyte/pull/33414) | Prepare for airbyte-lib |
| 0.0.5 | 2023-12-14 | [33411](https://github.com/airbytehq/airbyte/pull/33411) | Bump CDK version to auto-set primary key for document file streams and support raw txt files |
| 0.0.4 | 2023-12-06 | [33187](https://github.com/airbytehq/airbyte/pull/33187) | Bump CDK version to hide source-defined primary key |
| 0.0.3 | 2023-11-16 | [31458](https://github.com/airbytehq/airbyte/pull/31458) | Improve folder id input and update document file type parser |
Expand Down
1 change: 1 addition & 0 deletions docs/integrations/sources/google-sheets.md
View file
Open in desktop
Expand Up @@ -151,6 +151,7 @@ Airbyte batches requests to the API in order to efficiently pull data and respec

| Version | Date | Pull Request | Subject |
|---------|------------|----------------------------------------------------------|-----------------------------------------------------------------------------------|
| 0.3.12 | 2023-12-14 | [33414](https://github.com/airbytehq/airbyte/pull/33414) | Prepare for airbyte-lib |
| 0.3.11 | 2023-10-19 | [31599](https://github.com/airbytehq/airbyte/pull/31599) | Base image migration: remove Dockerfile and use the python-connector-base image |
| 0.3.10 | 2023-09-27 | [30487](https://github.com/airbytehq/airbyte/pull/30487) | Fix bug causing rows to be skipped when batch size increased due to rate limits. |
| 0.3.9 | 2023-09-25 | [30749](https://github.com/airbytehq/airbyte/pull/30749) | Performance testing - include socat binary in docker image |
Expand Down