Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

🎉 New Source: Apify Dataset #5069

Closed
wants to merge 18 commits into from
Closed

🎉 New Source: Apify Dataset #5069

wants to merge 18 commits into from

Conversation

mhamas
Copy link
Contributor

@mhamas mhamas commented Jul 29, 2021 •
edited by Phlair
Loading

What

Syncing the data from Apify dataset to Airbyte.

How

Adding a new connector written in Python which uses Apify Python client under the hood.

Recommended reading order

No particular order. Most of the code was auto-generating using the provided Airbyte generate.sh/

Pre-merge Checklist

Expand the checklist which is relevant for this PR.

Connector checklist

  • Issue acceptance criteria met
  • PR name follows PR naming conventions
  • Secrets are annotated with airbyte_secret in the connector's spec
  • Credentials added to Github CI if needed and not already present. instructions for injecting secrets into CI.
  • Unit & integration tests added as appropriate (and are passing)
    • Community members: please provide proof of this succeeding locally e.g: screenshot or copy-paste acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
  • /test connector=connectors/<name> command as documented here is passing.
    • Community members can skip this, Airbyters will run this for you.
  • Code reviews completed
  • Documentation updated
    • README.md
    • docs/SUMMARY.md if it's a new connector
    • Created or updated reference docs in docs/integrations/<source or destination>/<name>.
    • Changelog in the appropriate page in docs/integrations/.... See changelog example
    • docs/integrations/README.md contains a reference to the new connector
    • Build status added to build page
  • Build is successful
  • Connector version bumped like described here
  • New Connector version released on Dockerhub by running the /publish command described here
  • No major blockers
  • PR merged into master branch
  • Follow up tickets have been created
  • Associated tickets have been closed & stakeholders notified

@github-actions github-actions bot added area/connectors Connector related issues area/documentation Improvements or additions to documentation labels Jul 29, 2021
@mhamas mhamas marked this pull request as ready for review July 29, 2021 09:08
@Phlair Phlair self-requested a review July 29, 2021 09:10
@Phlair Phlair changed the title Initial version of Apify Dataset source connector 🎉 New Source: Apify Dataset Jul 29, 2021
Phlair
Phlair reviewed Jul 29, 2021
View reviewed changes
Copy link
Contributor

@Phlair Phlair left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is looking great, thanks for the contribution Matej!
I especially like the multi-threading across batches 💪

Just a couple of minor tweaks required , see my comments.

airbyte-config/init/src/main/resources/seed/source_definitions.yaml Show resolved Hide resolved
airbyte-integrations/connectors/source-apify-dataset/build.gradle Outdated Show resolved Hide resolved
airbyte-integrations/connectors/source-apify-dataset/integration_tests/configured_catalog.json Outdated Show resolved Hide resolved
airbyte-integrations/connectors/source-apify-dataset/setup.py Outdated Show resolved Hide resolved
airbyte-integrations/connectors/source-apify-dataset/source_apify_dataset/source.py Outdated Show resolved Hide resolved
airbyte-integrations/connectors/source-apify-dataset/source_apify_dataset/spec.json Outdated Show resolved Hide resolved
airbyte-integrations/connectors/source-apify-dataset/source_apify_dataset/source.py Show resolved Hide resolved
airbyte-integrations/connectors/source-apify-dataset/source_apify_dataset/source.py Show resolved Hide resolved
@mhamas mhamas force-pushed the feature/apify-dataset-source branch from 307c72f to 445b9cb Compare July 29, 2021 13:44
@mhamas
Copy link
Contributor Author

mhamas commented Jul 29, 2021

@Phlair I addressed the comments, can you please take another look? Thanks

@mhamas mhamas requested a review from Phlair July 29, 2021 13:49
Phlair
Phlair previously approved these changes Jul 29, 2021
View reviewed changes
Copy link
Contributor

@Phlair Phlair left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Amazing, thank you Matej 🚀

I'll now be following this process #3118 (issue exists because we'll be making the process less painful soon!) to get this merged and published!

...c/main/resources/config/STANDARD_SOURCE_DEFINITION/47f17145-fe20-4ef5-a548-e29b048adf84.json Show resolved Hide resolved
@Phlair Phlair dismissed their stale review July 29, 2021 14:42

need to follow procedure #3118

@Phlair Phlair mentioned this pull request Jul 29, 2021
Merged
21 tasks
Phlair
Phlair reviewed Aug 2, 2021
View reviewed changes
Copy link
Contributor

@Phlair Phlair left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

New changes make sense for retaining order. Very minor couple of comments :)

airbyte-integrations/connectors/source-apify-dataset/source_apify_dataset/source.py Outdated Show resolved Hide resolved
...c/main/resources/config/STANDARD_SOURCE_DEFINITION/47f17145-fe20-4ef5-a548-e29b048adf84.json Outdated Show resolved Hide resolved
@mhamas mhamas requested a review from Phlair August 2, 2021 15:50
Phlair
Phlair reviewed Aug 2, 2021
View reviewed changes
Copy link
Contributor

@Phlair Phlair left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm! I'll pull latest into here: #5081

@Phlair
Copy link
Contributor

Phlair commented Aug 3, 2021

closed by #5081

@Phlair Phlair closed this Aug 3, 2021
@sherifnada sherifnada mentioned this pull request Aug 13, 2021
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Phlair Phlair Phlair left review comments

Assignees

@mhamas mhamas

@Phlair Phlair

Labels
area/connectors Connector related issues area/documentation Improvements or additions to documentation connectors/source/apify-dataset connectors/sources-api
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@mhamas @Phlair @sherifnada @igrankova