New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
✨Source Apify Dataset: Migrate Python CDK to Low-code CDK #29859
Conversation
Before Merging a Connector Pull RequestWow! What a great pull request you have here! 🎉 To merge this PR, ensure the following has been done/considered for each connector added or updated:
If the checklist is complete, but the CI check is failing,
|
Checks: |
…irbyte into apifydatasetlowcode
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Left some comments.
start_date: | ||
type: string | ||
title: Start Date | ||
description: | ||
UTC date and time in the format 2017-01-25T00:00:00Z. Any data | ||
before this date will not be replicated. | ||
examples: | ||
- "2023-08-05T00:43:59.244Z" | ||
default: "2023-08-25T00:43:59.244Z" | ||
airbyte_secret: false |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You're adding a new parameter and removing another one. For the migration we must keep the same.
| 0.1.0 | 2021-07-29 | [PR\#5069](https://github.com/airbytehq/airbyte/pull/5069) | Initial version of the connector | | ||
| Version | Date | Pull Request | Subject | | ||
| :-------- | :---------- | :------------------------------------------------------------ | :-------------------------------------------------------------------------- | | ||
| 0.3.0 | 2023-08-25 | [TBD](https://github.com/airbytehq/airbyte/pull/TBD) | Migrate to lowcode | |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Update this.
incremental_base: | ||
type: DatetimeBasedCursor | ||
cursor_field: "{{ parameters.incremental_cursor }}" | ||
datetime_format: "%Y-%m-%dT%H:%M:%S.%f%z" | ||
cursor_granularity: "PT0.000001S" | ||
start_datetime: | ||
datetime: "{{ config['start_date'] }}" | ||
datetime_format: "%Y-%m-%dT%H:%M:%S.%f%z" | ||
end_datetime: | ||
datetime: "{{ today_utc() }}" | ||
datetime_format: "%Y-%m-%d" | ||
step: "P1M" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can you reference Apify documentation where you can use incremental?
requester: | ||
$ref: "#/definitions/requester" | ||
|
||
base_paginator: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not used anywhere. Please make sure to use the paginator.
6215cd2
to
68265a8
Compare
…pifydatasetlowcode
68265a8
to
b31b9e5
Compare
Reolved comments, somehow my apifydataset account got disabled. Thus couldn't test the changes :( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks @btkcodedev
…29859) Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com> Co-authored-by: marcosmarxm <marcosmarxm@gmail.com>
What
Migrating Source Apify Dataset to Low-Code CDK
Closes #29116
How
Developed using (Configuration Based Source) low-code CDK
Recommended reading order
spec.yaml
manifest.yaml
schemas/*
🚨 User Impact 🚨
Migrate Connector: Source Apify Dataset
Tests
Community member or Airbyter
airbyte_secret
./gradlew :airbyte-integrations:connectors:<name>:integrationTest
.README.md
bootstrap.md
docs/integrations/<source or destination>/<name>.md
including changelog. See changelog exampledocs/integrations/README.md
airbyte-integrations/builds.md