✨ low-code CDK: Allow connector developers to specify the type of an added field #31638

girarda · 2023-10-20T03:00:40Z

What

Allow connector developers to specify the type of an added field

Example usage:

      - type: AddFields
        fields:
          - path:
              - transaction_id
            value: "{{ record['transaction_info']['transaction_id'] }}"
            value_type: str

This feature is necessary to fix an regression introduced in source-paypal-transaction when it was migrated to low-code.
The values are implicitly casted to numbers, which is wrong because they should be treated as strings.

This should be followed up by

A PR fixing the paypal connector
An issue for exposing this field in the connector builder

How

Add value_type field to AddedFieldDefinition object
The field can be any value defined in the ValueType enum
The component factory converts the enum value to a type before propagating it to the AddFields component
A few unrelated mypy issues were fixed along the way

🚨 User Impact 🚨

None. This is an additive feature.

Pre-merge Actions

Expand the relevant checklist and delete the others.

New Connector

Community member or Airbyter

Community member? Grant edit access to maintainers (instructions)
Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
Connector version is set to 0.0.1
- Dockerfile has version 0.0.1
Documentation updated
- Connector's README.md
- Connector's bootstrap.md. See description and examples
- docs/integrations/<source or destination>/<name>.md including changelog with an entry for the initial version. See changelog example
- docs/integrations/README.md

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.

Updating a connector

Community member or Airbyter

Grant edit access to maintainers (instructions)
Unit & integration tests added

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.

Connector Generator

Issue acceptance criteria met
PR name follows PR naming conventions
If adding a new generator, add it to the list of scaffold modules being tested
The generator test modules (all connectors with -scaffold in their name) have been updated with the latest scaffold by running ./gradlew :airbyte-integrations:connector-templates:generator:generateScaffolds then checking in your changes
Documentation which references the generator is updated as needed

Updating the Python CDK

Airbyter

Before merging:

Pull Request description explains what problem it is solving
Code change is unit tested
Build and my-py check pass
Smoke test the change on at least one affected connector
- On Github: Run this workflow, passing --use-local-cdk --name=source-<connector> as options
- Locally: airbyte-ci connectors --use-local-cdk --name=source-<connector> test
PR is reviewed and approved

After merging:

Publish the CDK
- The CDK does not follow proper semantic versioning. Choose minor if this the change has significant user impact or is a breaking change. Choose patch otherwise.
- Write a thoughtful changelog message so we know what was updated.
Merge the platform PR that was auto-created for updating the Connector Builder's CDK version
- This step is optional if the change does not affect the connector builder or declarative connectors.

vercel · 2023-10-20T03:00:46Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
airbyte-docs	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Oct 23, 2023 8:17pm

github-actions · 2023-10-20T03:01:01Z

Before Merging a Connector Pull Request

Wow! What a great pull request you have here! 🎉

To merge this PR, ensure the following has been done/considered for each connector added or updated:

PR name follows PR naming conventions
Breaking changes are considered. If a Breaking Change is being introduced, ensure an Airbyte engineer has created a Breaking Change Plan.
Connector version has been incremented in the Dockerfile and metadata.yaml according to our Semantic Versioning for Connectors guidelines
You've updated the connector's metadata.yaml file any other relevant changes, including a breakingChanges entry for major version bumps. See metadata.yaml docs
Secrets in the connector's spec are annotated with airbyte_secret
All documentation files are up to date. (README.md, bootstrap.md, docs.md, etc...)
Changelog updated in docs/integrations/<source or destination>/<name>.md with an entry for the new version. See changelog example
Migration guide updated in docs/integrations/<source or destination>/<name>-migrations.md with an entry for the new version, if the version is a breaking change. See migration guide example
If set, you've ensured the icon is present in the platform-internal repo. (Docs)

If the checklist is complete, but the CI check is failing,

Check for hidden checklists in your PR description
Toggle the github label checklist-action-run on/off to re-run the checklist CI.

…into alex/typed_add_fields

airbyte-oss-build-runner · 2023-10-20T04:51:55Z

source-paypal-transaction test report (commit `323d826091`) - ✅

⏲️ Total pipeline duration: 01mn37s

Step	Result
Build source-paypal-transaction docker image for platform(s) linux/x86_64	✅
Acceptance tests	✅
Code format checks	✅
Validate metadata for source-paypal-transaction	✅
Connector version semver check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-paypal-transaction test

clnoll

LGTM @girarda! Just one small question.

clnoll · 2023-10-23T13:27:13Z

...dk/python/airbyte_cdk/sources/declarative/requesters/paginators/strategies/stop_condition.py

        self._delegate.reset()

    def get_page_size(self) -> Optional[int]:
        return self._delegate.get_page_size()
+
+    @property
+    def initial_token(self) -> Optional[Any]:


Why is this change needed?

the method is an abstractmethod in the parent PaginationStrategy class so it must be implemented by StopConditionPaginationStrategyDecorator

Okay, this is just something that was overlooked previously?

that's right. mypy should've caught it when the method was introduced by it had been accidentally removed from the build pipeline for a few weeks. It caught the issue this so we should be good moving forward

airbyte-oss-build-runner · 2023-10-23T15:13:01Z

source-paypal-transaction test report (commit `12b94d6725`) - ❌

⏲️ Total pipeline duration: 6.49s

Step	Result
Build source-paypal-transaction docker image for platform(s) linux/amd64	❌
Code format checks	✅
Validate metadata for source-paypal-transaction	✅
Connector version semver check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-paypal-transaction test

This reverts commit 12b94d6.

Update

b32cb89

octavia-squidington-iii added area/connectors Connector related issues CDK Connector Development Kit connectors/source/paypal-transaction labels Oct 20, 2023

girarda and others added 12 commits October 20, 2023 03:15

Automated Commit - Formatting Changes

176d2d1

generate components

bf50375

update tests

8aa3755

update

24a4bd1

merge

03644f8

set param in test

0ab9cd5

pass the type all the way down

a733bc5

update

334e5dd

more tests

d7fc104

use an enum

9b37f31

reset

13ffd59

Merge branch 'master' into alex/typed_add_fields

f6881a6

girarda changed the title ~~Allow connector developers to specify the type of an added field~~ low-code: Allow connector developers to specify the type of an added field Oct 20, 2023

octavia-squidington-iii removed the area/connectors Connector related issues label Oct 20, 2023

girarda added area/connectors Connector related issues and removed connectors/source/paypal-transaction labels Oct 20, 2023

girarda added 2 commits October 19, 2023 21:21

restrict valid types to primitives

ed12104

Merge branch 'alex/typed_add_fields' of github.com:airbytehq/airbyte …

8a1a23a

…into alex/typed_add_fields

octavia-squidington-iii removed the area/connectors Connector related issues label Oct 20, 2023

mypy

03bf141

girarda marked this pull request as ready for review October 20, 2023 04:44

girarda requested a review from a team as a code owner October 20, 2023 04:44

girarda requested review from clnoll and brianjlai October 20, 2023 04:44

Automated Commit - Formatting Changes

323d826

octavia-squidington-iii added area/connectors Connector related issues connectors/source/paypal-transaction labels Oct 20, 2023

octavia-squidington-iv requested a review from a team October 20, 2023 04:49

girarda removed the connectors/source/paypal-transaction label Oct 20, 2023

octavia-squidington-iii added the connectors/source/paypal-transaction label Oct 20, 2023

clnoll approved these changes Oct 23, 2023

View reviewed changes

bazarnov approved these changes Oct 23, 2023

View reviewed changes

girarda changed the title ~~low-code: Allow connector developers to specify the type of an added field~~ ✨ low-code CDK: Allow connector developers to specify the type of an added field Oct 23, 2023

pass cdk into publish command

12b94d6

erohmensing requested a review from a team October 23, 2023 15:05

girarda added 2 commits October 23, 2023 13:15

Revert "pass cdk into publish command"

0c7d9d1

This reverts commit 12b94d6.

Merge branch 'master' into alex/typed_add_fields

af8b535

octavia-squidington-iii removed the area/connectors Connector related issues label Oct 23, 2023

vercel bot deployed to Preview October 23, 2023 20:17 View deployment

girarda merged commit 7a764f8 into master Oct 23, 2023
22 checks passed

girarda deleted the alex/typed_add_fields branch October 23, 2023 21:13

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

✨ low-code CDK: Allow connector developers to specify the type of an added field #31638

✨ low-code CDK: Allow connector developers to specify the type of an added field #31638

girarda commented Oct 20, 2023 •

edited

vercel bot commented Oct 20, 2023 •

edited

github-actions bot commented Oct 20, 2023 •

edited by girarda

airbyte-oss-build-runner commented Oct 20, 2023

clnoll left a comment

clnoll Oct 23, 2023

girarda Oct 23, 2023

clnoll Oct 23, 2023

girarda Oct 23, 2023

airbyte-oss-build-runner commented Oct 23, 2023

✨ low-code CDK: Allow connector developers to specify the type of an added field #31638

✨ low-code CDK: Allow connector developers to specify the type of an added field #31638

Conversation

girarda commented Oct 20, 2023 • edited

What

How

Recommended reading order

🚨 User Impact 🚨

Pre-merge Actions

Community member or Airbyter

Airbyter

Community member or Airbyter

Airbyter

Airbyter

vercel bot commented Oct 20, 2023 • edited

github-actions bot commented Oct 20, 2023 • edited by girarda

Before Merging a Connector Pull Request

airbyte-oss-build-runner commented Oct 20, 2023

source-paypal-transaction test report (commit 323d826091) - ✅

clnoll left a comment

Choose a reason for hiding this comment

clnoll Oct 23, 2023

Choose a reason for hiding this comment

girarda Oct 23, 2023

Choose a reason for hiding this comment

clnoll Oct 23, 2023

Choose a reason for hiding this comment

girarda Oct 23, 2023

Choose a reason for hiding this comment

airbyte-oss-build-runner commented Oct 23, 2023

source-paypal-transaction test report (commit 12b94d6725) - ❌

girarda commented Oct 20, 2023 •

edited

vercel bot commented Oct 20, 2023 •

edited

github-actions bot commented Oct 20, 2023 •

edited by girarda

source-paypal-transaction test report (commit `323d826091`) - ✅

source-paypal-transaction test report (commit `12b94d6725`) - ❌