CDK: Add schema normalization to declarative stream #32786

keu · 2023-11-24T06:00:59Z

What

The implementation follows this proposal here

@dataclass
class RecordSelector(HttpSelector):
    ...
    schema_normalization: TypeTransformer
    ...

It applies schema_normalization at the very end - after we extracted records and transformed them.

How

Describe the solution

🚨 User Impact 🚨

Are there any breaking changes? What is the end result perceived by the user?

For connector PRs, use this section to explain which type of semantic versioning bump occurs as a result of the changes. Refer to our Semantic Versioning for Connectors guidelines for more information. Breaking changes to connectors must be documented by an Airbyte engineer (PR author, or reviewer for community PRs) by using the Breaking Change Release Playbook.

If there are breaking changes, please merge this PR with the 🚨🚨 emoji so changelog authors can further highlight this if needed.

Pre-merge Actions

Expand the relevant checklist and delete the others.

New Connector

Community member or Airbyter

Community member? Grant edit access to maintainers (instructions)
Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
Connector version is set to 0.0.1
- Dockerfile has version 0.0.1
Documentation updated
- Connector's README.md
- Connector's bootstrap.md. See description and examples
- docs/integrations/<source or destination>/<name>.md including changelog with an entry for the initial version. See changelog example
- docs/integrations/README.md

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.

Updating a connector

Community member or Airbyter

Grant edit access to maintainers (instructions)
Unit & integration tests added

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.

Connector Generator

Issue acceptance criteria met
PR name follows PR naming conventions
If adding a new generator, add it to the list of scaffold modules being tested
The generator test modules (all connectors with -scaffold in their name) have been updated with the latest scaffold by running ./gradlew :airbyte-integrations:connector-templates:generator:generateScaffolds then checking in your changes
Documentation which references the generator is updated as needed

Updating the Python CDK

Airbyter

Before merging:

Pull Request description explains what problem it is solving
Code change is unit tested
Build and my-py check pass
Smoke test the change on at least one affected connector
- On Github: Run this workflow, passing --use-local-cdk --name=source-<connector> as options
- Locally: airbyte-ci connectors --use-local-cdk --name=source-<connector> test
PR is reviewed and approved

After merging:

Publish the CDK
- The CDK does not follow proper semantic versioning. Choose minor if this the change has significant user impact or is a breaking change. Choose patch otherwise.
- Write a thoughtful changelog message so we know what was updated.
Merge the platform PR that was auto-created for updating the Connector Builder's CDK version
- This step is optional if the change does not affect the connector builder or declarative connectors.

vercel · 2023-11-24T06:01:04Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
airbyte-docs	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Dec 21, 2023 1:39am

CLAassistant · 2023-11-24T06:01:05Z

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
3 out of 4 committers have signed the CLA.

✅ keu
✅ girarda
✅ yevhenii-ldv
❌ eugene-kulak
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

girarda

Confirmed this works as expected.

example record without casting ID to a string:
{"type": "RECORD", "record": {"stream": "email_templates", "data": {"id": 4071586003,

example with the normalization enabled and the schema defining id as a string:
{"type": "RECORD", "record": {"stream": "email_templates", "data": {"id": "4071585003",

girarda · 2023-11-29T23:14:00Z

airbyte-cdk/python/airbyte_cdk/sources/declarative/declarative_component_schema.yaml

+    description: Responsible for normalization according to the schema.
+    type: string
+    enum:
+      - no


let's call this NO_TRANSFORM

I assume this is the option that user will see in the UI, right? wouldn't "No Trasform" will look better in that case?

good question. @lmossman does the FE rename enums or show them as they are in the schema?

The Connector Builder currently just shows enum values as they are, e.g. for HTTP Method we show GET and POST in the UI.

But it also would be pretty low effort to rename the values when adding them to the builder UI

thanks Lake! @keu let's go with something user-friendly then.

Thinking more about the name, I sort of dislike normalization because the name clashes with another Airbyte concept.

How about

None

Default

airbyte-cdk/python/airbyte_cdk/sources/declarative/declarative_component_schema.yaml

airbyte-cdk/python/airbyte_cdk/sources/declarative/extractors/http_selector.py

…/add-schema-normalization

keu · 2023-12-20T00:13:29Z

airbyte-cdk/python/airbyte_cdk/sources/declarative/parsers/model_to_component_factory.py

@@ -707,17 +709,15 @@ def create_http_requester(self, model: HttpRequesterModel, config: Config, *, na
            parameters=model.parameters or {},
        )

-        model_http_method = (


@girarda removed this after adding --set-default-enum-member flag

github-actions · 2023-12-21T01:34:34Z

Warning

Soft code freeze is in effect until 2024-01-02. Please avoid merging to master. #freedom-and-responsibility

girarda

this is great thanks @keu !

Co-authored-by: Eugene Kulak <kulak.eugene@gmail.com> Co-authored-by: Yevhenii Kurochkin <ykurochkin@flyaps.com> Co-authored-by: Alexandre Girard <alexandre@airbyte.io>

keu requested a review from a team as a code owner November 24, 2023 06:01

keu self-assigned this Nov 24, 2023

keu marked this pull request as draft November 24, 2023 06:01

octavia-squidington-iii added the CDK Connector Development Kit label Nov 24, 2023

eugene-kulak added 2 commits November 29, 2023 15:27

add schema normalization transformation to declarative stream

eb7f91c

add mapping and use enumb for schema normalization in record selector

5da985f

keu force-pushed the keu/declarative-cdk/add-schema-normalization branch from 686fdfd to 5da985f Compare November 29, 2023 13:28

keu marked this pull request as ready for review November 29, 2023 13:28

mypy

7b49fd1

keu force-pushed the keu/declarative-cdk/add-schema-normalization branch from 05eb109 to 7b49fd1 Compare November 29, 2023 13:38

girarda requested changes Nov 29, 2023

View reviewed changes

address changes requested by CR

3c6b935

keu force-pushed the keu/declarative-cdk/add-schema-normalization branch from 50a60b5 to 3c6b935 Compare November 30, 2023 10:21

eugene-kulak added 2 commits December 1, 2023 13:23

fix tests

6839d36

Merge remote-tracking branch 'origin/master' into keu/declarative-cdk…

6569eef

…/add-schema-normalization

keu requested a review from girarda December 1, 2023 11:24

eugene-kulak and others added 2 commits December 1, 2023 13:27

rename options to be more user-friendly

be26379

Merge branch 'master' into keu/declarative-cdk/add-schema-normalization

3adf05d

vercel bot deployed to Preview December 11, 2023 17:04 View deployment

Merge remote-tracking branch 'origin/master' into keu/declarative-cdk…

ea60962

…/add-schema-normalization

vercel bot deployed to Preview December 16, 2023 12:31 View deployment

fix mypy

75c9f1f

keu linked an issue Dec 18, 2023 that may be closed by this pull request

low-code cdk: add support for DefaultSchemaNormalization #30737

Closed

yevhenii-ldv and others added 5 commits December 18, 2023 09:10

run ./gradlew generateComponentManifestClassFiles

6d9bea1

fix default value of the schema_normalization option

ad73fa5

Merge remote-tracking branch 'origin/master' into keu/declarative-cdk…

cf242e8

…/add-schema-normalization

fix mypy

8565123

remove explicit http_method casting from str

44b28bb

keu changed the base branch from master to 09-14-add_connectorBuildOptions_to_connector_metadata December 20, 2023 00:10

keu requested review from a team as code owners December 20, 2023 00:10

keu changed the base branch from 09-14-add_connectorBuildOptions_to_connector_metadata to master December 20, 2023 00:10

eugene-kulak added 2 commits December 20, 2023 02:11

manual update to fix PR diff

a4d348a

revert

4da169e

keu commented Dec 20, 2023

View reviewed changes

eugene-kulak and others added 5 commits December 20, 2023 02:14

love mypy

0f8741c

more changes because of fixing HttpMethod enum usage

0fdab29

more changes because of fixing HttpMethod enum usage

1904c19

fixing tests

f53d36c

Merge branch 'master' into keu/declarative-cdk/add-schema-normalization

f64d72f

vercel bot deployed to Preview December 21, 2023 01:39 View deployment

girarda approved these changes Dec 21, 2023

View reviewed changes

keu merged commit 4061f08 into master Dec 21, 2023
21 of 22 checks passed

keu deleted the keu/declarative-cdk/add-schema-normalization branch December 21, 2023 12:51

midavadim mentioned this pull request Jan 8, 2024

🐛 CDK HttpRequester should support str http_method which comes from manifest.yaml file #34011

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CDK: Add schema normalization to declarative stream #32786

CDK: Add schema normalization to declarative stream #32786

keu commented Nov 24, 2023 •

edited

Loading

vercel bot commented Nov 24, 2023 •

edited

Loading

CLAassistant commented Nov 24, 2023 •

edited

Loading

girarda left a comment

girarda Nov 29, 2023

keu Nov 30, 2023

girarda Nov 30, 2023

lmossman Nov 30, 2023

girarda Nov 30, 2023

keu Dec 20, 2023

github-actions bot commented Dec 21, 2023

girarda left a comment

CDK: Add schema normalization to declarative stream #32786

CDK: Add schema normalization to declarative stream #32786

Conversation

keu commented Nov 24, 2023 • edited Loading

What

How

Recommended reading order

🚨 User Impact 🚨

Pre-merge Actions

Community member or Airbyter

Airbyter

Community member or Airbyter

Airbyter

Airbyter

vercel bot commented Nov 24, 2023 • edited Loading

CLAassistant commented Nov 24, 2023 • edited Loading

girarda left a comment

Choose a reason for hiding this comment

girarda Nov 29, 2023

Choose a reason for hiding this comment

keu Nov 30, 2023

Choose a reason for hiding this comment

girarda Nov 30, 2023

Choose a reason for hiding this comment

lmossman Nov 30, 2023

Choose a reason for hiding this comment

girarda Nov 30, 2023

Choose a reason for hiding this comment

keu Dec 20, 2023

Choose a reason for hiding this comment

github-actions bot commented Dec 21, 2023

girarda left a comment

Choose a reason for hiding this comment

keu commented Nov 24, 2023 •

edited

Loading

vercel bot commented Nov 24, 2023 •

edited

Loading

CLAassistant commented Nov 24, 2023 •

edited

Loading