Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Do not propagate all parameters to child nodes when using InlineSchemaLoader in low-code CDK #34755

Closed
ambirdsall opened this issue Feb 1, 2024 · 1 comment

Comments

@ambirdsall
Copy link
Contributor

Topic

No response

Relevant information

In this line, manifest_component_transformer.py propagates parameters in a way that is not valid for inline schemas; while this is not problem for manifests that are written to have inline schemas from the start (and thus, hopefully, don't use parameters to share schema data across multiple files), it is not a behavior that makes sense in this context and it complicates migrating an existing manifest to use inline schemas, which is a requirement for compatibility with the Connector Builder. Airbyte employees can reference this internal slack conversation for additional context.

@maxi297
Copy link
Contributor

maxi297 commented Feb 5, 2024

To add on the specific of source-activecampaign case, the issue is that there is a property named type which leads the propagation to continue as we don't hit this condition. The schema then ends up being modified like this (simplified version) which is not valid because properties.primary_key is expected to be an object

{
    "type": "object",
    "properties": {
        "type": {
            "type": "string",
            "name": "campaigns",
            "primary_key": "id",
            "$parameters": {
                "name": "campaigns",
                "primary_key": "id",
            }
        },
        "primary_key": "id",
        "$parameters": {
            "primary_key": "id",
            "name": "campaigns"
        }
    },
    "required": [
        "type"
    ]
}

@maxi297 maxi297 closed this as completed Feb 7, 2024
jatinyadav-cc pushed a commit to ollionorg/datapipes-airbyte that referenced this issue Feb 21, 2024
jatinyadav-cc pushed a commit to ollionorg/datapipes-airbyte that referenced this issue Feb 26, 2024
jatinyadav-cc pushed a commit to ollionorg/datapipes-airbyte that referenced this issue Feb 26, 2024
jatinyadav-cc pushed a commit to ollionorg/datapipes-airbyte that referenced this issue Feb 26, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants