Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DAT: Add test case for new field(s) appearing in data #12509

Closed
Phlair opened this issue May 2, 2022 · 1 comment · Fixed by #15372
Closed

DAT: Add test case for new field(s) appearing in data #12509

Phlair opened this issue May 2, 2022 · 1 comment · Fixed by #15372
Assignees
Labels
needs-triage type/enhancement New feature or request

Comments

@Phlair
Copy link
Contributor

Phlair commented May 2, 2022

Tell us about the problem you're trying to solve

Right now, most destinations can handle new fields coming from the source that weren't previously specified. Exactly how they handle this is currently unenforced (e.g. field doesn't appear in output, field in pre-normalized JSON). Some destinations may fail on this behaviour but it is currently unclear if any would.

See this slack thread for more context.

Describe the solution you’d like

We would like to enforce that a destination at least doesn't fail if new field(s) arrive in the data at any point after initial schema discovery and sync.
We can do this by adding a test case for this to the DAT.

Describe the alternative you’ve considered or used

Currently, there is no explicit contract for this behaviour so it is somewhat assumed but this isn't a safe assumption unless we're testing every destination to confirm.

An alternative would be to enforce a stricter contract, e.g. enforce that a destination propagates new field(s) to the output data or enforce that a destination ignores new field(s). This may become the right approach in time.

@yurii-bidiuk
Copy link
Contributor

testSyncNotFailsWithNewFields() is passed for all destination except destination-aws-datalake. It fails on CI when we verify that number of written messages is the same as was intended. But on the local env this test is passed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
needs-triage type/enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants