You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Right now, most destinations can handle new fields coming from the source that weren't previously specified. Exactly how they handle this is currently unenforced (e.g. field doesn't appear in output, field in pre-normalized JSON). Some destinations may fail on this behaviour but it is currently unclear if any would.
We would like to enforce that a destination at least doesn't fail if new field(s) arrive in the data at any point after initial schema discovery and sync.
We can do this by adding a test case for this to the DAT.
Describe the alternative you’ve considered or used
Currently, there is no explicit contract for this behaviour so it is somewhat assumed but this isn't a safe assumption unless we're testing every destination to confirm.
An alternative would be to enforce a stricter contract, e.g. enforce that a destination propagates new field(s) to the output data or enforce that a destination ignores new field(s). This may become the right approach in time.
The text was updated successfully, but these errors were encountered:
testSyncNotFailsWithNewFields() is passed for all destination except destination-aws-datalake. It fails on CI when we verify that number of written messages is the same as was intended. But on the local env this test is passed
Tell us about the problem you're trying to solve
Right now, most destinations can handle new fields coming from the source that weren't previously specified. Exactly how they handle this is currently unenforced (e.g. field doesn't appear in output, field in pre-normalized JSON). Some destinations may fail on this behaviour but it is currently unclear if any would.
See this slack thread for more context.
Describe the solution you’d like
We would like to enforce that a destination at least doesn't fail if new field(s) arrive in the data at any point after initial schema discovery and sync.
We can do this by adding a test case for this to the DAT.
Describe the alternative you’ve considered or used
Currently, there is no explicit contract for this behaviour so it is somewhat assumed but this isn't a safe assumption unless we're testing every destination to confirm.
An alternative would be to enforce a stricter contract, e.g. enforce that a destination propagates new field(s) to the output data or enforce that a destination ignores new field(s). This may become the right approach in time.
The text was updated successfully, but these errors were encountered: