Add primary key tests to TestDestination #2776
Conversation
...ion-test/src/main/java/io/airbyte/integrations/standardtest/destination/TestDestination.java
Outdated
Show resolved
Hide resolved
| s.withSyncMode(SyncMode.INCREMENTAL); | ||
| s.withDestinationSyncMode(DestinationSyncMode.APPEND_DEDUP); | ||
| s.withCursorField(List.of("_airbyte_emitted_at")); | ||
| s.withPrimaryKey(List.of(List.of("date"))); |
There was a problem hiding this comment.
this is a good test. would it make sense to test a compound primary key as well? exchange rates is actually a good candidate for this where you can imagine the primary key being [currency, date].
There was a problem hiding this comment.
Ok, but Exchange rates connector doesn't actually include a "base" currency column (as selected in the connector's spec.json), I'll add one in the catalog/message json files then.
However moving on this idea, I added a currency value column (actual conversion rate float number) as part of the primary key. It may not make much business sense to do that but it is a failing use case in #2684
| @@ -502,7 +497,11 @@ def generate_scd_type_2_model(self, from_table: str, column_names: Dict[str, Tup | |||
|
|
|||
| def get_cursor_field(self, column_names: Dict[str, Tuple[str, str]]) -> str: | |||
| if self.cursor_field and len(self.cursor_field) == 1: | |||
| return column_names[self.cursor_field[0]][0] | |||
| if not is_airbyte_column(self.cursor_field[0]): | |||
There was a problem hiding this comment.
column_names is a map of all original columns declared in the properties/JSON schema for the current table.
But, here we want to allow generated airbyte columns as cursor fields or primary key too (those may not be listed in the original JSON schema) so I add that special case with this if statement
Co-authored-by: Charles <giardina.charles@gmail.com>
|
/test connector=connectors/destination-bigquery
|
|
/test connector=connectors/destination-csv
|
|
/test connector=connectors/destination-local-json
|
|
/test connector=connectors/destination-meilisearch
|
|
/test connector=connectors/destination-postgres
|
|
/test connector=connectors/destination-redshift
|
|
/test connector=connectors/destination-snowflake
|
ChristopheDuong
changed the title
karinakuz
added
connectors/destination/bigquery
connectors/destinations-api
connectors/destinations-files
connectors/destination/local-csv
connectors/destinations-warehouse
connectors/destination/local-json
connectors/destination/meilisearch
connectors/destination/snowflake
and removed
connectors/destinations-api
labels
What
Closes #2749
Relates to #2683
Closes #2684
How
Describe the solution
For the test case, I also used the scenario where we would want to deduplicate using as the cursor the column
_airbyte_emitted_atcolumn as described in #2683This is because exchange rate API could have the "date" column as the primary key but we can have multiple rows for the same day and there isn't another date column in the data. By doing so, the cursor field is now optional for normalization.
This is also handling a column with type float as described in #2684 as the primary key.
(normalization should now be able to handle the empty/airbyte generated column use case as cursor and is a matter of handling that option in the UI too)
Pre-merge Checklist
Recommended reading order
airbyte-integrations/bases/standard-destination-test/src/main/java/io/airbyte/integrations/standardtest/destination/TestDestination.javaairbyte-integrations/bases/base-normalization/normalization/transform_catalog/stream_processor.py