destination-snowflake: truncate large records

[Stephane Geneix (stephane-airbyte)]

What

This change implements a large record truncation mechanism for the Snowflake destination connector to handle records exceeding Snowflake's 16MB row size limit.

How

• The truncator preserves primary key fields and truncates other fields to fit within the 16MB limit.
• Added metadata to indicate which fields were truncated due to size limitations.

User Impact

Users can now sync large records to Snowflake without encountering errors due to row size limitations. Fields may be truncated to fit within the 16MB limit, but primary keys are always preserved. Metadata is added to indicate which fields were affected.

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
airbyte-docs	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Sep 18, 2024 10:50pm

[Stephane Geneix (stephane-airbyte)]

• destination-snowflake: update all dependencies #45437
• destination-snowflake: truncate large records #45431 👈
• destination-snowflake: speed up metadata queries #45422
• destination-snowflake: fix destination state table collation #38585
• destination-snowflake: get tests to pass - durably #45370 : 1 other dependent PR (#45448 )
• change signature of orphanedThreadFilter #45469 : 1 other dependent PR (#45432 )
• master

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join Stephane Geneix (@stephane-airbyte) and the rest of your teammates on Graphite

[Johnny Schmidt (johnny-schmidt)]

Makes sense
1 remove the primary key fields, totaling their size; add them to final data
2 sort the remaining by size ascending
3 keep adding them to the final data until we hit the limit, then start nulling and tracking changes

Questions:

1. where does this limit come from 16 * 1_024 * 1_024?; does SF enforce that the string size of all the fields in a row be less than this? or is this to enforce serialization size in the raw table?
2. is this a hard limit? if so, what happens if the PK fields are too big (obviously unlikely); also, if this is for serialization size in the raw table, won't that size be larger due to the format? do the extra meta fields come into play? could the meta itself blow up if there were lots of fields past the limit
3. this leaves the fields sorted by size, I assume that's okay? nothing downstream is dependent on field order (including tests)?

[Edward Gao (edgao)]

(declining this review, unless Stephane Geneix (@stephane-airbyte) you want a third set of eyes here)

[Stephane Geneix (stephane-airbyte)]

Makes sense 1 remove the primary key fields, totaling their size; add them to final data 2 sort the remaining by size ascending 3 keep adding them to the final data until we hit the limit, then start nulling and tracking changes

Questions:

1. where does this limit come from 16 * 1_024 * 1_024?; does SF enforce that the string size of all the fields in a row be less than this? or is this to enforce serialization size in the raw table?

There's a weird 16MB limit that's on the VARIANT type. It's weird because it's not exact. They say it's compressed size (so raw size could be bigger), but could be smaller because of overhead...

2. is this a hard limit? if so, what happens if the PK fields are too big (obviously unlikely); also, if this is for serialization size in the raw table, won't that size be larger due to the format? do the extra meta fields come into play? could the meta itself blow up if there were lots of fields past the limit

If the sum of PK fields is bigger than 16MB, we just try to insert the record anyways, and we'll fail then. Seemed like dying was the only right thing to do short of losing data

3. this leaves the fields sorted by size, I assume that's okay? nothing downstream is dependent on field order (including tests)?

yeah, tests look at the json field names, they don't care about order. Neither do our T+D queries