feat: support larger seed files

[henriblancke]

Description

This change uploads seed files to s3 before creating the seed table. This makes larger seeds possible and removes the limitation of the athena query char limit.

It uploads the seeds as json to have better type casting support. OpenCSVSerde is not good at casting timestamps and inferring correct data types. Since seeds are mostly smaller files this should be fine. Writing them as parquet adds too much complexity to this adapter.

Checklist

• You followed contributing section
• You added unit testing when necessary
• You added functional testing when necessary

[nicor88]

Does seeds still support passing the seed file as csv?

[henriblancke]

Does seeds still support passing the seed file as csv?

Yep, files are still being passed ascsv, agate will convert to json before uploading the file to s3. So you still add your seed files to dbt as csv files from the end user's perspective.

[nicor88]

Looks good, I left some small nits.

[nicor88]

💯

[henriblancke]

@nicor88 @Jrmyy anything I can do here as a follow up?

[nicor88]

@henriblancke please rebase with main - then I will test and review this feature.

[henriblancke]

@nicor88 rebased ✅

[henriblancke]

@nicor88 fixed merge conflicts and rebased again, let me know if there is anything else I can do to help

[nicor88]

@henriblancke nice job here, please address this #161 (comment) and then we can merge, if also @Jrmyy gives the ok.
I tried the feature - with a csv that is 1MB and works fine, data types included!

[Jrmyy]

I will make a test this morning and let you know !

[Jrmyy]

The test I made works therefore once my comment is resolved, this is a go for merging 👍🏻
Thanks a lot for the contribution 🔥

[nicor88]

@henriblancke please resolve this one #161 (comment) and we merge this and include in the next release.

[henriblancke]

@henriblancke please resolve this one #161 (comment) and we merge this and include in the next release.

@Jrmyy @nicor88 thanks again for the review, I've addressed #161 (comment)