Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Blank double quoted fields in CSVs treated as varchar instead of null in Snowflake External Tables #132

Closed
1 of 3 tasks
nic-sharesies opened this issue Jan 31, 2022 · 1 comment
Labels
bug Something isn't working

Comments

@nic-sharesies
Copy link

nic-sharesies commented Jan 31, 2022

Describe the bug

After we upgraded to dbt v1.0.0, we started having errors selecting from Snowflake External Tables.
This was due to the CSV files in S3 containing blank double quoted values, e.g. "".
Snowflake was storing these as text as opposed to null, leading to cast type errors.

Steps to reproduce

  1. Have a CSV file in S3 that contains blank double quoted fields using a non char data type, i.e. timestamp
  2. Create a Snowflake External Table using that CSV as the source
  3. Select from that table in Snowflake

Expected results

Blank fields cast to data type as expected and Snowflake query returns result set.

Actual results

Snowflake fails to cast to data type and throws an error.

Screenshots and log output

System information

The contents of your packages.yml file:

packages:
  - package: dbt-labs/dbt_external_tables
    version: 0.8.0

  - package: calogica/dbt_expectations
    version: 0.5.1

Which database are you using dbt with?

  • redshift
  • snowflake
  • other (specify: ____________)

The output of dbt --version:

installed version: 1.0.0
   latest version: 1.0.0

Up to date!

Plugins:
  - snowflake: 1.0.0
  - postgres: 1.0.0

The operating system you're using:
MacOS Monterey 12.1
The output of python --version:
3.9.9

Additional context

PR created with fix
#131

@jtcohen6
Copy link
Collaborator

Thanks @nic-sharesies, for the detailed issue and the PR with the fix! Thanks also to @jeremyyeo for bumping this one to my attention, so that we could make the connection to #118.

Rather than an increasingly complex case when statement defined within the macro, I'm going to close this one, in favor of consolidating with the change requested + proposed in #118.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants