-
Notifications
You must be signed in to change notification settings - Fork 5
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add support for nullable fields #6
Comments
peel
pushed a commit
that referenced
this issue
Nov 3, 2020
Previously all the fields with nullable values were discarded as invalid. However there is a wide range of fields that cannot be treated as non-nullable. For example for enrichment failures bad row, we get following structure (trimmed): ``` column_name | is_nullable | data_type -------------------------------------------+-------------+----------------------------- processor.artifact | NO | character varying processor.version | NO | character varying failure.messages | YES | jsonb failure.timestamp | YES | timestamp without time zone payload.enriched.app_id | YES | character varying payload.enriched.base_currency | YES | character varying ... ``` Where only processor is non-nullable. Therefore enrichment failures would not be written with evaluations like: ``` payload.enriched.br_cookies -> Some(null) -> BigInt -> Left(Invalid type BIGINT for value Some(null)) ``` This PR allows marking fields nullable by default. We might want to consider a smarter approach at some point.
peel
pushed a commit
that referenced
this issue
Nov 10, 2020
Previously all the fields with nullable values were discarded as invalid. However there is a wide range of fields that cannot be treated as non-nullable. For example for enrichment failures bad row, we get following structure (trimmed): ``` column_name | is_nullable | data_type -------------------------------------------+-------------+----------------------------- processor.artifact | NO | character varying processor.version | NO | character varying failure.messages | YES | jsonb failure.timestamp | YES | timestamp without time zone payload.enriched.app_id | YES | character varying payload.enriched.base_currency | YES | character varying ... ``` Where only processor is non-nullable. Therefore enrichment failures would not be written with evaluations like: ``` payload.enriched.br_cookies -> Some(null) -> BigInt -> Left(Invalid type BIGINT for value Some(null)) ``` This PR allows marking fields nullable by default. We might want to consider a smarter approach at some point.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Previously all the fields with nullable values were discarded as invalid.
However there is a wide range of fields that cannot be treated as non-nullable.
For example for enrichment failures bad row, we get following structure (trimmed):
Where only processor is non-nullable. Therefore enrichment failures would not be written with evaluations like:
The text was updated successfully, but these errors were encountered: