Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for nullable fields #6

Closed
peel opened this issue Nov 3, 2020 · 0 comments · Fixed by #7
Closed

Add support for nullable fields #6

peel opened this issue Nov 3, 2020 · 0 comments · Fixed by #7
Labels
bug Something isn't working
Milestone

Comments

@peel
Copy link

peel commented Nov 3, 2020

Previously all the fields with nullable values were discarded as invalid.
However there is a wide range of fields that cannot be treated as non-nullable.

For example for enrichment failures bad row, we get following structure (trimmed):

                column_name                | is_nullable |          data_type
-------------------------------------------+-------------+-----------------------------
 processor.artifact                        | NO          | character varying
 processor.version                         | NO          | character varying
 failure.messages                          | YES         | jsonb
 failure.timestamp                         | YES         | timestamp without time zone
 payload.enriched.app_id                   | YES         | character varying
 payload.enriched.base_currency            | YES         | character varying
...

Where only processor is non-nullable. Therefore enrichment failures would not be written with evaluations like:

payload.enriched.br_cookies -> Some(null) ->  BigInt -> Left(Invalid type BIGINT for value Some(null))
peel pushed a commit that referenced this issue Nov 3, 2020
Previously all the fields with nullable values were discarded as invalid.
However there is a wide range of fields that cannot be treated as non-nullable.

For example for enrichment failures bad row, we get following
structure (trimmed):
```
                column_name                | is_nullable |          data_type
-------------------------------------------+-------------+-----------------------------
 processor.artifact                        | NO          | character varying
 processor.version                         | NO          | character varying
 failure.messages                          | YES         | jsonb
 failure.timestamp                         | YES         | timestamp without time zone
 payload.enriched.app_id                   | YES         | character varying
 payload.enriched.base_currency            | YES         | character varying
...
```
Where only processor is non-nullable. Therefore enrichment failures would not be
written with evaluations like:
```
payload.enriched.br_cookies -> Some(null) ->  BigInt -> Left(Invalid type
BIGINT for value Some(null))
```
This PR allows marking fields nullable by default. We might want to consider a
smarter approach at some point.
@peel peel linked a pull request Nov 3, 2020 that will close this issue
1 task
@chuwy chuwy added the bug Something isn't working label Nov 3, 2020
@peel peel added this to the 0.1.1 milestone Nov 10, 2020
@peel peel closed this as completed in #7 Nov 10, 2020
peel pushed a commit that referenced this issue Nov 10, 2020
Previously all the fields with nullable values were discarded as invalid.
However there is a wide range of fields that cannot be treated as non-nullable.

For example for enrichment failures bad row, we get following
structure (trimmed):
```
                column_name                | is_nullable |          data_type
-------------------------------------------+-------------+-----------------------------
 processor.artifact                        | NO          | character varying
 processor.version                         | NO          | character varying
 failure.messages                          | YES         | jsonb
 failure.timestamp                         | YES         | timestamp without time zone
 payload.enriched.app_id                   | YES         | character varying
 payload.enriched.base_currency            | YES         | character varying
...
```
Where only processor is non-nullable. Therefore enrichment failures would not be
written with evaluations like:
```
payload.enriched.br_cookies -> Some(null) ->  BigInt -> Left(Invalid type
BIGINT for value Some(null))
```
This PR allows marking fields nullable by default. We might want to consider a
smarter approach at some point.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants