Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Should we make blank values and empty string to None in csv? #4939

Closed
haohuaijin opened this issue Oct 16, 2023 · 4 comments · Fixed by #4942
Closed

Should we make blank values and empty string to None in csv? #4939

haohuaijin opened this issue Oct 16, 2023 · 4 comments · Fixed by #4942
Labels
arrow Changes to the arrow crate bug question Further information is requested

Comments

@haohuaijin
Copy link
Contributor

Which part is this question about

arrow-csv

DataType::Utf8 => Ok(Arc::new(
rows.iter()
.map(|row| Some(row.get(i)))
.collect::<StringArray>(),

Describe your question

related to apache/datafusion#7797
In our current implement, we make blank values and empty string to empty string.
But in spark, the blank values and empty strings are treated equally to NULL, should we also make blank values and empty value to NULL.

Additional context

@haohuaijin haohuaijin added the question Further information is requested label Oct 16, 2023
@tustvold
Copy link
Contributor

I think making the null_regex apply to string columns makes sense to me, I honestly thought that was already the case

@haohuaijin
Copy link
Contributor Author

haohuaijin commented Oct 16, 2023

I find we don't apply null_regex to string columns. If you don't mind, I want to fix it.

@alamb
Copy link
Contributor

alamb commented Oct 16, 2023

This also makes sense to me -- thank you @haohuaijin

@alamb alamb added the bug label Oct 16, 2023
@tustvold tustvold added the arrow Changes to the arrow crate label Oct 18, 2023
@tustvold
Copy link
Contributor

label_issue.py automatically added labels {'arrow'} from #4942

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
arrow Changes to the arrow crate bug question Further information is requested
Projects
None yet
3 participants