Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Need to manually strip newlines #89

Closed
frekw opened this issue Mar 25, 2019 · 1 comment
Closed

Need to manually strip newlines #89

frekw opened this issue Mar 25, 2019 · 1 comment

Comments

@frekw
Copy link

frekw commented Mar 25, 2019

Given the current input data

id,joke
1,"What did the bartender say to the jumper cables? You better not try to start anything."
2,"Don't you hate jokes about German sausage? They're the wurst!"
3,"Two artists had an art contest... It ended in a draw"
4,"Why did the chicken cross the playground? To get to the other slide."
5,"What gun do you use to hunt a moose? A moosecut!"
6,"If life gives you melons, you might have dyslexia."
7,"Broken pencils... ...are pointless."
8,"What did one snowman say to the other snowman? 'Do you smell carrots?'"
9,"How many hipsters does it take to change a lightbulb? It's a really obscure number. You've probably never heard of it."
10,"Where do sick boats go? The dock!"

The follwing fails

File.stream!("priv/jokes.csv")
|> Stream.take(10)
|> CSV.decode(headers: true, preprocessor: :lines)

with

{:error,
 "Stray quote on line 1 near \"What did the bartender say to the jumper cables? You better not try to start anything.\""}

Due to the string being parsed becoming: "1,\"What did the bartender say to the jumper cables? You better not try to start anything.\"\n". The \n makes it fail, which I would have expected the library to handle out of the box.

No funny encoding is going on:

file priv/jokes.csv
priv/jokes.csv: ASCII text, with CRLF line terminators

In order to fix it I had to manually strip the newlines, like so:

File.stream!("priv/jokes.csv")
|> Stream.map(fn x -> String.trim(x) end)
|> Stream.take(10)
|> CSV.decode(headers: true, preprocessor: :lines)

Is this intended behaviour?

@beatrichartz
Copy link
Owner

Thanks for raising this, this is indeed not intended behavior. A fix has been released in 2.3.1, can you update and give it a try?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants