How to preprocess row before deserializing to struct?

[evbo]

Hi, I have a disgusting csv format where someone thought it was clever to omit delimiters when the remaining columns of a row are all null.

So for example:

x,y,z,
1,2,3,
1

instead of :

x,y,z,
1,2,3,
1,,,

What I would like to do is append as many , as are missing to the StringRecord and then ask csv to deserialize it into a struct:

struct Xyz {
  x: i32,
  y: Option<i32>,
  z: Option<i32>
}

I know how to append to the as_slice() result from StringRecord. but how can I then ask this library to deserialize the &str after I've appended the missing ,?

[BurntSushi]

Once you have a StringRecord, you can use its StringRecord::deserialize method to invoke serde. Or just do it by hand if your fields are just numbers.

[BurntSushi]

It's odd I guess. Honestly, you have an odd use case, and I think you're just not going to have a nice paved path here.

I think if I were in your shoes I would have just written a custom parser for my data-that-looks-like-csv-but-isn't.

[evbo]

one last suggestion if it's okay: serde handles this scenario by presuming a missing value should be None:

#[serde(deserialize_with = "object_empty_as_none")]

Do you see supporting that same feature for missing csv columns being possibly valuable? I did learn the other day Excel/Libreoffice will load csv with inconsistent row lengths (missing delimiters)

[BurntSushi]

Have you tried ReaderBuilder::flexible? If so, what happens?

[evbo]

yes flexible is required for it to read this sort of csv. I mean the step involving deserializing could support missing delimiters as assuming they are None values for those columns missing.

[BurntSushi]

Maybe. I'm not sure if the serde deserializer for CSV can easily do it. You're welcome to try submitting a patch for it. But the code is extremely hairy. It's a big part of what this crate hasn't evolved much and why there are so many serde caveats. Look at the issue tracker. There are tons of issues and almost all of them are related to serde. But changing the deserializer is just something that I've personally found very difficult.

[korrat]

I'm working on a similarly bad format. What I do is pad the record just before deserialization:

record.extend(core::iter::repeat("").take(headers.len() - record.len()));
let record = match record.deserialize(Some(headers));