Validate does not seem to handle double quotes like most CSV writers/readers

[jordanpadams]

The PDS4 Standards Reference says it is based on RFC 4180, which states:

If double-quotes are used to enclose fields, then a double-quote
appearing inside a field must be escaped by preceding it with
another double quote.

But when we try to encode double-quotes like ""allophane"", we get this error:

 ERROR  [error.validation.invalid_field_value]   table 1, record 1, field 3: The field value 'The four "allophane" and gel spectra were added to spectral libraries used to model Mini-TES spectra of fresh and altered rocks from Gusev (Adirondack, Clovis, Wishstone, and Watchtower) [16].' that starts with double quote should not contain double quote(s) 

To make validate happy, we have been either:
(1) post-processing CSV content to convert internal "" (as written by the Pandas CSV writer) to ''
(2) pre-processing strings to convert " to '' before writing to CSV

Neither one is a great solution. What gives?

[pds-ops]

Great question, Jordan! The PDS Delimiter Separate Value (DSV) formats is "based upon" RFC 4180, but has been customized. Per section 4C.1, bullet 6 of the PDS4 Standards Reference:

Fields may optionally be bracketed by a pair of bounding double quotes...
If bracketing double quotes are used, there may be no double quotes within the field itself.

So unfortunately, this validation failure is correct. Preprocessing would be required in order to make this valid according to the PDS DSV. Hope that helps!

[wkiri]

I would be interested to know the reason for the deviation from RFC 4180 - does it yield benefits elsewhere? If not, I'm interested in proposing a change to align the PDS standard with RFC 4180 so that users can use standard CSV libraries to generate compatible content without extra work. Thanks!

[jordanpadams]

@wkiri very late on this comment, but per this comment:

I would be interested to know the reason for the deviation from RFC 4180
Unfortunately, this is a quirk of the PDS4 system. I agree following an existing standard out of the box would have made more sense, but the PDS had some older, nuanced data sets they wanted to support, and modified it for those needs.

[wkiri]

@jordanpadams Thanks for the follow-up!