Option to ignore trailing and leading whitespace

[hayfield]

At the moment, the Datastore deals with values as-is. This means that if attribute values have erroneous trailing or leading whitespace, queries will not identify this data.

The problem was identified in #247.

The Datastore is designed to return data as it is provided, without modifying the data. As such, any enhancement must be at the querying stage rather than the data-input stage.

#253 is a PR to provide potential technical solutions to this problem.

[hayfield]

The ideal solution would be that all data is valid, though there is other work surrounding validation before that will be possible.

[andylolz]

**¡Controversial opinion!**

I’m just not sure the datastore should attempt to solve this… If codes (from complete codelists) have leading or trailing spaces then (I think) they’re not valid, and the publisher should fix this. So this seems like a data problem, and I think the datastore silently fixing data problems sets a bad precedent.

[hayfield]

This would also be a non-backwards compatible change that is not a bug fix.

At the moment, if you query a valid value with no whitespace, you can guarantee that the returned data is exactly what you asked for. If the Datastore changed it results to return anything that is invalid against a query for valid data, any system using this data must then be modified to deal with the spurious results.

As such, any change that leads to returning anything more than is strictly queried for would have to be explicitly opt-in.

[andylolz]

any change that leads to returning anything more than is strictly queried for would have to be explicitly opt-in.

Ahhh! Okay, gotcha – that’s a useful clarification.