Transform2: Basic TAP validation

[kcoyle]

• Does this have a header row?
• Are the column headers from the TAP vocabulary? (case insensitive?)
  • message: report any that are not TAP, but continue
• Columns can be in any order

[kcoyle]

as per conversation between @kcoyle and @tombaker

configuration file can:

• be used to verify TAP column headers (the program can allow other headers, but should offer to message these)
• be used to give data types for columns (and could be modified to carry data types for non-TAP columns)
• be used to translate from table headers to TAP headers if shortened or other language versions are desired

[kcoyle]

Should we consider TAP column headers to be case sensitive or case insensitive? So should

valueDatatype = valueDataType ?

The options seem to be:

1. echo non-matching case in the same way as unknown column headers
2. accept headers if they match as case insensitive

[philbarker]

I suggest following Postel's law (aka the robustness principle) of being "conservative in what you do, be liberal in what you accept from others". In this case our documentation and examples should be case sensitive, lowerCamelCase, but implementation would be well advised to transform to lower case and remove any spaces. So both valueDataType and value data type, while being non-conformant, should work (but perhaps with a warning about the non-conformance).

[tombaker]

@philbarker So we should implement some sort of preprocessing of a CSV before the CSV parser initializes the dictionary of its contents??

[kcoyle]

I agree that the TAP headers should be allowed to be case insensitive. I'm less sold on allowing spaces, but don't oppose that.

That said, I see this as an eventual enhancement and wouldn't want to require it for our July demonstration at LD4 unless it is an easy fix.

[tombaker]

@philbarker @kcoyle
The script now ignores, in headers:

• upper/lowercase (normalized to lower)
• spaces
• dashes
• underscores

So for valueConstraintType, it will recognize:

• Value Constraint Type
• value_constraint-type

As an undocumented (and at-risk) feature, it also recognizes short forms - see issue 34

[kcoyle]

See documentation of dctap-python for particulars.