Column-to-Field mapping strategy

[roll]

Overview

We have a list of columns and list of fields. There are two ways to link them:

• based on order (I suppose it's how JTS says we should do)
• based on name

There is a check for it - https://github.com/frictionlessdata/goodtables-py/blob/next/goodtables/checks/head/non_matching_header.py - any logic could be implemented.

For now it uses current goodtables approach checking names using order but removing column from inspection if there is non matching name. Also there is not implemented order_fields option. But this approach could lead to problems described below. It could be solved removing whitespaces from names.

But also may be instead of order_fileds to use option to set mapping strategy:

• map fields to columns using order
• map fields to columns using names

Problems

@rgrp has wrote:

e.g. column named "Number" and JTS has " Number" (edt: see space!) then number column is ignored.

Qu: Why do we even use column names vs JTS names - can't we use JTS order in checking or is the idea we could validate arbitrary JSON data?

Tasks

• For now non_matching_headers(order_fields=True) raises NotImplementedError but it should re-order fields if order_fields option is True. Needed some tricky algorithm because headers and fields could be duplicated etc. [IF NEEDED]

[rufuspollock]

@roll i get this is tricky. For example, we'd generally like to be able to validate a stream of rows where each row is either an array or a dict (basically either what you get from python csv.reader or csv.DictReader ...). My point in the original issue is not that we must accomodate order stuff but just that we should not ignore something if not there - if we expected a column called name and not there i think we should warn or error.

[roll]

So goodtables.next is going to do the next thing based on raised problem:

• if there is any difference between header and field_name it will add non-matching-header error
• but if slugified names are the same ("Number" and "/space/Number" -> "number") it will NOT stop the future inspection of the field (for other errors)

Also order_fields options will force the system to re-order fields as much as possible close to headers before check above.