Updating the reference mechanism

[pierrot0]

Right now, references are expected to be of the form "#{[ref]}":

A reference to another object defined within the dataset. References are string of the form #{[ref]}, where ref is either the name of an object defined in the dataset (e.g., a FileObject, or a RecordSet), or one of its components (e.g., a Field in a RecordSet). For the latter case, ref uses '/' to represent nesting (e.g., "#{recordset2/field5}").

(https://github.com/mlcommons/datasets_format/blob/main/docs/croissant-spec.md#reference)

The problems:

1. It doesn't seem like the #{} syntax brings us benefits, aside from unnecessary complexity: references always appear in values which are expected to be of type Reference (containedIn, source, key, references, data), so we know those are references and not plain string.
2. It is not currently possible to distinguish between #{file.csv/filename} where filename is the property filed by Croissant, and #{file.csv/filename} where filename is a CSV column.

Possible solution:

1. Drop the #{} syntax.
2. Always use a "Croissant accessor", possibly using some kind of parameters. For example: csvfile/column:filename to refer to the filename column (column is the accessor, filename is the parameter), and csvfile/filename to refer to the csv filename (accessor with no parameter). This way we can envision other types of accessors, eg: image-files/exif:author to get the author from image EXIF metadata, file.json/jsonpath:$.store.book[*].author to get the authors list from a json file, etc...

What do you folks think?

[josvandervelde]

Sidenote: we should make sure we're allowing for special characters in the variables. Maybe csvfile/column:"including / and space"?
Or just csvfile/column:including / and space?

[pierrot0]

Well the full string is going to be included into quotes already, so the second option seems ok to me:

{
  "source": "csvfile/column:including / and space"
}

The restrictions are the following:

1. resource can include any character except "/" (or it is escaped). If the resource name is a file name, it is unlikely it has a / in there.
2. accessor has to be picked from a list of supported accessor names (eg: column, exif, jsonpath, etc.). Accessor names could theoretically include any character except for ":", but given we decide how the accessors are named, I would stick to [a-zA-Z0-9_]+, at least to start with.
3. parameter can be anything, given it's till the end of the string. This way we can use already existing syntaxes (jsonpath, xpath, etc.) with no escaping needed, beside ". If an accessor would need several parameters, they can define their own syntax, which makes sense for the considered use case (eg: A|B|C vs A,B,C vs something else).

[benjelloun]

2 cents:

• Dropping the #{} syntax makes sense if the properties always expect a value of type reference
• I'm a bit less convinced about the other part... I'm afraid we end up inventing another referencing mechanism.
• For the ambiguity aspect with predefined properties like filename, we can add some prefix to those, e.g., _filename or sth.
• I wonder if we should consider a more structured approach where we use different fields for the source and the reference within the source. This way the source part is always an identifier, and the reference part can become more complex as needed (e.g., json query, etc.)
• A spec that may be interesting to look at: https://rml.io/specs/rml/
  WDYT?

[benjelloun]

Actually, I think it's better to optimize for readability / understandability, than to introduce generic properties.

So, along the lines of your original proposal, how about something like:

{
  "source": {
      "file": "file.csv",
      "column": ["column_name"]
  }
}
{
  "source": {
      "fileset": "fileset-name",
      "file_property": "filename"
  }
}
"source": {
    "fileset": "image-files",
    "file_property": "content"
}
{
  "source": {
      "file": "annotations.json",
      "json_query": ".annotations"
}

[pierrot0]

I agree that non-generic properties are easier to read, at least on small examples.

However generic properties allow adding supported extraction methods or file formats without actually adding properties to the Croissant config format, so I think that's a big plus. This way a library/tool can add support for a specific format or extraction method without the need for the Croissant format to be changed. Ultimately - while we want to document which file formats and methods should be supported for a tool to be compliant with the Croissant format - we also want to allow support for additional formats / methods without changes to the config format. In fact, in the long run, the procedure to add support for a file format or an extraction method should probably to first add unofficial support in a few tools, then define the support as optional and then later when usage has spread, mark the support as mandatory.

If we decide not to use generic properties, we may end-up with many different properties like json_query, and then it becomes difficult to differentiate between properties which are mutually exclusive (eg: json_query, xpath, etc.) from others (eg: content, apply_transform, ?) and to correctly figure out their order. At this point generic properties will become more readable IMO (fewer ambiguities).

I also think it's easier to document if you have fewer properties, for example:

source

Properties

• file|fileset: ...
• data_extraction : one of
  • filename: ...
  • fullpath: ...
  • image: ...
  • exif_metadata: ...
  • content: ...
  • csv_column: Extract a column from CSV file. Parameters:
  • colunn_name: the name of the column to extract.
  • json_path: ...
  • xpath: ...
• data_extraction_params: list of parameters, refer to expected parameters above, based on data_extraction value.
• applyTransform: [optional] transformations to apply on top of the extracted method as specified through data_extraction and data_extraction_params.

<end_of_documentation_example>

For those reasons I feel strongly in favor of generic properties.
We could also update on the syntax using generic properties, maybe something like the following would be easier to read:

{
  "source": {
    "fileset": "csv-files",
    "data_extraction": {
      "method": "csv_column",
      "params": ["foo"]
    }
  }
}

[benjelloun]

I would push back on this a bit. I don't think we need that many mechanisms to support the most common data formats, and it seems important to make those as easy to express as possible.

A halfway proposal: How about relying primarily on non-generic properties, and having an escape hatch generic mechanism to describe formats that don't fit or are not yet supported?

A similar discussion can be had for "applyTransform", where we want to define common operations with specific properties (e.g., "regex" or "separator"), but may want to have an extension mechanism for non-standard transformations.

[marcenacp]

@pierrot0 What do you think of dropping equivalent representations? Instead of having csv_column, json_path and xpath, we could have only json_path, because those three extractions methods boil down to a JSON path representation.

[pierrot0]

I suppose I am fine with that. But here this was used as an example of methods which could be different, so I expect there will be other methods which are different.

[pierrot0]

A halfway proposal: How about relying primarily on non-generic properties, and having an escape hatch generic mechanism to describe formats that don't fit or are not yet supported?

I don't think that would be a good compromise: now you have two ways of specifying the same information, which doesn't help with making config format easy to document and understand, nor to implement support for it.

A similar discussion can be had for "applyTransform", where we want to define common operations with specific properties (e.g., "regex" or "separator"), but may want to have an extension mechanism for non-standard transformations.

Agree, we should try to be coherent when possible. So as it stands, I would support something like:

"applyTransform": {
    "method": "regexp_extract",
    "args": [".*_(val|train)2014\\.json$"],
}

over the less generic alternative:

"applyTransform": {
    "regex": ".*_(val|train)2014\\.json$"
}

Now whether that's an option or another, I think I would prefer the group to decide sooner than later.
@josvandervelde @marcenacp to read the latest arguments. Maybe we can figure out a consensus at the upcoming meeting.

[benjelloun]

Just as a data point: schema.org tends to favor semantic attributes over generic ones. I believe the argument is that ease of definition and immediacy of usage are more important than future extensibility.

[marcenacp]

Could we have both the readability/compliance to schema.org and the non ambiguity by writing:

{
  "source": {
    "fileset": "csv-files",
    "data_extraction": {
      "csv_column": "foo"
    }
    ...
  }
}

data_extraction is typed oneof: csv_column | file_property | ....

Like this:

1. All things related to ml:data_extraction have their own isolated RDF node, just like applyTransform .
2. We still respect schema.org (no generic attribute) and the readability.

[pierrot0]

I am good with that approach. @benjelloun would that work for you?

[benjelloun]

Yes, I think this makes sense. The added benefit of creating this data extraction node is that we can make the data selection richer as needed. For instance we may want to specify a more complex query that only retrieves specific records, e.g. based on a specific split or language.

Nit on naming: I would favor the term "selection" over "extraction", and use a verb form, to be consistent with "applyTransform". So maybe sth like "selectData"?

[pierrot0]

Cool thanks for the discussions, I'll try to send a few PRs and then we can iron out things there.