Migration ARFF to Parquet on the OpenML server

[PGijsbers]

This is a centralised discussion about the server side changes (being) made to the datasets in their conversion from ARFF to Parquet. Related on-going discussions that reference the server state of different datasets:

• OpenML Sparse Dataset support #46
• Unit tests fail due to switch to Parquet openml-python#1157 (comment)
• OpenML API parquet migration - Phase 2 OpenML#1154

Let's keep the relevant information about the migration as it relates to server data in this thread.
This is not for connector specific discussions (for example, how openml-python handles this).
@joaquinvanschoren @prabhant @sebffischer

[sebffischer]

A couple of remarks:

• The column "LRS.name" disappeared in parquet version of dataset with id 313 OpenML#1165 this is kind of a weird bug
• The data features should be recalculated somehow I think. For example the data type booolean was not available in arff I think

[sebffischer]

Also it would be great if the types string and categorical were distinguishable from looking at the data features (if this information is somehow updated using the new parquet files)

[sebffischer]

Also it seems that the parquet urls from the test server are wrong.
With wrong I mean that they point to the parquet urls of the publis server.

Edit: more info

[PGijsbers]

Also it would be great if the types string and categorical were distinguishable from looking at the data features (if this information is somehow updated using the new parquet files)

Do you have an example? We want to see if this issue was with the ARFF file or specifically introduced in the conversion.

[PGijsbers]

Also it seems that the parquet urls from the test server are wrong.

Parquet URLS from the test server have been disabled for now, until we have a separate minio (or bucket) for the test server.

[PGijsbers]

openml/OpenML#1165 this is kind of a weird bug

We'll look into that, and for the conversion scripts we'll have a closer look to preserve the feature data, or encode it into correct data types where ARFF was previously not expressive enough (e.g., boolean, 8-bit integers).

[sebffischer]

This was a confusion from my side, sorry!

[PGijsbers]

The following will be changed for the conversion script:

• feature names will have leading and trailing whitespace removed.
• numeric features with only integer values in the range [0, 255] will be stored as uint8.

Additionally feature meta-data needs to be updated:

• reflect which features are boolean (instead of categorical)