Unexpected `Missing` appearing in schema

[ablaom]

julia> iris = DataFrames.DataFrame(iris);

julia> schema(iris).scitypes
(Union{Missing, Continuous}, Union{Missing, Continuous}, Union{Missing, Continuous}, Union{Missing, Continuous}, Union{Missing, Multiclass{3}})

But the iris dataset has no missing values.

@jbrea Any idea how Missing is getting into the eltype of these columns?

[jbrea]

By default the ARFFFiles reader assumes all columns could contain missing values.
Maybe we should convert after loading all columns with any(ismissing.(col)) == false here or directly in the ARFFFiles package (@cjdoris)?

[ablaom]

Thanks for this clarification.

It seems to me this an option ARFFFIiles should provide. One possibility is to just pass any extra kwargs specified in OpenML.load(id; parser=..., kwargs...) off to the specified parser. Then for now we could at least do OpenML(61, missingcols=false).

Let's see what @cjdoris has to say.

[cjdoris]

Hello, sorry for the slow response, I have been on holiday.

To be honest, I'd rather not support this in ARFFFiles, since it adds an extra layer of complexity that is not ARFF-specific. If I were you I'd simply make a little utility function to narrow the types of the columns and construct a new ColumnsDictTable.

[jbrea]

I'd rather not support this in ARFFFiles, since it adds an extra layer of complexity that is not ARFF-specific.

Well, ARFFFiles have nice support for missing values with the ? syntax. When I get a column of type Union{Missing, T} I usually expect to find at least one missing value in this column. I would therefore find it most natural, if ARFFFiles only returned Union{Missing, T} if there is actually at least one missing value in a given column.

I think it would also be very cheap to detect missing values while parsing. E.g. one could start with Union{Missing, T} by default but have colmissings .== false at the start and change only when entering here. For all columns where at the end of parsing colmissings is still false one could change the type here.

If you're not totally against it I could try a PR in ARFFFiles. Otherwise I'm happy to add a little utility function here.

[cjdoris]

Ok I'm convinced! Feel free to make a PR. I think we should just add an argument to readcolumns so that at the end where it constructs the table it optionally tries to narrow columns which may have contained missing but don't.

You may also want to consider using the missingnan option to convert missing to NaN in numeric columns.

[jbrea]

This is fixed with the new release of ARFFFiles v1.4.0.