Documentation is missing many important details

[lukehutch]

I have been trying to use MLTK for a regression problem, and I found there are many important details missing from the documentation:

• There is no info on how to set up datasets in memory programmatically, rather than reading from file (see Need for programmatic setup of datasets #14).
• On the page https://github.com/yinlou/mltk/wiki/Basics , there is no explanation of "(class)". I eventually figured out that it's how you set the target variable. That doesn't make sense for regression though, the target variable is not a class. It's also made more confusing because the section "Example attribute file" has the "(class)" attribute last, but under "Sparse input format", you list the target first, implying that the position is significant (which it is not).
• There is no explanation that if the number of attributes doesn't match the number of columns, that lines are simply ignored. This is more confusing because it's not clear comparing sections "Example attribute file", "Example data file" and "Sparse Input Format" whether the target attribute even needs to be set in the attribute file or not, or whether its discrete/continuous nature is inferred based on whether you're doing classification or regression (and whether it is inferred that the first column is always the target). If you have the wrong number of attributes, no instances are read.
• If you don't specify the target attribute, it is set to NaN, and the regression will fail unless you go through all the Instances and set a target value on each one. I couldn't figure out why I was getting NaN until I read through the code.
• You do say that the attribute file parameter to InstancesReader.read is "optional", but even the JavaDoc doesn't explain that you need to set this to null for this purpose. The way that attributes are automatically inferred for sparse and dense cases if you do set this to null is not explained. (For example, the target is always unset if you don't specify an attributes file, so you'll always get NaN for the target values unless you manually set the target, as described above.)
• There is no example explaining the usage with a dense datafile.

That's as far as I have gotten so far... I finally got my first regression results, but it took me several hours to figure out how to use this properly... hopefully this feedback helps!

[yinlou]

Hey @lukehutch, thanks for the feedback. I'm still working on the wiki pages (they are not complete yet).

1. In dense format, class (or target) attribute can appear at any position since for most use cases on dense format, we have reasonable features to handle and therefore providing an additional attribute file can help. In sparse format, usually we have many features and the input is highly sparse. Therefore I use the standard libsvm-like format. In this case, an attribute file is not provided and therefore we need to follow some convention to get the target position.
2. I don't think that's the case. Can you email me a data file with attribute file so that I can test? It's actually perfectly fine if the number of attributes in attribute file is less than the number of columns in data file. The remaining columns will be read but when building models, all those columns will be ignored.
3. I will make this clear in the wiki.
4. This is something that I'll improve in the future. Here're the cases where attribute file is null and how it is parsed by mltk. For dense format, that means there would be no target value (NaN is assigned). This might be used in the future for unsupervised learning algorithms (similarly, if there's no target attribute specified in attribute file, there will be no target value). For sparse format, current the first one has to be the target.

Thanks again for the feedback.

[lukehutch]

On #2 (assuming you're referring to my third bullet point?): Here is the relevant code:

https://github.com/yinlou/mltk/blob/master/src/main/java/mltk/core/io/InstancesReader.java#L72

In the case of dense instances with the wrong number of attributes, the line is skipped by the if-statement in line 76.

[yinlou]

Now I remember. In the input, number of features in the attribute file does need to match the number of columns in the data file. I wanted to enforce that so that feature description in the attribute file always matches the corresponding data file. In which scenario do you want to have a smaller attribute file?

[lukehutch]

The situation arose because I misunderstood the docs. I didn't know that "(class)" implied the target, because I was doing regression, not classification, and this tag was not described in the text. As I tried to portray in the original bug report, it is not clear from the examples as to how you should specify the target. In the case where you show "(class)" in the attributes file, you have the class occurring last. However, under "Sparse input format", you show the target occurring first. Because I didn't see a mention of target in the attributes, and because I didn't know "(class)" meant "target, whether regression or classification", I assumed that the target was always supposed to be in the first column. Thus my attributes file had one fewer attribute than there were columns.

I think the way to fix the documentation is to have two complete examples: One for classification, one for regression. Or at least make all the examples of attribute and datafiles refer to a single consistent example, and explain that "(class)" must be present and means the target, whether you're doing regression or classification.