Introduce EntityEmbeddings

[EssamWisam]

Basic Description

This PR extends the NeuralNetworkClassifier, NeuralNetworkBinaryClassifier, NeuralNetworkRegressor, MultitargetNeuralNetworkRegressor models of the MLJFlux.jl library such that:

• These model now support tables with categorical columns. Iff any are present, an extra entity embedding layer is introduced after the input as described in the paper Entity Embeddings of Categorical Variables by Cheng Guo, Felix Berkhahn.

• It's possible, after training any of such models, to transform the categorical columns of a new sample, seen or unseen in training for the purposes of encoding the categorical column for a further model or transformer in a pipeline

See the updated documentation to learn more.

Implementation Plan

The following was my plan for implementing this feature which is more nontrivial than it may seem in the first glance.

• Implement EntityEmbedder layer and test it with theory-based example
• Introduce the EntityEmbedder layer to MLJFlux with more formal tests that compare a mathematical implementation to the actual one
• Introduce OrdinalEncoding with tests (as well as multi-column encoding transformer from MLJTransforms for later)
• Introduce functionality to infer categorical variables, number of levels and prepare EntityEmbedder input
• Integrate ordinal encoding and preparing categorical embedding inputs in MLJInterface.fit
• Let MLJInterface.fit insert EntityEmbedder into the model chain when needed (input has categorical columns)
• Likewise, adapt MLJInterface.update accordingly
• Adapt the predict and fitresult for ordinal encoding and storing embedding matrices in classifier.jl and regressor.jl
• Separate the case where there is no entity embedding in MLJInterface.fit (refactoring)
• Make categorical embedder completely transparent when no categorical variables are there (instead of being there free of parameters)
• Refactor code in MLJModelsInterface.jl for less redundancy and more organization
• Use better default for the new dimensionality of EntityEmbedder
• Expose new dimensionality argument in types.jl
• Allow transform for each method that accesses embedding matrices of the EntityEmbedder
• Better variable names
• Modify documentation to take into account EntityEmbedder (plan to also make a tutorial(s) later)
• Ensure existing tests pass with no problem
• Finish writing tests for the majority if not all functional components introduced (mainly entity-embedding.jl, entity-embedding-utils.jl and encoders.jl
• Finish writing some end-to-end tests for entity embeddings over the four models

[codecov]

Codecov Report

Attention: Patch coverage is 96.82540% with 8 lines in your changes missing coverage. Please review.

Project coverage is 96.48%. Comparing base (70dff6e) to head (e5d8141).
Report is 2 commits behind head on dev.

Files with missing lines	Patch %	Lines
src/entity_embedding_utils.jl	94.82%	3 Missing ⚠️
src/encoders.jl	97.01%	2 Missing ⚠️
src/fit_utils.jl	92.85%	2 Missing ⚠️
src/mlj_model_interface.jl	97.82%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##              dev     #267      +/-   ##
==========================================
+ Coverage   92.42%   96.48%   +4.06%     
==========================================
  Files          11       14       +3     
  Lines         330      512     +182     
==========================================
+ Hits          305      494     +189     
+ Misses         25       18       -7     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[EssamWisam]

@ablaom I handle differently depending on whether it's an integer or float as in the docs.

[ablaom]

I think MLJBase, which depends on ScientificTypes, re-exports all the public ScientificTypes methods, so you may be able to dump it here.

[EssamWisam]

@ablaom So replace using ScientificTypes: coerce, Multiclass, OrderedFactor with using MLJBase: coerce, Multiclass, OrderedFactor in the test file? This is minor because it's in the test only right and it shouldn't redownload already downloaded package?

[ablaom]

No, I think using MLJBase suffices. All those objects are re-exported.

[ablaom]

As you are now acting on instances instead of types, I'd change the name of your trait from is_embedding_enabled_type to is_embedding_enabled, but this is just a suggestion.

[EssamWisam]

Sure thing.

[ablaom]

Also, do we have any test for transform(::MLJFluxModel, ...) ?

[EssamWisam]

I think transform is tested in entity_embedding.jl

[ablaom]

Okay, I see it's covered in Codecov.

Can I ask that this method (and corresponding test) be moved to "mlj_model_interface.jl", where the other implementations of MLJModelInterface methods (such as fit and predict) live?

[ablaom]

Thanks for the speedy response to my review.

A couple of minor points left. Note in particular the need to annotate the type of model (transformer) in MLJModelInterface.transform overloading. And we should have a test for this method for one of the models.

[EssamWisam]

@ablaom have addressed everything (except removing ScientificTypes from tests). I noticed that after adding categorical variables to integration tests, GPU testing fails :(. Will look into that when I get the chance.

[ablaom]

@EssamWisam You can try raising the tolerance near test/classifier.jl:114, which seems to be the fail.

[EssamWisam]

Hooray it worked 🎉 @ablaom

[ablaom]

Thanks for the updates.

Just a couple more details, as flagged.

[EssamWisam]

@ablaom let's finalize this!

[ablaom]

Thanks again @EssamWisam for this valuable contribution.

Just made a few doc string tweaks after reviewing these again. Ready to go. 🎉