Realize performance improvements for models implementing new data front-end #501
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
add fit_only! logic that caches reformatted/resampled user-data add test minor comment change "Bump version to 0.16.3" resolve #499 add test add tests for reformat front-end oops add model api tests for reformat front-end tidy up add test for reformat logic in learning network context do not update state of a Machine{<:Static} on fit! unless hyperparams change add tests allow speedup buy-out with machine(model, args..., cache=false) oops have KNN models buy into reformat data front-end for better testing introduce "back-end" resampling in evaluate! implement data front-end on prediction/transformation side update machine show more tests; add cache=... to evaluate(model, ...) more tests make `cache` hyperparam of Resampler for passing onto wrapped machines more tests Composite models do not cache data by default (no benefit) correct comment bump [compat] MLJModelInterface = "^0.3.7" (essential)
Codecov Report
@@ Coverage Diff @@
## dev #501 +/- ##
==========================================
- Coverage 82.72% 82.57% -0.15%
==========================================
Files 39 39
Lines 2935 2962 +27
==========================================
+ Hits 2428 2446 +18
- Misses 507 516 +9
Continue to review full report at Codecov.
|
Merged
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Continuation of #492 with squashed commits.
Addresses key parts of #309.
Builds on the MLJModelInterface PR 76 (now part of MLJModelInterface 0.3.7) which provides a way for the implementer of MLJ's model interface to articulate: (i) how user-data (eg, a table) is transformed into some model-specific representation needed by the core algorithm (eg, a matrix) - the implementer overloads
reformat; and (ii) how one can construct samples of the model-specific representation given a vector of observation indices - the implementer overloadsselectrows. Detailed responsibilities of this optional "data front-end" implementation are given in this proposed doc PR.This PR introduces optional caching of the model-specific data representations for machines and for subsamples of these representations, which could speed up computations (at the cost of larger memory allocations). Machines have a new type parameter
Cwhich determines whether or not they buy into data caching, withC=truethe default. A user opts-out during machine construction, as inmach = machine(model, X, y, cache=false).Some key implementation details to help with a review:
C(see above) defaulting totrue, except for composite models, where caching provides no obvious advantage. (Machines within in a learning network defining the composite may still cache with some benefit.)dataandresampled_datafor caching model-specific representations for the what is returned by[arg() for arg in args()](recallargsis a tuple ofSourceorNodeobjects). We will refer to the latter as user-supplied data.fit_only!at https://github.com/alan-turing-institute/MLJBase.jl/blob/2eda522877451fdd26838faf279ac74c5a5a837c/src/machines.jl#L417 is adapted to applyreformatto the user-supplied data whenfit!is first called, or the user-supplied data has changed because of upstream changes in a learning network. This data is cached for future calls tofit!;datais resampled according to therowsspecified in thefit!call and also cached. Future calls tofit!only performreformatandselectrowsas needed. All this assumesC=trueand otherwisereformatandselectrowsare called every time and nothing is cached.predict, say, is called on a machine with therows=....syntax, then, assuming the machine has caching enabled,selectrowsis applied to thedatacached, avoiding a call toreformat. If new dataXnewis supplied instead, as inpredict(mach, Xnew), then naturallyreformatis then necessary. The relevant changes are in src/operations.jlpredict(mach, Xtest)as currently, we callpredict(mach, rows=test). This avoids one call toreformat. However, there is one case distinction: the uncommon case where one of the measures hasis_feature_dependent=true(for measures withXin the signature), in which caseXtestmust be still be created, using the slowselectrows(X, test)fallback.cache=...in machine constructors, and theResampledobjects. An interface point forcacheinTunedModelsis also needed and addressed in this MLJTuning PR which passes locally (Julia 1.5).