Programmatic creation of pipelines

[CameronBieganek]

Consider the following code:

julia> using MLJ

julia> function make_pipeline(model, name::AbstractString; drop_last=false)
           @pipeline(
               OneHotEncoder(drop_last=drop_last),
               model,
               name=name
           )
       end
ERROR: LoadError: UndefVarError: drop_last not defined
Stacktrace:
 [1] top-level scope at REPL[5]:1
 [2] eval at ./boot.jl:331 [inlined]
 [3] eval(::Expr) at ./client.jl:449
 [4] pipeline_preprocess(::Module, ::Expr, ::Vararg{Any,N} where N) at /Users/bieganek/.julia/packages/MLJBase/2UxSl/src/composition/models/pipelines.jl:310
 [5] pipeline_(::Module, ::Expr, ::Vararg{Any,N} where N) at /Users/bieganek/.julia/packages/MLJBase/2UxSl/src/composition/models/pipelines.jl:439
 [6] @pipeline(::LineNumberNode, ::Module, ::Vararg{Any,N} where N) at /Users/bieganek/.julia/packages/MLJBase/2UxSl/src/composition/models/pipelines.jl:562
in expression starting at REPL[5]:2

I'm not sure precisely what's going on here, but it looks like @pipeline was only designed to be called at the top-level, which makes it more laborious to create a bunch of related pipelines that use different supervised ML models for the target prediction.

I'm not sure how feasible this is, but it would be nice if there were a simple interface for creating pipelines without using a macro (in the spirit of this Discourse comment).

Version Info:

(@v1.4) pkg> status MLJ MLJBase MLJModels MLJLinearModels
Status `~/.julia/environments/v1.4/Project.toml`
  [add582a8] MLJ v0.12.0
  [a7f614a8] MLJBase v0.14.2
  [6ee0df7b] MLJLinearModels v0.5.0
  [d491faf4] MLJModels v0.11.0

julia> versioninfo()
Julia Version 1.4.0
Commit b8e9a9ecc6 (2020-03-21 16:36 UTC)
Platform Info:
  OS: macOS (x86_64-apple-darwin18.6.0)
  CPU: Intel(R) Core(TM) i7-9750H CPU @ 2.60GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-8.0.1 (ORCJIT, skylake)
Environment:
  JULIA_EDITOR = vim

[ablaom]

Yes, unfortunately, the arguments of @pipeline get evaluated in top-level scope of the calling module, which limits its use 😞 .

A non-macro solution is difficult to imagine here because a new struct is being defined. I thought about defining a "universal" composite model type whose single field is a dictionary of component models (with a type parameter that is different for each new "architecture") but this seemed unnatural and I did not pursue it. For the special case of pipelines, it might be worth revisiting, I suppose.

Note that for your particular use-case, the pipeline's fields are mutable. So you can do things like this:

pipe = @pipeline OneHotEncoder RidgeRegressor 
for rgs in [`RidgeRegressor`, `KNNRegressor`, ...]
    pipe.ridge_regressor = rgs
    evaluate(pipe, X, y, measure=rms)
end

[CameronBieganek]

pipe = @pipeline OneHotEncoder RidgeRegressor 
for rgs in [`RidgeRegressor`, `KNNRegressor`, ...]
    pipe.ridge_regressor = rgs
    evaluate(pipe, X, y, measure=rms)
end

That's not a bad solution! Though it has the odd side effect that my random forest model lives in the ridge_regressor field of the pipeline.

I thought about defining a "universal" composite model type whose single field is a dictionary of component models (with a type parameter that is different for each new "architecture") but this seemed unnatural and I did not pursue it.

Just thinking out loud here... It seems like this is a case where a little type instability is not a big deal. So we could have a "universal" composite type called Pipeline with non-concrete fields that looks something like this:

struct Pipeline
    steps::AbstractVector
end

Pipeline(args...) = Pipeline([args...])

pipe = Pipeline(
    OneHotEncoder(),
    RandomForestClassifier()
)

[DilumAluthge]

This would be type-stable:

struct Pipeline{T <: Tuple}
    steps::T
end

[ablaom]

That's not a bad solution! Though it has the odd side effect that my random forest model lives in the ridge_regressor field of the pipeline.

Yeah. I originally thought of generating names like model1, model2, and so forth. But I reckon that for the main-case the status-quo makes inspecting fitted_params and reports easier.

And yes, your suggestion is more-or-less what I had in mind. But how, for example, do you access the nested objects? A lot currently depends on the getproperty access. Maybe if steps is a named tuple? Then you could do pipe.steps.classifier (but pipe.classifier is nicer)??

[CameronBieganek]

Yeah, maybe it would make sense for steps to be a named tuple. And then maybe getproperty could be overloaded?

getproperty(p::Pipeline, s::Symbol) = getproperty(p.steps, s)

And to build on @DilumAluthge's suggestion, perhaps the struct definition could be

struct Pipeline{T <: NamedTuple}
    steps::T
end

However, I don't claim to have an understanding of the broader implications...

[ablaom]

Good idea about the overloading!

BTW I should have said that you can always roll your own composites using the learning network syntax. Using @from_network you could use whatever names for the fields you like. Or, to do what you want without macros in for your use case:

mutable struct OneHotWrapper{P} <: ProbabilisticComposite
    probabilistic_model::P
    drop_last::Bool
end

function MLJ.fit(wrapper::OneHotWrapper, verbosity, X, y)
    Xs = source(X)
    ys = source(y)

    hot = OneHotEncoder(drop_last=wrapper.drop_last)
    mach1 = machine(hot, Xs)
    W = transform(mach1, Xs)

    mach2 = machine(wrapper.probabilistic_model, W, ys)
    yhat = predict(mach2, W)

    mach = machine(Probabilistic(), Xs, ys, predict=yhat)
    fit!(mach, verbosity=verbosity - 1)
    return mach()
end


@load DecisionTreeClassifier
wrapper = OneHotWrapper(DecisionTreeClassifier(), true)

X, y = @load_iris;

# add a categorical feature:
bogus = coerce(rand(["blue", "red"], length(y)), Multiclass);
X = merge(X, (bogus=bogus,))

julia> evaluate(wrapper, X, y, measure=cross_entropy)
Evaluating over 6 folds: 100%[=========================] Time: 0:00:00
┌───────────────┬───────────────┬───────────────────────────────────────────── ⋯
│ _.measure     │ _.measurement │ _.per_fold                                   ⋯
├───────────────┼───────────────┼───────────────────────────────────────────── ⋯
│ cross_entropy │ 3.12          │ [2.22e-16, 2.22e-16, 4.33, 2.88, 4.33, 7.21] ⋯
└───────────────┴───────────────┴───────────────────────────────────────────── ⋯

editted to add a categorical feature to the demo data

[CameronBieganek]

Ok, good to know! I haven't played around with the learning network stuff yet. Though of course the appeal of syntax like

pipe = Pipeline(
    OneHotEncoder(),
    RandomForestClassifier()
)

is that you get to avoid the "boiler plate" calls to machine, transform, and predict.

[CameronBieganek]

@ablaom If you don't mind, I'd like to play around with this. I will need to do some experimentation, of course, and then I can put together an RFC pull request.

cc-ing @ExpandingMan since he's commented about @from_network and @pipeline.

[ablaom]

Happy of course for you to have a go! And I believe you are well-equipped to make a valuable contribution here, if you have the time.

I should warn you there is quite a lot of functionality built into the current @pipeline and my own priorities are currently in other areas. I think a POC would be a fantastic start, and I am happy to provide guidance. However, I should say I may not have the resources to move a POC to a complete solution without a lot of further assistance.

Let me know if you'd like to zoom at some point in the process. This can sometimes save a lot of time.

[CameronBieganek]

Ok, sounds good!

[CameronBieganek]

@ablaom Quick question. According to the @pipeline docstring,

At most one of the models may be a supervised model, but this model can appear in any position.

Off the top of my head, I can't think of a use-case for not having the supervised model as the last model in the pipeline. (Not counting target transformations.) Can you give me an example where the supervised model is not the last?

[ExpandingMan]

If you consider a transformation of the output (i.e. an inverse transformation of something that was applied at the input) to be a model (really it's just the application of a model), then this would be a very common case in which this is needed.

[CameronBieganek]

Sure, but that's already handled explicitly in @pipeline by the target and inverse keyword arguments. So, I'm looking for an example where the supervised model is not the last model in the sequence of arguments supplied in the vararg positional arguments to @pipeline. 🙂