Add cookbook of cross validation with pipeline by vinx13 · Pull Request #4380 · shogun-toolbox/shogun

vinx13 · 2018-07-20T14:05:30Z

data pr shogun-toolbox/shogun-data#164
also need to merge #4377 first

karlnapf · 2018-07-20T18:59:46Z

@@ -0,0 +1,34 @@
+============================
+Cross Validation on Pipeline


on "a" pipeline

karlnapf · 2018-07-20T19:00:12Z

+Cross Validation on Pipeline
+============================
+
+In this example, we illustrate how to use cross-validation with :sgclass:`CPipeline`.


maybe link to some cross-validation cookbook. There is some way to link to notebooks, I think MKL does this

ha it is just below, nevermind :)

karlnapf · 2018-07-20T19:01:43Z

+-------
+Example
+-------
+We'll use as example a binary classification problem solvable by a pipeline consisted of a transformer :sgclass:`CPruneVarSubMean` and a machine :sgclass:`CLibLinear`.


Slightly weight English:
We demonstrate a pipeline consisting of a transformer ..., and LibLinear for binary classification.
(maybe also link to the liblinear cookbook)

karlnapf · 2018-07-20T19:02:55Z

+-------
+We'll use as example a binary classification problem solvable by a pipeline consisted of a transformer :sgclass:`CPruneVarSubMean` and a machine :sgclass:`CLibLinear`.
+
+Imagine we have files with training data. We create :sgclass:`CDenseFeatures` (here 64 bit floats aka RealFeatures) as


I know this is copy pasted, but this is now outdates when we use factories.
Just say: "We create CFeatures and CLabels via loading from files"

karlnapf · 2018-07-20T19:03:18Z

+.. sgexample:: cross_validation_pipeline:create_features
+
+
+We use :sgclass:`CPruneVarSubMean` to normalize the features and then use :sgclass:`CLibLinear` for classification.


This sentence basically repeats the intro, so I would just remove it

karlnapf · 2018-07-20T19:06:40Z

+
+
+We use :sgclass:`CPruneVarSubMean` to normalize the features and then use :sgclass:`CLibLinear` for classification.
+The transformer and the machine are chained as a :sgclass:`CPipeline`.


We create a Cpipeline, and chain the transformer and the classifier.

actually, I would mention that "we first chain all transformers, and then finalize the pipeline with the classifier" (since you use "then")

karlnapf · 2018-07-20T19:07:03Z

+.. sgexample:: cross_validation_pipeline:create_pipeline
+
+Next, we initialize a splitting strategy :sgclass:`CStratifiedCrossValidationSplitting` to divide the dataset into :math:`k-` folds for the :math:`k-` fold cross validation.
+We also have to decide on an evaluation criterion class (from :sgclass:`CEvaluation`) to evaluate the performance of the trained models.


see CEvaluation

karlnapf · 2018-07-20T19:07:15Z

+.. sgexample:: cross_validation_pipeline:create_pipeline
+
+Next, we initialize a splitting strategy :sgclass:`CStratifiedCrossValidationSplitting` to divide the dataset into :math:`k-` folds for the :math:`k-` fold cross validation.
+We also have to decide on an evaluation criterion class (from :sgclass:`CEvaluation`) to evaluate the performance of the trained models.


last word model (singular)

karlnapf · 2018-07-20T19:09:06Z

+Next, we initialize a splitting strategy :sgclass:`CStratifiedCrossValidationSplitting` to divide the dataset into :math:`k-` folds for the :math:`k-` fold cross validation.
+We also have to decide on an evaluation criterion class (from :sgclass:`CEvaluation`) to evaluate the performance of the trained models.
+In this case, we use :sgclass:`CAccuracyMeasure`.
+We then instantiate :sgclass:`CCrossValidation` and set the number of cross validation's runs.


Please also mention something like "The pipeline instance behaves just like a machine and this can be directly passed to CCrossValidation"

karlnapf · 2018-07-20T19:09:43Z

+#![create_pipeline]
+
+#![create_cross_validation]
+StratifiedCrossValidationSplitting splitting_strategy(labels_train, 2)


dont we have factories for those things?
I would prefer if all newly added examples would fully use the new api so we dont have to refactor later

we don't have factory for CSplittingStrategy

Would you mind creating one? That should be pretty easy!

@karlnapf since there are multiple subclasses of CSplittingStrategy, are we going to do some string comparison by name?

check the factory machine in factory.h, you just have to add some macro lines
the call should be splitting_strategy("StratifiedCrossValidationSplitting", labels=labels_train, k=2)

so we need to do initialization in init() method, instead of the constructor
e.g.

shogun/src/shogun/evaluation/StratifiedCrossValidationSplitting.cpp

Line 21 in 1be1dbb

CStratifiedCrossValidationSplitting::CStratifiedCrossValidationSplitting(

this need refactor right?

ah yes, this will be moved into the method that is called from outside, i.e. build_subsets ... putting it into a helper method makes sense

karlnapf · 2018-07-20T19:11:00Z

+Transformer subMean = transformer("PruneVarSubMean")
+Machine svm = machine("LibLinear")
+
+PipelineBuilder builder()


Same here, could you pls use a factory for this. We dont want to use constructors in the examples

could you explain how to use factory here?
we could create a factory PipelineBuilder* pipeline_builder(), or?

yes exactly, though I would just name it pipeline

karlnapf · 2018-07-20T19:11:24Z

+#![create_cross_validation]
+StratifiedCrossValidationSplitting splitting_strategy(labels_train, 2)
+Evaluation evaluation_criteron = evaluation("AccuracyMeasure")
+CrossValidation cross(pipeline, feats_train, labels_train, splitting_strategy, evaluation_criteron, False)


factory if possible (this might be more tricky)

karlnapf

Great example! I made some comments

karlnapf · 2018-07-23T10:45:09Z

Let me know if the factory creating worked....

karlnapf · 2018-07-23T12:38:44Z

 Machine svm = machine("LibLinear")

-PipelineBuilder builder()
+PipelineBuilder builder = pipeline()


exactly like this! :)

BTW why is the C++ ype not just called Pipeline and the Machine called PipelineMachine?

i think Pipeline and PipelineMachine might be a bit confusing, while PipelineBuilder can indicate its usage as a builder

TBH, I dont really agree, I think then should actually return CMachine (since the subsequent object will be used in this fashion) ... why do we need to know that a machine is a pipeline if it behaves as a machine?
And then Pipeline is the thing that builds the stuff.
Makes for a cleaner API imo

@vigsterkr @lisitsyn @iglesias what are your thoughts?

@karlnapf are we talking now about PipelineMachine and PipelineBuilder, or why then returns Pipeline* ? :) imo those are different things or?

So CPipeline -> CPipelineMachine, CPipelineBuilder ->CPipeline, then returns CMachine

imo builder is just clearer but i dont have any strong feelings about it...
note my second comment about extraction and observation of pipeline.

yes the stages thing is a problem

i.e. until we cannot register and cleanly extract stages from a pipeline this sort of explicit exposure is required :) otherwise the whole thing becomes totally opaque once built

got it! thx for clarifying.
I still would slightly prefer different names (CPipelineMachine, CPipeline) but it is only a minor difference, also no strong feelings. Can leave it as it is

karlnapf · 2018-07-23T16:57:32Z

rebase and merge :)

iglesias · 2018-07-26T16:26:49Z

+
+.. sgexample:: cross_validation_pipeline:create_features
+
+We first chain all transformers, and then finalize the pipeline with the classifier.


Is it one as said above? Then change or remove "all".

iglesias · 2018-07-26T16:27:38Z

+Labels labels_test = labels(f_labels_test)
+#![create_features]
+
+#![create_pipeline


iglesias · 2018-07-26T16:29:21Z

+
+PipelineBuilder builder = pipeline()
+builder.over(subMean)
+Pipeline pipeline = builder.then(svm)


Why does the API contain these two builder and pipeline concepts? What about just a pipeline where steps are added (and of course the order of addition matters).

the idea is to separate construction of pipeline into a single class so that the pipeline object is immutable, and then we don't need to verify elements of pipeline everytime

This is pretty much the same thing that got me confused about the builder vs the pipeline.

the second one, Pipeline can be Machine here because that's enough, we use it as machine in xval. however, if we want to get elements in the pipeline, we still need Pipeline type for now

what about adding a factory for it? So users can call
PipelineMachine(machine)?
Alternatively, for the c++ folks, there is as

I understand that there might be mutability reason justifying two types. Why both part of the interface though?

vigsterkr · 2018-07-27T09:31:04Z

We always have these multipurpose classes where multiple functionality/concept is crammed into one class... here’s a clear cut between a builder pattern and cmachine

…

On 27 Jul 2018, at 11:20, Heiko Strathmann ***@***.***> wrote: @karlnapf commented on this pull request. In examples/meta/src/evaluation/cross_validation_pipeline.sg: > +File f_labels_test = csv_file("../../data/classifier_binary_2d_linear_labels_test.dat") + +#![create_features] +Features feats_train = features(f_feats_train) +Features feats_test = features(f_feats_test) +Labels labels_train = labels(f_labels_train) +Labels labels_test = labels(f_labels_test) +#![create_features] + +#![create_pipeline +Transformer subMean = transformer("PruneVarSubMean") +Machine svm = machine("LibLinear") + +PipelineBuilder builder = pipeline() +builder.over(subMean) +Pipeline pipeline = builder.then(svm) what about adding a factory for it? So users can call PipelineMachine(machine)? Alternatively, for the c++ folks, there is as — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

vigsterkr · 2018-07-27T12:40:07Z

You need the builder for building, but u would like to query a trained pipeline for various stages of it as well. Or what do you mean?

…

On 27 Jul 2018, at 14:35, Fernando J. Iglesias García ***@***.***> wrote: @iglesias commented on this pull request. In examples/meta/src/evaluation/cross_validation_pipeline.sg: > +File f_labels_test = csv_file("../../data/classifier_binary_2d_linear_labels_test.dat") + +#![create_features] +Features feats_train = features(f_feats_train) +Features feats_test = features(f_feats_test) +Labels labels_train = labels(f_labels_train) +Labels labels_test = labels(f_labels_test) +#![create_features] + +#![create_pipeline +Transformer subMean = transformer("PruneVarSubMean") +Machine svm = machine("LibLinear") + +PipelineBuilder builder = pipeline() +builder.over(subMean) +Pipeline pipeline = builder.then(svm) I understand that there might be mutability reason justifying two types. Why both part of the interface though? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

iglesias · 2018-07-27T20:04:01Z

What I was wondering was why a user needs to use two different classes to use a pipeline: e.g. the pipeline_builder to build/construct a pipeline object, and then this pipeline object to do all the preprocessing, cross-validation, etc, etc, she wants. Of course I also understand your point. It is not clean if a class implements many different concepts altogether. It gets confusing quickly, classes get bloated, etc. It is just a trade-off I think. To clarify :-) These were not comments critizing any design choice. Completely far from that! I asked to gain understanding.

…

On Fri, 27 Jul 2018 at 14:40, Viktor Gal ***@***.***> wrote: You need the builder for building, but u would like to query a trained pipeline for various stages of it as well. Or what do you mean? > On 27 Jul 2018, at 14:35, Fernando J. Iglesias García < ***@***.***> wrote: > > @iglesias commented on this pull request. > > In examples/meta/src/evaluation/cross_validation_pipeline.sg: > > > +File f_labels_test = csv_file("../../data/classifier_binary_2d_linear_labels_test.dat") > + > +#![create_features] > +Features feats_train = features(f_feats_train) > +Features feats_test = features(f_feats_test) > +Labels labels_train = labels(f_labels_train) > +Labels labels_test = labels(f_labels_test) > +#![create_features] > + > +#![create_pipeline > +Transformer subMean = transformer("PruneVarSubMean") > +Machine svm = machine("LibLinear") > + > +PipelineBuilder builder = pipeline() > +builder.over(subMean) > +Pipeline pipeline = builder.then(svm) > I understand that there might be mutability reason justifying two types. Why both part of the interface though? > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub, or mute the thread. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#4380 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABGrdgioF75DNVdSzkqZ7SjMgFAtjxv8ks5uKwoygaJpZM4VYFzE> .

karlnapf · 2018-07-30T13:22:50Z

+
+PipelineBuilder builder = pipeline()
+builder.over(subMean)
+Machine pipeline = builder.then(svm)


does this overloading of var name and the factory work? Just asking

yes this overload the factory name, the factory works in this case. thanks for letting me know.

I like the flow of this, so good to merge from my side

karlnapf · 2018-08-08T17:55:59Z

Can you rebase data and then we can merge this as well!

karlnapf · 2018-11-12T13:24:47Z

Shall we get this in soon? :)

vinx13 · 2018-11-12T14:14:51Z

@karlnapf As I can remember, previously we faced the choice on whether pipeline builder should return CMachine* or CPipeline*. The first choice, returning CMachine* and then providing a pipeline factory, works though being ugly. According to @vigsterkr 's idea, in my last commit I tried to make CPipeline as base class, but got some different errors on travis

what():  �[1;31m[ERROR]�[0m In file /opt/shogun/src/shogun/base/SGObject.h line 369: Cannot put parameter CrossValidation::machine of type shogun::CMachine*, incompatible provided type shogun::CPipeline*.

Maybe @lisitsyn has idea?

karlnapf · 2018-11-12T17:32:24Z

Ah I remember now.
Well the real question is whether Pipeline should be part of the modular interfaces (i.e. the base types) or not.

karlnapf

Cool that it works. I like that this is a general solution to the problem, despite being a not super elegant.

Any further thoughts on this idea of implicitly casting a pipeline to CMachine via an argument of a factory? @lisitsyn @vigsterkr

karlnapf · 2019-01-29T11:01:05Z

            "get_real_vector": "$object.get($arguments)",
-            "get_real_matrix": "$object.get($arguments)"
+            "get_real_matrix": "$object.get($arguments)",
+            "put_machine": "$object.put($arguments)"


this will not work as we cannot extract the arguments yet and you would need to pass the second argument to the machine factory

we can do this once #4490 is solved

karlnapf · 2019-01-29T11:03:17Z

+
+PipelineBuilder builder = pipeline()
+builder.over(subMean)
+Machine pipeline = builder.then(svm)


This type here should be "Pipeline" as that is what the builder returns (and now Pipeline is part of swig as well)

karlnapf · 2019-01-29T11:04:42Z

+SplittingStrategy strategy = splitting_strategy("StratifiedCrossValidationSplitting", labels=labels_train, num_subsets=2)
+Evaluation evaluation_criterion = evaluation("AccuracyMeasure")
+MachineEvaluation cross = machine_evaluation("CrossValidation", features=feats_train, labels=labels_train, splitting_strategy=strategy, evaluation_criterion=evaluation_criterion, autolock=False, num_runs=2)
+cross.put_machine("machine", pipeline)


change to cross.put(machine(pipeline)) and it should work
we can change that to put_machine later once #4490 is in

karlnapf · 2019-01-29T11:09:12Z

+#![create_cross_validation]
+SplittingStrategy strategy = splitting_strategy("StratifiedCrossValidationSplitting", labels=labels_train, num_subsets=2)
+Evaluation evaluation_criterion = evaluation("AccuracyMeasure")
+MachineEvaluation cross = machine_evaluation("CrossValidation", features=feats_train, labels=labels_train, splitting_strategy=strategy, evaluation_criterion=evaluation_criterion, autolock=False, num_runs=2)


OT: Can I suggest that we don't provide features and labels to crossvalidation but instead as parameters of CMachineEvaluation::evaluate(CFeatures*, CLabels*) ? Different PR though
@lisitsyn @vigsterkr @iglesias

Hey mate, I can't say much based on this snippet. What's your point? Also based on the last line, CrossValidation here seems to be a type of MachineEvaluation, no?

I’d like to pass it as function arguments if the evaluation function rather than as fields before that....

karlnapf

;)

karlnapf · 2019-01-30T07:26:47Z

Great that this now works !
Shall we address the other comments and then finally merge this?

karlnapf

I think only some minor things are left

karlnapf · 2019-01-30T07:32:17Z

    """
    _obj = getattr(sys.modules[__name__], object_name)
-    def _internal_factory(name, **kwargs):
+    def _internal_factory(name, *args, **kwargs):


Not needed abymore

karlnapf · 2019-01-30T07:33:33Z

 	{
 		return get_machine()->get_machine_problem_type();
 	}
+


Can’t this be in factory?

karlnapf · 2019-01-30T07:34:16Z

-	                    ->then(machine);
+	                    ->then(machine)
+	                    ->as<CPipeline>();



Is that needed?

This reverts commit 3efa0cd.

This reverts commit d2a3a74.

karlnapf · 2019-01-31T10:42:48Z

~~Something about include paths is broken in the example ...~~ sorry

karlnapf · 2019-01-31T10:44:33Z

@@ -0,0 +1,33 @@
+============================


minor. the ==== should be as long as the text

karlnapf · 2019-01-31T10:46:15Z

+We also have to decide on an evaluation criterion class (see :sgclass:`CEvaluation`) to evaluate the performance of the trained model.
+In this case, we use :sgclass:`CAccuracyMeasure`.
+We then instantiate :sgclass:`CCrossValidation` and set the number of cross validation's runs.
+The pipeline instance behaves just like a :sgclass:`CMachine` and this can be directly passed to :sgclass:`CCrossValidation`.


This is not exactly true anymore, as we need to call the factory now. I suggest we just avoid this and say
"We next create a cross-validation instance and pass the generated pipeline."

karlnapf · 2019-01-31T10:49:29Z

+	 * See also CPipelineBuilder and CPipeline.
+	 * @return new instance of CPipelineBuilder
+	 */
+	CPipelineBuilder* pipeline()


@gf712 would you be up for making the definitions of the factory methods a bit neater? :)

sure! what do you have in mind?

Actually, just checked, not sure factory.h can even be made that much simpler?

karlnapf · 2019-01-31T10:50:49Z

Cool!
Thanks for the big amount of work. It is awesome to have this in now :)

* Add meta example and cookbook of pipeline cross validation

karlnapf reviewed Jul 20, 2018

View reviewed changes

karlnapf requested changes Jul 20, 2018

View reviewed changes

karlnapf reviewed Jul 23, 2018

View reviewed changes

iglesias requested changes Jul 26, 2018

View reviewed changes

vinx13 force-pushed the cookbook/pipeline_xval branch from 0e3b84b to f8120b9 Compare July 30, 2018 12:22

karlnapf reviewed Jul 30, 2018

View reviewed changes

karlnapf approved these changes Jul 30, 2018

View reviewed changes

vinx13 force-pushed the cookbook/pipeline_xval branch from f8120b9 to 5bd958c Compare August 8, 2018 13:15

vinx13 force-pushed the cookbook/pipeline_xval branch 2 times, most recently from 7651aa9 to 8752675 Compare August 14, 2018 12:00

karlnapf reviewed Jan 26, 2019

View reviewed changes

vinx13 force-pushed the cookbook/pipeline_xval branch from 6fe7b65 to 7a00f6e Compare January 29, 2019 09:22

Try using put_machine

d2a3a74

vinx13 force-pushed the cookbook/pipeline_xval branch from 7a00f6e to d2a3a74 Compare January 29, 2019 09:25

karlnapf reviewed Jan 29, 2019

View reviewed changes

vinx13 added 3 commits January 30, 2019 11:41

machine factory from pipeline

dc485b3

Specialize put<CPipeline>

3efa0cd

Fix factory

5ff3ac4

karlnapf reviewed Jan 30, 2019

View reviewed changes

karlnapf requested changes Jan 30, 2019

View reviewed changes

vinx13 added 4 commits January 30, 2019 23:43

Revert "Specialize put<CPipeline>"

0f30ac3

This reverts commit 3efa0cd.

Revert "Try using put_machine"

6ab96b8

This reverts commit d2a3a74.

Clean up

ebcdf51

Fix

736d5b5

karlnapf reviewed Jan 31, 2019

View reviewed changes

Update cross_validation_pipeline.rst

811e5cc

karlnapf reviewed Jan 31, 2019

View reviewed changes

Update cross_validation_pipeline.rst

662d5fa

Update cross_validation_pipeline.rst

41d06be

karlnapf merged commit ab0aa3c into shogun-toolbox:develop Jan 31, 2019

gf712 mentioned this pull request Jan 31, 2019

Centralise type definitions #4489

Merged

vigsterkr pushed a commit to vigsterkr/shogun that referenced this pull request Mar 9, 2019

Add cookbook of cross validation with pipeline (shogun-toolbox#4380)

a87d742

* Add meta example and cookbook of pipeline cross validation

ktiefe pushed a commit to ktiefe/shogun that referenced this pull request Jul 30, 2019

Add cookbook of cross validation with pipeline (shogun-toolbox#4380)

92a307c

* Add meta example and cookbook of pipeline cross validation

		@@ -0,0 +1,34 @@
		============================
		Cross Validation on Pipeline

		.. sgexample:: cross_validation_pipeline:create_features


		We use :sgclass:`CPruneVarSubMean` to normalize the features and then use :sgclass:`CLibLinear` for classification.



		We use :sgclass:`CPruneVarSubMean` to normalize the features and then use :sgclass:`CLibLinear` for classification.
		The transformer and the machine are chained as a :sgclass:`CPipeline`.


		.. sgexample:: cross_validation_pipeline:create_features

		We first chain all transformers, and then finalize the pipeline with the classifier.

Uh oh!

Uh oh!

Conversation

vinx13 commented Jul 20, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

karlnapf Jul 23, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

karlnapf left a comment

Choose a reason for hiding this comment

Uh oh!

karlnapf commented Jul 23, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

karlnapf commented Jul 23, 2018

Uh oh!

karlnapf Jul 23, 2018 •

edited

Loading