Add pipeline cookbook #4323

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Merged

vigsterkr merged 7 commits into shogun-toolbox:feature/transformers from vinx13:meta/pipeline

Jun 28, 2018

Member

vinx13 commented Jun 2, 2018 •

edited

Loading

add pipeline meta example and cookbook
add transformer factory

vinx13 mentioned this pull request

Add pipeline integraton data shogun-toolbox/shogun-data#157

Merged

karlnapf reviewed

View reviewed changes

doc/cookbook/source/examples/pipeline/pipeline.rst Outdated

+              Pipeline
+              ========
+              Pipeline is a machine that chains multiple transformers and machines. It consists of a sequence of transformers as intermediate stages and a machine as the final stage. Features are transformed by transformers and fed into the next stage sequentially.

Member

karlnapf Jun 4, 2018

maybe add sgclass to link to the API docs

karlnapf reviewed

View reviewed changes

doc/cookbook/source/examples/pipeline/pipeline.rst Outdated

+              Pipeline
+              ========
+              Pipeline is a machine that chains multiple transformers and machines. It consists of a sequence of transformers as intermediate stages and a machine as the final stage. Features are transformed by transformers and fed into the next stage sequentially.

Member

karlnapf Jun 4, 2018

could you start every sentence in a new line? I.e. newline after each ".", this makes subsequent diffs way easier to review

karlnapf reviewed

View reviewed changes

doc/cookbook/source/examples/pipeline/pipeline.rst Outdated

+              Pipeline
+              ========
+              Pipeline is a machine that chains multiple transformers and machines. It consists of a sequence of transformers as intermediate stages and a machine as the final stage. Features are transformed by transformers and fed into the next stage sequentially.

Member

karlnapf Jun 4, 2018

"stage" of what? Maybe add "stage of training"

karlnapf reviewed

View reviewed changes

doc/cookbook/source/index.rst

+              .. toctree::
+                  :maxdepth: 1
+                  :glob:

Member

karlnapf Jun 4, 2018

+1

karlnapf reviewed

View reviewed changes

examples/meta/src/pipeline/pipeline.sg

+              # additional integration variables
+              #![extract_centers_and_radius]
+              RealMatrix c = kmeans.get_cluster_centers()

Member

karlnapf Jun 4, 2018

can this be done with the generic get?

Member Author

vinx13 Jun 4, 2018

no, this is not a parameter :(

Member

karlnapf Jun 4, 2018

I see.
Ok, one of the next things I want to do is to register callback functions with get so that we can lazily evaluate things, like in this case

karlnapf reviewed

View reviewed changes

src/shogun/machine/Pipeline.cpp Outdated

+              		REQUIRE(
+              		    m_stages.empty() ||
+              		        holds_alternative<CTransformer*>(m_stages.back()),
+              		    "Transformers can not be placed after machines.\n");

Member

karlnapf Jun 4, 2018

Could you maybe make this "Last element in pipeline is %s", and then put the machine name there?

karlnapf reviewed

View reviewed changes

src/shogun/machine/Pipeline.cpp Outdated

+              		REQUIRE(
+              		    m_stages.empty() ||
+              		        holds_alternative<CTransformer*>(m_stages.back()),
+              		    "Multiple machines are added to pipeline.\n");

Member

karlnapf Jun 4, 2018

same story, pls print the name of the last added machine so the user knows what she did wrong

karlnapf reviewed

View reviewed changes

src/shogun/machine/Pipeline.cpp Outdated

+              	{
+              		REQUIRE(
+              		    !m_stages.empty() && holds_alternative<CMachine*>(m_stages.back()),
+              		    "Machine has not been added.\n");

Member

karlnapf Jun 4, 2018

Could you be more elaborate here as well
"Pipline cannot be trained without an added machine. Last element is %s"

karlnapf reviewed

View reviewed changes

src/shogun/machine/Pipeline.cpp Outdated

+              	CLabels* CPipeline::apply(CFeatures* data)
+              	{
+              		REQUIRE(
+              		    !m_stages.empty() && holds_alternative<CMachine*>(m_stages.back()),

Member

karlnapf Jun 4, 2018

as above. user facing error messages need to be more explicit

karlnapf reviewed

View reviewed changes

src/shogun/machine/Pipeline.cpp Outdated

+              	{
+              		REQUIRE(
+              		    !m_stages.empty() && index < m_stages.size() - 1,
+              		    "Index out of bound. There are only %d transformers.\n",

Member

karlnapf Jun 4, 2018

print the index as well
"Requested index (%d) out of bounds " ...

karlnapf reviewed

View reviewed changes

src/shogun/machine/Pipeline.h Outdated

+              namespace shogun
+              {
+              	class CPipeline : public CMachine

Member

karlnapf Jun 4, 2018

Could you write some minimal class docs?

karlnapf reviewed

View reviewed changes

tests/unit/machine/Pipeline_unittest.cc Outdated

+              	EXPECT_EQ(pipeline->get_transformer(1), transformer2.get());
+              	EXPECT_EQ(pipeline->get_machine(), machine.get());
+              }

Member

karlnapf Jun 4, 2018

great set of tests!

karlnapf reviewed

View reviewed changes

Member

karlnapf left a comment

Great work!
I made some comments on user facing parts of this

vigsterkr force-pushed the feature/transformers branch 2 times, most recently from b7d2ff9 to f211b65 Compare

June 8, 2018 14:15

vigsterkr force-pushed the feature/transformers branch from f211b65 to 0fa4faa Compare

June 11, 2018 18:30

vinx13 added 2 commits

June 13, 2018 18:23


          Add transformer factory

f729cf0


          Add pipeline cookbook

c6e65ad

vinx13 force-pushed the meta/pipeline branch from f8e42ff to c6e65ad Compare

June 13, 2018 10:24


          Update integration data

fd85b08

vigsterkr requested changes

View reviewed changes

examples/meta/src/pipeline/pipeline.sg Outdated

+              #![create_pipeline]
+              Pipeline pipeline()
+              pipeline.with(subMean)

Member

vigsterkr Jun 13, 2018

this needs fixing, because of reserved keyword

examples/meta/src/pipeline/pipeline.sg Outdated

+              #![create_pipeline]
+              Pipeline pipeline()
+              pipeline.with(subMean)
+              pipeline.with(pca)

Member

vigsterkr Jun 13, 2018

change it

vinx13 added 3 commits

June 14, 2018 14:03


          Update pipeline cookbook to new api

dab5513


          Fix typo

a8374ce


          Init random with fixed seed

c7957b0

Member

vigsterkr commented Jun 14, 2018

pipeline.cs(31,1): error CS0246: The type or namespace name `PipelineBuilder' could not be found. Are you missing an assembly reference?
pipeline.cs(32,1): error CS0841: A local variable `builder' cannot be used before it is declared
pipeline.cs(33,1): error CS0841: A local variable `builder' cannot be used before it is declared
pipeline.cs(34,1): error CS0246: The type or namespace name `Pipeline' could not be found. Are you missing an assembly reference?
pipeline.cs(38,1): error CS0841: A local variable `pipeline' cannot be used before it is declared
pipeline.cs(39,25): error CS0841: A local variable `pipeline' cannot be used before it is declared
Compilation failed: 6 error(s), 0 warnings```

Member

vigsterkr commented Jun 14, 2018

and in java

[ 97%] Generating pipeline/pipeline.class
/opt/shogun/build/examples/meta/java/pipeline/pipeline.java:12: error: cannot find symbol
import org.shogun.Pipeline;
                 ^
  symbol:   class Pipeline
  location: package org.shogun
/opt/shogun/build/examples/meta/java/pipeline/pipeline.java:13: error: cannot find symbol
import org.shogun.PipelineBuilder;
                 ^
  symbol:   class PipelineBuilder
  location: package org.shogun
/opt/shogun/build/examples/meta/java/pipeline/pipeline.java:51: error: cannot find symbol
PipelineBuilder builder = new PipelineBuilder();
^
  symbol:   class PipelineBuilder
  location: class pipeline
/opt/shogun/build/examples/meta/java/pipeline/pipeline.java:51: error: cannot find symbol
PipelineBuilder builder = new PipelineBuilder();
                              ^
  symbol:   class PipelineBuilder
  location: class pipeline
/opt/shogun/build/examples/meta/java/pipeline/pipeline.java:54: error: cannot find symbol
Pipeline pipeline = builder.then(kmeans);
^
  symbol:   class Pipeline
  location: class pipeline
5 errors
examples/meta/java/CMakeFiles/java-pipeline-pipeline.dir/build.make:62: recipe for target 'examples/meta/java/pipeline/pipeline.class' failed
make[2]: *** [examples/meta/java/pipeline/pipeline.class] Error 1
CMakeFiles/Makefile2:5049: recipe for target 'examples/meta/java/CMakeFiles/java-pipeline-pipeline.dir/all' failed
make[1]: *** [examples/meta/java/CMakeFiles/java-pipeline-pipeline.dir/all] Error 2
Makefile:160: recipe for target 'all' failed
make: *** [all] Error 2```


          Fix swig include

56acfd9

vigsterkr merged commit 550639e into shogun-toolbox:feature/transformers

vigsterkr pushed a commit that referenced this pull request


          Add pipeline cookbook (#4323)

af8af72

* Add transformer factory
* Add pipeline cookbook
* Fix swig include

vigsterkr pushed a commit that referenced this pull request


          Add pipeline cookbook (#4323)

77a9f85

* Add transformer factory
* Add pipeline cookbook
* Fix swig include

vigsterkr pushed a commit that referenced this pull request


          Add pipeline cookbook (#4323)

8711ea6

* Add transformer factory
* Add pipeline cookbook
* Fix swig include

vigsterkr pushed a commit that referenced this pull request


          Add pipeline cookbook (#4323)

1ac4765

* Add transformer factory
* Add pipeline cookbook
* Fix swig include

ktiefe pushed a commit to ktiefe/shogun that referenced this pull request


          Add pipeline cookbook (shogun-toolbox#4323)

f9286b3

* Add transformer factory
* Add pipeline cookbook
* Fix swig include

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet