New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add pipeline #4317
Add pipeline #4317
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
minor stuff
src/shogun/machine/Pipeline.cpp
Outdated
{ | ||
if (holds_alternative<CTransformer*>(stage)) | ||
{ | ||
CTransformer* transformer = shogun::get<CTransformer*>(stage); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
oooootoooo, auto :)
src/shogun/machine/Pipeline.cpp
Outdated
if (holds_alternative<CTransformer*>(stage)) | ||
{ | ||
CTransformer* transformer = shogun::get<CTransformer*>(stage); | ||
if (transformer->train_require_labels()) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this could be a simple ternary expression:
transformer->train_require_labels() ? transformer->fit(data, m_labels) : transformer->fit(data);
src/shogun/machine/Pipeline.cpp
Outdated
} | ||
} | ||
|
||
return NULL; // unreachable |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nullptr
src/shogun/machine/Pipeline.cpp
Outdated
{ | ||
if (holds_alternative<CTransformer*>(stage)) | ||
{ | ||
CTransformer* transformer = shogun::get<CTransformer*>(stage); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
auto
src/shogun/machine/Pipeline.cpp
Outdated
} | ||
else | ||
{ | ||
CMachine* machine = shogun::get<CMachine*>(stage); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
auto
src/shogun/machine/Pipeline.h
Outdated
protected: | ||
virtual bool train_machine(CFeatures* data = NULL) override; | ||
|
||
std::vector<variant<CTransformer*, CMachine*>> m_stages; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
sadly we wont be able to register this for a while :)
i.e. no serialization :D
src/shogun/machine/Pipeline.cpp
Outdated
} | ||
else | ||
{ | ||
CMachine* machine = shogun::get<CMachine*>(stage); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
auto?
@vinx13 would be great to add a simple xval cookbook where we use a CPruneVarSubMean transformer before doing PCA and applying k-means on a toy data :D |
|
aaaah and another thing: getter methods for the pipeline elements would be desirable. i.e. get the trained transformer etc. |
e260a20
to
860a9ed
Compare
src/shogun/machine/Pipeline.cpp
Outdated
return require_labels; | ||
} | ||
|
||
void CPipeline::list_stages() const |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
how about renaming this to
std::string to_string() const;
and basically return a string instead of directly writing SG_INFO.... that of course means that SGObject::to_string needs to be virtual
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I like this more as well! users can then print the strings themselves
src/shogun/machine/Pipeline.cpp
Outdated
} | ||
} | ||
|
||
CTransformer* CPipeline::get_transformer(size_t index) const |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
maybe instead of accessing the pipeline elements like this it'd be good to have a way to access them by name? see for example http://scikit-learn.org/stable/tutorial/text_analytics/working_with_text_data.html#evaluation-of-the-performance-on-the-test-set
of course this would mean to be able to add elements as a tuple, say
Pipline().with(("customName", ZeroMean).with....
but that could be that we support both, meaning either having
Pipline().with(("customName", ZeroMean).with....
and
Pipline().with(ZeroMean).with....
and in the 2nd case we use the get_name() string...
45162b6
to
8ac8703
Compare
src/shogun/machine/Pipeline.cpp
Outdated
void CPipeline::check_pipeline() | ||
{ | ||
REQUIRE(!m_stages.empty(), "Pipeline is empty"); | ||
REQUIRE( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
note, that for now the only way to infer the reason of exception is actually parsing the exception message... that's not really 'nice' from an automation/developer's perspective
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Actually, this was part of the "user experience" GSoC project. So yes makes total sense imo
src/shogun/machine/Pipeline.cpp
Outdated
return shogun::get<CTransformer*>(stage.second); | ||
} | ||
|
||
SG_ERROR("Transformer with name %s not found.\n", name.c_str()); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
imo here std::invalid_argument
be the more appropriate...
but in lot of the REQUIRE()
case it's the same... namely a 'more specific type' of exception would be much more appropriate from a shogun user... if its getting integrated in a larger framework.
pipeline->with(transformer1); | ||
|
||
auto features = some<NiceMock<MockCFeatures>>(); | ||
EXPECT_THROW(pipeline->train(features), ShogunException); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
b7d2ff9
to
f211b65
Compare
src/shogun/lib/ShogunException.h
Outdated
* whenever an object in Shogun in invalid state is used. For example, a machine | ||
* is used before training. | ||
*/ | ||
class InvalidStateException : public ShogunException |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
i would define these in separate headers and cpp files :)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
and in the case of "a machine is used before training" i would call that exception MachineNotTrainedException
as invalid state could be a lot of things and the not trained case i believe should be explicitly spelled out for the user :)
@lisitsyn @karlnapf so as trying to add pipeline to the swig interface it came out that how about changing and another idea here: imo we should have a @vinx13 i would still like to have a builder that supports adding a list of elements. something like: add_stages([transf1, transf1, machine]) |
f211b65
to
0fa4faa
Compare
@karlnapf we are getting travis problems here as well..... this branch is rebased over HEAD od develop, but octave is throwing this error
|
the error also happens on python
|
Add initial pipeline implementation Add InvalidStateException and use it in pipeline Move exception to shogun/lib/exception Add MachineNotTrainedException
Add initial pipeline implementation Add InvalidStateException and use it in pipeline Move exception to shogun/lib/exception Add MachineNotTrainedException
Add initial pipeline implementation Add InvalidStateException and use it in pipeline Move exception to shogun/lib/exception Add MachineNotTrainedException
Add initial pipeline implementation Add InvalidStateException and use it in pipeline Move exception to shogun/lib/exception Add MachineNotTrainedException
No description provided.