iterative machine unit test #4370

shubham808 · 2018-07-13T13:02:52Z

Tests that train 3 models:

Complete train till convergence
Train with max number of iteration set low at x
Train a model then prematurely stop it at x iterations, the continue training till convergence

The results are compared to ensure proper state update in every iteration.
Eventually we would want this done automatically for all Iterative Machines.

karlnapf · 2018-07-13T13:05:34Z

tests/unit/classifier/svm/NewtonSVM_unittest.cc

+	auto svm_one = some<CNewtonSVM>();
+	svm_one->set_labels(labels);
+
+	// trained only till 5 iteraetions


karlnapf · 2018-07-15T22:23:37Z

tests/unit/classifier/svm/NewtonSVM_unittest.cc

+* Redistribution and use in source and binary forms, with or without
+* modification, are permitted provided that the following conditions are met:
+*
+* 1. Redistributions of source code must retain the above copyright notice, this


could you pls use the short version

karlnapf · 2018-07-15T22:24:16Z

tests/unit/classifier/svm/NewtonSVM_unittest.cc

+	SG_UNREF(results_one);
+	SG_UNREF(results_stop);
+	SG_UNREF(results);
+	SG_UNREF(results_complete);


no memory leaks/erriors in here?

there should not be since i only introduced bias change but i will check again

karlnapf · 2018-07-15T22:25:12Z

tests/unit/classifier/svm/NewtonSVM_unittest.cc

+	auto results_complete = svm_stop->apply_binary(test_features);
+
+	// compare model with completely trained reference
+	EXPECT_TRUE(results_complete->get_labels().equals(results->get_labels()));


you could actually call the clone method of SGObject here, then you do not need the get_labels() and can also use apply rather than apply_binary above

this is still true, also for the above label comparison

results_complete->equals(results)

(and then use apply instead of apply_binary

karlnapf · 2018-07-15T22:25:26Z

tests/unit/classifier/svm/NewtonSVM_unittest.cc

+
+	svm_stop->set_callback(nullptr);
+
+	// continue training till converged


karlnapf · 2018-07-15T22:26:14Z

tests/unit/classifier/svm/NewtonSVM_unittest.cc

+	EXPECT_TRUE(svm->train(features));
+	auto results = wrap(svm->apply(test_features));
+	auto acc = some<CAccuracyMeasure>();
+	EXPECT_EQ(acc->evaluate(results, test_labels), 1.0);


This is not the best unit test.
Better would be to compare the solver against liblinear or some reference implementation of the newton svm

karlnapf · 2018-08-04T10:06:28Z

tests/unit/classifier/Perceptron_unittest.cc

+	auto perceptron_one = some<CPerceptron>();
+	perceptron_one->set_labels(labels);
+	perceptron_one->put<int32_t>("max_iterations", 1);
+	perceptron_one->set_compute_bias(false);


this can be removed

karlnapf · 2018-08-05T20:25:51Z

tests/unit/classifier/Perceptron_unittest.cc

@@ -122,13 +122,24 @@ TEST(Perceptron, continue_training_consistency)

 	ASSERT(!perceptron_stop->is_complete());

+	auto perceptron_one = some<CPerceptron>();


karlnapf · 2018-08-05T20:26:14Z

tests/unit/classifier/svm/NewtonSVM_unittest.cc

+#include <shogun/lib/Signal.h>
+
+#include "environments/LinearTestEnvironment.h"
+#include <gtest/gtest.h>


pls include gtest.h first, they use some magic macros that cause problems with some compilers otherwise

karlnapf · 2018-08-05T20:26:26Z

tests/unit/classifier/svm/NewtonSVM_unittest.cc

+	svm->set_labels(labels);
+	svm->train(features);
+
+	// reference for completly trained model


typo in comment

shubham808 · 2018-11-21T19:09:14Z

@karlnapf this solves the problem for now :) The CI seems to be acting weird but :/

karlnapf · 2018-11-22T20:36:51Z

src/shogun/machine/IterativeMachine.h

-
+
+		/** Stores features to continue training */
+		CDotFeatures* m_continue_features;


I think this would need to be CFeatures*, or? Why would we restrict to CDotFeatures?
Sorry it has been a while, we might have discussed this before ;)

It should be CFeatures* but then we will need to check feature type in each iteration... for now we are only using IterativeMachine in LinearMachines which are DotFeatures.

I think we could get away with a static cast and rely on the fact that the algorithm knows the feature type is correct (inside iteration), since it was checked before the algorithm started (by your last piece of gsoc work)

karlnapf · 2018-11-22T20:39:32Z

src/shogun/machine/IterativeMachine.h

@@ -32,7 +32,8 @@ namespace shogun
 		{
 			m_current_iteration = 0;
 			m_complete = false;
-
+			m_continue_features = NULL;


karlnapf · 2018-11-22T20:42:35Z

src/shogun/machine/IterativeMachine.h

@@ -57,7 +59,8 @@ namespace shogun
 		virtual bool continue_train()
 		{
 			this->reset_computation_variables();
-


let me understand this again. Is continue_train called from train? I.e. can we make sure the continure features are set on training start?

Or where exactly are those set?

yes it is called from within train_machine

karlnapf

What was the problem with this thing before?

Travis is currently down but will soon be up again, to be replaced by a microsoft service ... Stay tuned :)

shubham808 · 2018-12-07T08:14:53Z

The problem was we needed to make sure features do not change when we resume training. We have locally stored the features here that we will not be changed which solves our problem.

karlnapf · 2018-12-07T12:36:19Z

ok cool, so let's change the type to generic features and then merge this and move on ! :)

karlnapf · 2018-12-11T12:32:58Z

Ok cool
PR CI builds are currently not active, so you will need to check all tests locally
If you give your ok we can merge (and then closely look at the CI and buildbot outputs)
Shall I? :)

shubham808 · 2018-12-11T12:45:09Z

alright ! lets merge it

karlnapf · 2018-12-11T17:35:37Z

Ok pls have an eye on the buoldbot and the new azure builds

karlnapf · 2018-12-11T18:08:37Z

azure seems fine: https://dev.azure.com/shogunml/shogun/_build?definitionId=2

karlnapf · 2018-12-11T18:11:43Z

Cool!
Now what was next? :)

shubham808 · 2018-12-12T10:01:54Z

now lets look into the automated tests for this

karlnapf · 2018-12-12T17:46:58Z

I have a local branch with an initial attempt on this, but not complete. Will share it

* m_continue_features and iterative machine tests

karlnapf reviewed Jul 13, 2018

View reviewed changes

karlnapf reviewed Jul 15, 2018

View reviewed changes

tests/unit/classifier/svm/NewtonSVM_unittest.cc Outdated

svm_stop->set_callback(nullptr);

// continue training till converged

Copy link

Member

karlnapf Jul 15, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

until

karlnapf reviewed Jul 15, 2018

View reviewed changes

karlnapf reviewed Aug 4, 2018

View reviewed changes

shubham808 force-pushed the feature/iterative_machine_tests branch from 7fa47df to 65a07f2 Compare August 5, 2018 13:28

karlnapf reviewed Aug 5, 2018

View reviewed changes

shubham808 force-pushed the feature/iterative_machine_tests branch from 65a07f2 to cf9737a Compare November 18, 2018 04:05

karlnapf reviewed Nov 22, 2018

View reviewed changes

m_continue_features and iterative machine tests

3822471

shubham808 force-pushed the feature/iterative_machine_tests branch from cf9737a to ed74e6d Compare December 9, 2018 09:48

use generic features

e174d4b

shubham808 force-pushed the feature/iterative_machine_tests branch from ed74e6d to e174d4b Compare December 9, 2018 14:06

fix include order

d2e1c39

karlnapf merged commit e18d0a5 into shogun-toolbox:develop Dec 11, 2018

vigsterkr pushed a commit to vigsterkr/shogun that referenced this pull request Mar 9, 2019

iterative machine unit test (shogun-toolbox#4370)

eac542d

* m_continue_features and iterative machine tests

ktiefe pushed a commit to ktiefe/shogun that referenced this pull request Jul 30, 2019

iterative machine unit test (shogun-toolbox#4370)

2886332

* m_continue_features and iterative machine tests

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

iterative machine unit test #4370

iterative machine unit test #4370

shubham808 commented Jul 13, 2018

karlnapf Jul 13, 2018

karlnapf Jul 15, 2018

karlnapf Jul 15, 2018

karlnapf Aug 5, 2018

shubham808 Aug 5, 2018

karlnapf Jul 15, 2018

karlnapf Aug 5, 2018

karlnapf Aug 6, 2018

karlnapf Aug 6, 2018

karlnapf Jul 15, 2018

karlnapf Jul 15, 2018

karlnapf Aug 4, 2018

karlnapf Aug 5, 2018

karlnapf Aug 5, 2018

karlnapf Aug 5, 2018

shubham808 commented Nov 21, 2018

karlnapf Nov 22, 2018

shubham808 Dec 7, 2018

karlnapf Dec 7, 2018

karlnapf Nov 22, 2018

karlnapf Nov 22, 2018

karlnapf Nov 22, 2018

shubham808 Dec 7, 2018

karlnapf Dec 7, 2018

karlnapf left a comment

shubham808 commented Dec 7, 2018

karlnapf commented Dec 7, 2018

karlnapf commented Dec 11, 2018

shubham808 commented Dec 11, 2018

karlnapf commented Dec 11, 2018

karlnapf commented Dec 11, 2018

karlnapf commented Dec 11, 2018

shubham808 commented Dec 12, 2018

karlnapf commented Dec 12, 2018


		svm_stop->set_callback(nullptr);

		// continue training till converged

		@@ -122,13 +122,24 @@ TEST(Perceptron, continue_training_consistency)

		ASSERT(!perceptron_stop->is_complete());

		auto perceptron_one = some<CPerceptron>();



		/** Stores features to continue training */
		CDotFeatures* m_continue_features;

iterative machine unit test #4370

iterative machine unit test #4370

Conversation

shubham808 commented Jul 13, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shubham808 commented Nov 21, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

karlnapf left a comment

Choose a reason for hiding this comment

shubham808 commented Dec 7, 2018

karlnapf commented Dec 7, 2018

karlnapf commented Dec 11, 2018

shubham808 commented Dec 11, 2018

karlnapf commented Dec 11, 2018

karlnapf commented Dec 11, 2018

karlnapf commented Dec 11, 2018

shubham808 commented Dec 12, 2018

karlnapf commented Dec 12, 2018