Adding ZDT Test Suite #273

jonpsy · 2021-03-16T10:46:18Z

Pizza!

Declared files

TODO

Write docs for each declared files.
Write the interface for each declared files.
Test working against pagmo
Test numerical stability and corner cases & add to test case.
Add in nsga2_test.cpp
Add Pareto Front for each problem suite.

Pareto Fronts plotting: https://colab.research.google.com/drive/1Hgmvj4ZQVdT6OgopLPabldeyaOm0DInI?usp=sharing

jonpsy · 2021-03-16T19:11:53Z

All this in just one day! Phew!

@rcurtin, @zoq If you find the time, please review it :) . Meanwhile, I'll test try and make them more robust to corner cases and try and optimize it. Cheers!

jonpsy · 2021-03-19T15:33:43Z

Few updates:
a) I've tested all the ZDT(1-6 excluding 5) against pagmo, results match. I can create a dummy repository if one wants to verify this.
b) Interestingly, NSGA-II (current implementation) fails on this, but #263 passes, mostly owing to the population being out-of-bounds.
c) I think this is good to merge, critique's welcome.

zoq

Looks pretty good to me, I made some comments about the style, I'll check the function implementation next, the optimizer is already able to solve the problems, so that is a good sign.

include/ensmallen_bits/problems/zdt1_function.hpp

zoq · 2021-03-24T00:56:44Z

include/ensmallen_bits/problems/zdt1_function.hpp

+
+      return arma::Col<ElemType>(numVariables, 1, arma::fill::zeros);
+    }
+


Might be a good idea to add a comment to ObjectiveF1, ObjectiveF2.

include/ensmallen_bits/problems/zdt2_function.hpp

include/ensmallen_bits/problems/zdt3_function.hpp

jonpsy · 2021-03-24T07:52:48Z

Looks pretty good to me, I made some comments about the style, I'll check the function implementation next, the optimizer is already able to solve the problems, so that is a good sign.

Just to make your job easier, I've created a repository comparing pagmo results to mine. Ofcourse, these are trivial cases, if you have any edge cases in mind do let me know so I can test and compare them. Cheers!

zoq · 2021-03-25T23:42:42Z

Looks pretty good to me, I made some comments about the style, I'll check the function implementation next, the optimizer is already able to solve the problems, so that is a good sign.

Just to make your job easier, I've created a repository comparing pagmo results to mine. Ofcourse, these are trivial cases, if you have any edge cases in mind do let me know so I can test and compare them. Cheers!

Usually I take a look at the function definition (paper, wikipedia) and compare it with the implementation, since there is a chance that an existing implementation does something different; but it's a nice basis.

jonpsy · 2021-03-25T23:52:29Z

I see. Here's the link to the research paper: https://www.mitpressjournals.org/doi/abs/10.1162/106365600568202

jonpsy · 2021-04-11T23:46:55Z

Now that NSGA-II fix is close to merge, we can start working on this. ZDT5 is also remaining, but since it only works on bitstring type I would need to use std::enable_if < is_bitset<....>>, something along these lines.

=> Documentation done. => Declared the class.

1) ZDT1 2) ZDT2 3) ZDT3 4) ZDT4 5) ZDT6

=> Indent fix

Co-authored-by: Marcus Edel <marcus.edel@fu-berlin.de>

jonpsy · 2021-04-18T17:43:06Z

Keep open

jonpsy · 2021-05-01T09:46:52Z

Updates on this: pymoo has implemented the Pareto front for each of these so I have included the Ipynb notebook in this pull request. I was thinking we could test if the Pareto Front from the Pymoo implementation is the same as ours, see if the outputs match? Your thoughts @zoq ?

zoq · 2021-05-25T18:33:52Z

Looking at the Travis output, this is an easy issue to fix.

jonpsy · 2021-05-26T16:25:19Z

Indeed. Currently, two things bother me:

a) How do we evaluate the performance on ZDT?

In this regard, the paper has defined the g objective and its optimal value on any given coordinates. We can use coords and evaluate it with g such that it is approximately equal to g*.

For instance, look at this code, this test passed when asserted 3000 times with random seed. We will have to create an API to expose the g function instead of manually writing, for eg.

 ...
 opt.Optimize(objectives, coords);
 double gValue = ZDT_ONE.GetGValue();
 double expectedGValue = ZDT_ONE.GetOptimalGValue();

 REQUIRE(gValue == Approx(expectedGValue).margin(0.99));

b) ZDT5

This has been bothering me for a while now, the paper, pymoo and pagmo all have very different interpretations on the bitstring thingy and its confusing me a lot. Still working on it.

zoq · 2021-05-26T16:37:33Z

a) How do we evaluate the performance on ZDT?

In this regard, the paper has defined the g objective and its optimal value on any given coordinates. We can use coords and evaluate it with g such that it is approximately equal to g*.

For instance, look at this code, this test passed when asserted 3000 times with random seed. We will have to create an API to expose the g function instead of manually writing, for eg.

I'm not sure it's feasible to test e.g. NSGAII on each problem as part of the CI and at the same time keep the runtime low. At the end its just to make sure our implementation is correct and not providing a comprehensive benchmark. If people want to test out other problems, they can do that and for that they can pretty much look at an existing test and adapt it accordingly. So I think we should select one or two and test it against NSGAII. Since we do it just for a limited number of functions, I don't think we have to create another API to expose g. I like to keep it simple so having it explicitly written down in a single method instead of using some problem specific API would make it a lot easier to use the test code as an example.

jonpsy · 2021-05-26T17:01:08Z

Very well, I'll comment that the g objective function is taken from the ZDT1 implementation and to refer the docs for optimal g value. Reg. the CI-timeout, if you recall in NSGA-II improvisation we were able to bring down the numGenerations from 1500 =>300 to achieve similar results, which should've loaned us a huge amount of CI-time. However, I agree with few examples should be enough, so I'll only test against ZDT_ONE as per the code I pasted before, do we agree?

zoq · 2021-05-26T18:07:51Z

Very well, I'll comment that the g objective function is taken from the ZDT1 implementation and to refer the docs for optimal g value. Reg. the CI-timeout, if you recall in NSGA-II improvisation we were able to bring down the numGenerations from 1500 =>300 to achieve similar results, which should've loaned us a huge amount of CI-time. However, I agree with few examples should be enough, so I'll only test against ZDT_ONE as per the code I pasted before, do we agree?

Sounds like a good plan to me.

jonpsy · 2021-05-27T19:28:01Z

Great, only ZDT5 is left. It's a lot confusing if you could weigh in that'd be awesome.

zoq · 2021-05-28T19:14:03Z

Great, only ZDT5 is left. It's a lot confusing if you could weigh in that'd be awesome.

I looked at the pymoo implementation (https://github.com/msu-coinlab/pymoo/blob/master/pymoo/problems/multi/zdt.py#L99) which looks like a reasonable approach to me, also I got the impression that a lot of frameworks just don't implement the function. Anyway, I would just go with the pymoo interpretation, let me know if you like me to look into something specific.

jonpsy · 2021-05-28T21:44:37Z

I didn't quite understand what they're doing. To begin with, x_1 should be a bitstring of length 30 and x2 .... x11 should be a bitstring of length 5. I'm not sure how they're ensuring that the coordinates are a bitstring or are they converting it in some way? Also the paper was clear that there are 11 dimensions, but this implementation somehow calculates numDimensions in a very obscure way.

To make things worse, here's pagmos implementation, especially two parts

a)

    // Convert the input vector into rounded values (integers)
    vector_double x;
    std::transform(x_double.begin(), x_double.end(), std::back_inserter(x),
                   [](double item) { return std::round(item); });

So basically, if coordinates are 0.03 it becomes 0. But how could they guarantee coordinates would be within 0-1?

b)

 // Counts how many 1s are there in the first (30 dim)
  u[0] = static_cast<vector_double::size_type>(std::count(x.begin(), x.begin() + 30, 1.));

Not getting what the "first 30" dim is supposed to mean.

And here's the excerpt from the original paper.

The paper is guaranteeing that the num dimension is 11 but neither of the two implementation seem to follow it. So weird.

zoq · 2021-05-30T02:27:57Z

Thanks for the extra insights, will dig into the two implementations and the paper, and will let you know what I think.

jonpsy · 2021-06-01T09:29:22Z

As discussed, let's skip ZDT5 as of now. Let me know if you find any other changes peculiar 👍 @zoq

zoq · 2021-06-01T16:37:39Z

As discussed, let's skip ZDT5 as of now. Let me know if you find any other changes peculiar @zoq

Yes, let us fix the build before we merge this 👍

jonpsy · 2021-06-06T06:09:27Z

Allright, this should work smooth like a butter.
@zoq Yoooooo, its working!

jonpsy · 2021-06-07T17:33:32Z

I think this is ready.

tests/nsga2_test.cpp

zoq

Looks great to me, no more comments from my side.

mlpack-bot

Second approval provided automatically after 24 hours. 👍

zoq · 2021-06-14T16:31:00Z

Thanks again, great work 👍

mlpack-bot bot added s: needs review s: unanswered s: unlabeled labels Mar 16, 2021

zoq added c: tests t: added feature and removed s: unanswered s: unlabeled labels Mar 18, 2021

zoq reviewed Mar 24, 2021

View reviewed changes

jonpsy and others added 11 commits April 14, 2021 11:54

initial push

6924cf1

=> Documentation done. => Declared the class.

Added

7ac386e

1) ZDT1 2) ZDT2 3) ZDT3 4) ZDT4 5) ZDT6

Test working against pagmo2

f9f3dc2

inner access outer.

a62416b

=> MatType return

7d20f81

=> Indent fix

ElemType

8cbac22

ZDT history

6894b7e

minor indent fix T_T

69520fb

Update include/ensmallen_bits/problems/zdt1_function.hpp

499458d

Co-authored-by: Marcus Edel <marcus.edel@fu-berlin.de>

Update include/ensmallen_bits/problems/zdt1_function.hpp

2436789

Co-authored-by: Marcus Edel <marcus.edel@fu-berlin.de>

Minor style issues

c2dfef1

jonpsy force-pushed the ZDT branch from 7e4f6bc to c2dfef1 Compare April 15, 2021 09:56

jonpsy and others added 2 commits May 4, 2021 13:05

Merge branch 'master' into ZDT

e7b2611

ZDT history

e6b130b

jonpsy and others added 2 commits May 25, 2021 19:08

add ZDT to nsga2 (not working)

d40b12f

GetParetoFront => GetReferenceFront

2c22ab2

jonpsy added 2 commits May 26, 2021 22:50

Add ZDT1 Test

4136639

Document that reference front plot is in the PR.

a6e8291

remov e ZDT5

e6c4513

zoq and others added 2 commits June 5, 2021 22:25

Merge branch 'master' into ZDT

447f808

Fix re-ordering warning

6e9287d

zoq reviewed Jun 7, 2021

View reviewed changes

tests/nsga2_test.cpp Show resolved Hide resolved

Add docs for ZDT_ONE Test

c22c6fc

zoq approved these changes Jun 8, 2021

View reviewed changes

mlpack-bot bot approved these changes Jun 9, 2021

View reviewed changes

mlpack-bot bot removed the s: needs review label Jun 9, 2021

zoq merged commit db90638 into mlpack:master Jun 14, 2021

jonpsy deleted the ZDT branch June 14, 2021 16:31

This was referenced Jul 5, 2021

Release version 2.17.0: "25 Din Mein Paisa Double" #308

Closed

Release version 2.17.0: "Pachis Din Me Pesa Double" #310

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adding ZDT Test Suite #273

Adding ZDT Test Suite #273

jonpsy commented Mar 16, 2021 •

edited

jonpsy commented Mar 16, 2021 •

edited

jonpsy commented Mar 19, 2021

zoq left a comment

zoq Mar 24, 2021

jonpsy commented Mar 24, 2021 •

edited

zoq commented Mar 25, 2021 •

edited

jonpsy commented Mar 25, 2021

jonpsy commented Apr 11, 2021

jonpsy commented Apr 18, 2021

jonpsy commented May 1, 2021 •

edited

zoq commented May 25, 2021

jonpsy commented May 26, 2021

zoq commented May 26, 2021

jonpsy commented May 26, 2021

zoq commented May 26, 2021

jonpsy commented May 27, 2021

zoq commented May 28, 2021

jonpsy commented May 28, 2021 •

edited

zoq commented May 30, 2021

jonpsy commented Jun 1, 2021 •

edited

zoq commented Jun 1, 2021

jonpsy commented Jun 6, 2021 •

edited

jonpsy commented Jun 7, 2021

zoq left a comment

mlpack-bot bot left a comment

zoq commented Jun 14, 2021


		return arma::Col<ElemType>(numVariables, 1, arma::fill::zeros);
		}

Adding ZDT Test Suite #273

Adding ZDT Test Suite #273

Conversation

jonpsy commented Mar 16, 2021 • edited

jonpsy commented Mar 16, 2021 • edited

jonpsy commented Mar 19, 2021

zoq left a comment

Choose a reason for hiding this comment

zoq Mar 24, 2021

Choose a reason for hiding this comment

jonpsy commented Mar 24, 2021 • edited

zoq commented Mar 25, 2021 • edited

jonpsy commented Mar 25, 2021

jonpsy commented Apr 11, 2021

jonpsy commented Apr 18, 2021

jonpsy commented May 1, 2021 • edited

zoq commented May 25, 2021

jonpsy commented May 26, 2021

zoq commented May 26, 2021

jonpsy commented May 26, 2021

zoq commented May 26, 2021

jonpsy commented May 27, 2021

zoq commented May 28, 2021

jonpsy commented May 28, 2021 • edited

zoq commented May 30, 2021

jonpsy commented Jun 1, 2021 • edited

zoq commented Jun 1, 2021

jonpsy commented Jun 6, 2021 • edited

jonpsy commented Jun 7, 2021

zoq left a comment

Choose a reason for hiding this comment

mlpack-bot bot left a comment

Choose a reason for hiding this comment

zoq commented Jun 14, 2021

jonpsy commented Mar 16, 2021 •

edited

jonpsy commented Mar 16, 2021 •

edited

jonpsy commented Mar 24, 2021 •

edited

zoq commented Mar 25, 2021 •

edited

jonpsy commented May 1, 2021 •

edited

jonpsy commented May 28, 2021 •

edited

jonpsy commented Jun 1, 2021 •

edited

jonpsy commented Jun 6, 2021 •

edited