-
Notifications
You must be signed in to change notification settings - Fork 119
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Adding ZDT Test Suite #273
Conversation
Few updates: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks pretty good to me, I made some comments about the style, I'll check the function implementation next, the optimizer is already able to solve the problems, so that is a good sign.
|
||
return arma::Col<ElemType>(numVariables, 1, arma::fill::zeros); | ||
} | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Might be a good idea to add a comment to ObjectiveF1, ObjectiveF2
.
Just to make your job easier, I've created a repository comparing pagmo results to mine. Ofcourse, these are trivial cases, if you have any edge cases in mind do let me know so I can test and compare them. Cheers! |
Usually I take a look at the function definition (paper, wikipedia) and compare it with the implementation, since there is a chance that an existing implementation does something different; but it's a nice basis. |
I see. Here's the link to the research paper: https://www.mitpressjournals.org/doi/abs/10.1162/106365600568202 |
Now that NSGA-II fix is close to merge, we can start working on this. ZDT5 is also remaining, but since it only works on bitstring type I would need to use |
=> Documentation done. => Declared the class.
=> Indent fix
Co-authored-by: Marcus Edel <marcus.edel@fu-berlin.de>
Co-authored-by: Marcus Edel <marcus.edel@fu-berlin.de>
Keep open |
Updates on this: pymoo has implemented the Pareto front for each of these so I have included the Ipynb notebook in this pull request. I was thinking we could test if the Pareto Front from the Pymoo implementation is the same as ours, see if the outputs match? Your thoughts @zoq ? |
Looking at the Travis output, this is an easy issue to fix. |
Indeed. Currently, two things bother me: a) How do we evaluate the performance on ZDT? In this regard, the paper has defined the g objective and its optimal value on any given coordinates. We can use For instance, look at this code, this test passed when asserted 3000 times with random seed. We will have to create an API to expose the ...
opt.Optimize(objectives, coords);
double gValue = ZDT_ONE.GetGValue();
double expectedGValue = ZDT_ONE.GetOptimalGValue();
REQUIRE(gValue == Approx(expectedGValue).margin(0.99)); b) ZDT5 This has been bothering me for a while now, the paper, pymoo and pagmo all have very different interpretations on the bitstring thingy and its confusing me a lot. Still working on it. |
I'm not sure it's feasible to test e.g. NSGAII on each problem as part of the CI and at the same time keep the runtime low. At the end its just to make sure our implementation is correct and not providing a comprehensive benchmark. If people want to test out other problems, they can do that and for that they can pretty much look at an existing test and adapt it accordingly. So I think we should select one or two and test it against NSGAII. Since we do it just for a limited number of functions, I don't think we have to create another API to expose |
Very well, I'll comment that the g objective function is taken from the ZDT1 implementation and to refer the docs for optimal g value. Reg. the CI-timeout, if you recall in NSGA-II improvisation we were able to bring down the numGenerations from 1500 =>300 to achieve similar results, which should've loaned us a huge amount of CI-time. However, I agree with few examples should be enough, so I'll only test against ZDT_ONE as per the code I pasted before, do we agree? |
Sounds like a good plan to me. |
Great, only ZDT5 is left. It's a lot confusing if you could weigh in that'd be awesome. |
I looked at the pymoo implementation (https://github.com/msu-coinlab/pymoo/blob/master/pymoo/problems/multi/zdt.py#L99) which looks like a reasonable approach to me, also I got the impression that a lot of frameworks just don't implement the function. Anyway, I would just go with the pymoo interpretation, let me know if you like me to look into something specific. |
I didn't quite understand what they're doing. To begin with, To make things worse, here's pagmos implementation, especially two parts a) // Convert the input vector into rounded values (integers)
vector_double x;
std::transform(x_double.begin(), x_double.end(), std::back_inserter(x),
[](double item) { return std::round(item); }); So basically, if coordinates are b) // Counts how many 1s are there in the first (30 dim)
u[0] = static_cast<vector_double::size_type>(std::count(x.begin(), x.begin() + 30, 1.)); Not getting what the "first 30" dim is supposed to mean. And here's the excerpt from the original paper. The paper is guaranteeing that the num dimension is 11 but neither of the two implementation seem to follow it. So weird. |
Thanks for the extra insights, will dig into the two implementations and the paper, and will let you know what I think. |
As discussed, let's skip ZDT5 as of now. Let me know if you find any other changes peculiar 👍 @zoq |
Yes, let us fix the build before we merge this 👍 |
Allright, this should work smooth like a butter. |
I think this is ready. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks great to me, no more comments from my side.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Second approval provided automatically after 24 hours. 👍
Thanks again, great work 👍 |
Pizza!
Declared files
zdt1_function.hpp
zdt2_function.hpp
zdt3_function.hpp
zdt4_function.hpp
zdt5_function.hpp[skipping for now]zdt6_function.hpp
TODO
Pareto Fronts plotting: https://colab.research.google.com/drive/1Hgmvj4ZQVdT6OgopLPabldeyaOm0DInI?usp=sharing