New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Review Request: Stojic #36

Closed
wants to merge 5 commits into
base: master
from

Conversation

Projects
None yet
4 participants
@hstojic

hstojic commented Aug 20, 2017

AUTHOR

Dear @ReScience/editors,

I request a review for the following replication:

Original article

Title: G. E. Hinton, and S. J. Nowlan
Author(s): How learning can guide evolution
Journal (or Conference): Complex Systems, 1 (3), 495-502
Year: 1987
DOI: NA
PDF: http://www.complex-systems.com/pdf/01-3-6.pdf

Replication

Author(s): H. Stojic
Repository: https://github.com/hstojic/ReScience-submission/tree/Stojic-2017
PDF: https://github.com/hstojic/ReScience-submission/raw/Stojic-2017/article/stojic2017.pdf
Keywords: Evolution; Learning; Baldwin effect
Language: R
Domain: Cognitive science

Results

  • Article has been fully replicated
  • Article has been partially replicated
  • Article has not been replicated

Potential reviewers

Potential editor: Nicolas P. Rougier, Benoît Girard, C. Titus Brown
Potential reviewers: Yoav Ram, Anne Urai, Anya Vostinar, David Garcia, Qihong Lu


EDITOR

  • Editor acknowledgment (Aug 21, 2017)
  • Reviewer 1 (@anne-urai, Aug 21, 2017)
  • Reviewer 2 (@qihongl, Aug 22, 2017)
  • Review 1 decision [accept] Sep 11, 2017
  • Review 2 decision [accept] Sep 13, 2017
  • Editor decision [accept] Sep 13, 2017
@rougier

This comment has been minimized.

Member

rougier commented Aug 21, 2017

Thanks for your submission. I will edit it and will invite reviewers soon.
Don't hesitate to ask here for progress in the review.

@rougier rougier self-assigned this Aug 21, 2017

@rougier

This comment has been minimized.

Member

rougier commented Aug 21, 2017

@anne-urai @qihongl Could you review this submission ?

@anne-urai

This comment has been minimized.

anne-urai commented Aug 21, 2017

Hi @rougier, I'm happy to review and will aim to do so before the end of this week.

@rougier

This comment has been minimized.

Member

rougier commented Aug 21, 2017

@anne-urai Great, thank you !

@anne-urai

This comment has been minimized.

anne-urai commented Aug 22, 2017

Dear @hstojic, @rougier,

I have ran the code provided (using Rstudio 1.0.153, on my Macbook Pro laptop running OS X El Capitan) and I was able to reproduce the author's original figures. I applaud the author for reviving this classic, and found the paper generally clear and concise. I have a few suggestions for further improvement.

Major suggestions:

  • The author's point about the difference in the asymptote of P(Undecided) is an important one, which could be strengthened by visualising data that's already present. Specifically, I suggest including a figure with all simulations overlaid (possibly in separate panels for the 3 types of alleles), and a histogram showing the range of endpoints after 50 generations. For comparison, the endpoint of the original Hinton & Nowlan allele frequencies could be added. This way, it should become clear if the original figure could have resulted from a draw of the same process as specified here. Showing that the original draw does not fall within the simulation distribution would give more weight to the argument that the original fitness function used was a flawed one.
  • In Algorithm 1, line 14, 1-19 should be 1+19 (according to both the R code and the original Hinton & Nowlan paper).

Suggestions for improving the useability of the code:

  • rather than listing the necessary R packages in the README, it would be helpful if util.R could automatically check the user's installed packages, and install any missing ones.
  • since the simulations take quite long, please output some form of a status message - preferably including time spent and estimated time to finish.
  • when simulating only 1 trace, no plot is created, and when simulating 2 traces, there are no grey error bars visible. Please check that the code can deal with a single trace, and confirm that error bars are indeed plotted but barely visible due to the high number of simulations.
  • the code is well commented, but the link between the lines in Algorithm 1 and the R script are not immediately clear. It would be very helpful to explicitly refer to sections of Algorithm 1 in the code.

Suggestions for the paper:

  • consider providing a legend with Algorithm 1, explaining the notation used. For readers more familiar with code than mathematical notation (myself, and probably many cognitive scientists), it would help a lot to clarify e.g. \forall and why |G| is used rather than noAgents.
  • a quick Google search for "Hinton Nowlan code" returns several hits for previously published code (e.g. here and here). While I do not think this is a problem for the current submission, please remove from the paper "I am also not aware of other implementations elsewhere."
    • More generally, what is @ReScience/editors take on algorithms which implementation is available elsewhere, albeit in a less clear and non-peer reviewed form?
  • If you have their permission, it would be nice to mention which of the author you've corresponded with, and summarize the discussion.
  • I find "This result has made a substantial impact in the cognitive science literature." weak without accompanying citations.
  • For the reader unfamiliar with the topic, it may be nice to briefly recap the of the Baldwin effect and cite the original paper. Specifically, I would contrast this with Lamarckian inheritance where learned information is passed on to the offspring's genotype. Note that this is not of vital importance since the reference article explains this well, but it would still be nice to very briefly recap the debate to which the reference paper applies.
  • In footnote 1, it would be helpful to explicitly state original figures. Also, a full stop at the end of the footnote is missing.
@qihongl

This comment has been minimized.

Member

qihongl commented Aug 22, 2017

Sorry for the delayed response! The notification was buried by other messages...

I would love to review it! I will try to respond by the end of this week!

@rougier rougier added the 02 - Review label Aug 22, 2017

@rougier

This comment has been minimized.

Member

rougier commented Aug 22, 2017

@anne-urai @qihongl Thank you both for accepting the review. I'm supposed to write now that review can start but @anne-urai already sent her review 😄.

@rougier

This comment has been minimized.

Member

rougier commented Aug 22, 2017

@hstojic You can wait for second review to make suggested changes if your prefer to address comments all at once.

@hstojic

This comment has been minimized.

hstojic commented Aug 22, 2017

@rougier @anne-urai @qihongl Thank you very much for accepting the editor/reviewer roles and for being so prompt about it.

I will wait for Quihong's coments and then address both reviews.

@qihongl

This comment has been minimized.

Member

qihongl commented Aug 25, 2017

Hi, Folks ( @hstojic , @rougier, @anne-urai ),

Sorry for the waiting! Nice replication! I was able to run your code (R version 3.4.0 (2017-04-21), MacOS 10.12.6). I really enjoyed reading Hinton and others' early work! @anne-urai nicely covered many points in my mind. Though I do have some additional thoughts that might be worth considering.

First of all, I think having a clean and well-documented code for the referenced work is valuable, since the original paper is of interest to many people. It is highly cited. Multiple attempts of replicating the result also reflect people's interest.

I really appreciate your effort for trying to understand the interesting difference between your result and the original result. I think it would be even better if you can recover the original plot, if the original finding is flawed as you said. I agree with @anne-urai that summarizing your discussion with the author would be very informative. However, in my opinion, successfully replicating the original plot is more convincing than the author's approval. I guess some potential readers of your work would be interested to see concrete evidence as well.

By the way, in the MATLAB replication listed by @anne-urai, the frequency of learnable genes converged at around 0.2. (instead of decreasing monotonically). This trend is more similar to the original result. Understanding how does this implementation differ from yours might be useful for recovering the original finding. (Thanks! @anne-urai)

Second, your code can be slightly more general. For example, the number of alleles in your code is set to 20. I think it is useful to let it be a parameter that user can specify at the beginning of the simulation code, so that the user can explore some potential effect of varying the number of alleles. One reason why this is interesting to explore is articulated here: https://egtheory.wordpress.com/2014/02/07/learning-guide-evolution/

This is just one example. My general point is that, though the purpose of the code (and this journal) is to replicate the finding, having slightly a more generic implementation (that goes above and beyond replicating the original result) might provide additional educational value.

Some detail:

  1. errorbar
    "...barely visible grey ribbons are standard errors"
  • How many standard errors? (alternatively, It might be helpful to translate them to a (1-alpha) confidence interval.)
  • I replicated what @anne-urai mentioned, so I share her concerns.
  1. In table 1, the description of p_x is "Probability that allele is of type x". It is probably more reader friendly to say what "type x" is. For example, I would describe p_? as "Probability that allele is genetically undecided (or learnable?)". Anyway, make sure the choices of language are consistent with the plots.

  2. I found using "noSim", "noAgents" to represent the number of simulations and the number of agents to be slightly confusing. I would use "numSim" and "numAgent", or "nSim" and "nAgent". Nevermind if your choice is commonly accepted coding in your field though...

  3. At the beginning of the simulation file, It might be useful to add "rm(list=ls())" to remove all environmental variables and functions.

  4. Finally, I think some wording can be slightly more concise.

Example 5.1:

  • "The main aim of this article is to replicate the simulations reported in the reference article [4]. I have corresponded with one of the authors and learned that the original implementation of the simulations is no longer available. I am also not aware of other implementations elsewhere. The simulations are relatively simple, and I thus propose a replication based on description from the reference article. Replication code relies on R programming language [6] and several R packages [8, 10, 9, 1, 2, 3, 7, 5]."

Something like the following might be better?

  • "I replicated the reference article [4] in R [6] with some auxiliary packages [8, 10, 9, 1, 2, 3, 7, 5], because other existing replications are ... "

Example 5.2:
The following discussion might be unnecessary if you can recover the original plot:

"... One explanation is that the results reported in Figure 2 in the reference article stem from a non-typical draw. This is unlikely, as upon examining the evolutionary paths of Undecided alleles in all simulation runs, they all show a strong downward trend already in the first 50 generations, differing only in the exact generation when this decrease onsets."

Thank you very much for spending the time to read my thoughts. I hope some of them are useful. And feel free to implement them as you see fit!

@hstojic

This comment has been minimized.

hstojic commented Aug 28, 2017

Thank you all for quick responses and all the comments - they are very helpful and will allow me to improve the article. I'll think about all the issues raised in the next few days and give you my response.

@hstojic

This comment has been minimized.

hstojic commented Sep 5, 2017

Response to reviews

I provide a comment below each suggestion of the reviewer.

Anne

Dear @hstojic, @rougier,

I have ran the code provided (using Rstudio 1.0.153, on my Macbook Pro laptop running OS X El Capitan) and I was able to reproduce the author's original figures. I applaud the author for reviving this classic, and found the paper generally clear and concise. I have a few suggestions for further improvement.

Major suggestions:

  • The author's point about the difference in the asymptote of P(Undecided) is an important one, which could be strengthened by visualising data that's already present. Specifically, I suggest including a figure with all simulations overlaid (possibly in separate panels for the 3 types of alleles), and a histogram showing the range of endpoints after 50 generations. For comparison, the endpoint of the original Hinton & Nowlan allele frequencies could be added. This way, it should become clear if the original figure could have resulted from a draw of the same process as specified here. Showing that the original draw does not fall within the simulation distribution would give more weight to the argument that the original fitness function used was a flawed one.
    • HS: Indeed, this is how I checked it, thanks for the suggestion for adding it to the paper. I opted for adding a figure with paths and a histogram of end points for Undecided alleles since these are the focus of that discussion.
  • In Algorithm 1, line 14, 1-19 should be 1+19 (according to both the R code and the original Hinton & Nowlan paper).
    • HS: Thank you for spotting this, corrected!

Suggestions for improving the useability of the code:

  • rather than listing the necessary R packages in the README, it would be helpful if util.R could automatically check the user's installed packages, and install any missing ones.

    • HS: Good point. I actually modified the code so that utils.R automatically installs the version of the package with which the code is developed. This is really needed so that the code does not break with more recent version of the packages. It will overwrite the packages if some of them are already installed though.
  • since the simulations take quite long, please output some form of a status message - preferably including time spent and estimated time to finish.

    • HS: Good suggestion, I provided some rudimentary estimates.
  • when simulating only 1 trace, no plot is created, and when simulating 2 traces, there are no grey error bars visible. Please check that the code can deal with a single trace, and confirm that error bars are indeed plotted but barely visible due to the high number of simulations.

    • HS: Thanks for the suggestion. I have made sure the figure is plotted regardless of the number of simulations. I also confirmed that error bars are plotted (computation before was hardcoded, thats why ribbons were not showing with 2 simulations).
  • the code is well commented, but the link between the lines in Algorithm 1 and the R script are not immediately clear. It would be very helpful to explicitly refer to sections of Algorithm 1 in the code.

    • HS: Algorithm 1 is only a pseudocode, the structure of the code itself is a bit different for efficiency reasons. I have put more comments in the code trying to connect it to the Algorithm 1 more clearly. I also changed the names of some variables in the code to make it more similar.

Suggestions for the paper:

  • consider providing a legend with Algorithm 1, explaining the notation used. For readers more familiar with code than mathematical notation (myself, and probably many cognitive scientists), it would help a lot to clarify e.g. \forall and why |G| is used rather than noAgents.

    • HS: I have eliminated the math notation.
  • a quick Google search for "Hinton Nowlan code" returns several hits for previously published code (e.g. here and here). While I do not think this is a problem for the current submission, please remove from the paper "I am also not aware of other implementations elsewhere."

    • HS: It seems I have used wrong keywords, but true, I did not put a serious effort in searching for the code, so I dropped the sentence. And thanks for spotting them, It will be useful for comparison, to see if we get similar results.
  • More generally, what is @ReScience/editors take on algorithms which implementation is available elsewhere, albeit in a less clear and non-peer reviewed form?

  • If you have their permission, it would be nice to mention which of the author you've corresponded with, and summarize the discussion.

    • HS: Email exchange with Geoff Hinton was brief, what I have provided is a summary. In another correspondence he agreed to name him here.
  • I find "This result has made a substantial impact in the cognitive science literature." weak without accompanying citations.

    • HS: I was referring primarily to the number of citations, so I rephrased the sentence in that manner. I also saw it on reading lists in machine learning / AI courses.
  • For the reader unfamiliar with the topic, it may be nice to briefly recap the of the Baldwin effect and cite the original paper. Specifically, I would contrast this with Lamarckian inheritance where learned information is passed on to the offspring's genotype. Note that this is not of vital importance since the reference article explains this well, but it would still be nice to very briefly recap the debate to which the reference paper applies.

    • HS: I have thought this was relatively clear from the introduction. I have put the footnote about the Baldwin effect in the text now and tried to contrast them more clearly.
  • In footnote 1, it would be helpful to explicitly state original figures. Also, a full stop at the end of the footnote is missing.

    • HS: I am not sure that I understand the suggestion. It does not make sense to put whole captions, they are rather big.

Quihong

Hi, Folks ( @hstojic , @rougier, @anne-urai ),

Sorry for the waiting! Nice replication! I was able to run your code (R version 3.4.0 (2017-04-21), MacOS 10.12.6). I really enjoyed reading Hinton and others' early work! @anne-urai nicely covered many points in my mind. Though I do have some additional thoughts that might be worth considering.

First of all, I think having a clean and well-documented code for the referenced work is valuable, since the original paper is of interest to many people. It is highly cited. Multiple attempts of replicating the result also reflect people's interest.

  • I really appreciate your effort for trying to understand the interesting difference between your result and the original result. I think it would be even better if you can recover the original plot, if the original finding is flawed as you said. I agree with @anne-urai that summarizing your discussion with the author would be very informative. However, in my opinion, successfully replicating the original plot is more convincing than the author's approval. I guess some potential readers of your work would be interested to see concrete evidence as well.

    • HS: I agree that would be great, but without the original code it would be too difficult to replicate the original result, I simply do not see how to figure that out in an efficient manner. I would welcome any suggestions on how to do that.
  • By the way, in the MATLAB replication listed by @anne-urai, the frequency of learnable genes converged at around 0.2. (instead of decreasing monotonically). This trend is more similar to the original result. Understanding how does this implementation differ from yours might be useful for recovering the original finding. (Thanks! @anne-urai)

    • HS: The Matlab code actually has a bug in lines 66-68, and it runs the simulation only once. When these issues are resolved I get the same results as in my simulations. This suggests that I correctly implemented the simulations.
  • Second, your code can be slightly more general. For example, the number of alleles in your code is set to 20. I think it is useful to let it be a parameter that user can specify at the beginning of the simulation code, so that the user can explore some potential effect of varying the number of alleles. One reason why this is interesting to explore is articulated here: https://egtheory.wordpress.com/2014/02/07/learning-guide-evolution/

    • HS: Thanks. I made it a parameter defined at the top of the script. And thanks for the reference, that was an interesting discussion on limitations of the original simulations.

This is just one example. My general point is that, though the purpose of the code (and this journal) is to replicate the finding, having slightly a more generic implementation (that goes above and beyond replicating the original result) might provide additional educational value.

Some detail:

  • errorbar "...barely visible grey ribbons are standard errors". How many standard errors? (alternatively, It might be helpful to translate them to a (1-alpha) confidence interval.)

    • HS: Thanks. I made it more precise in the caption. These are usual standard errors of the mean, sd/sqrt(N).
  • I replicated what @anne-urai mentioned, so I share her concerns. In table 1, the description of p_x is "Probability that allele is of type x". It is probably more reader friendly to say what "type x" is. For example, I would describe p_? as "Probability that allele is genetically undecided (or learnable?)". Anyway, make sure the choices of language are consistent with the plots.

    • HS: Thanks. I made ? more reader friendly, while 0 and 1 has to stay as they are, as there is no correspondence with figures - Correct allele can be either 1 or 0.
  • I found using "noSim", "noAgents" to represent the number of simulations and the number of agents to be slightly confusing. I would use "numSim" and "numAgent", or "nSim" and "nAgent". Nevermind if your choice is commonly accepted coding in your field though...

    • HS: This choice is arbitrary, no. is usual acronym for numbers I believe, but indeed, choices might differ in different fields. I would stick with the one I used.
  • At the beginning of the simulation file, It might be useful to add "rm(list=ls())" to remove all environmental variables and functions.

    • HS: Thanks, I corrected it.

Finally, I think some wording can be slightly more concise.

Example 5.1:

  • "The main aim of this article is to replicate the simulations reported in the reference article [4]. I have corresponded with one of the authors and learned that the original implementation of the simulations is no longer available. I am also not aware of other implementations elsewhere. The simulations are relatively simple, and I thus propose a replication based on description from the reference article. Replication code relies on R programming language [6] and several R packages [8, 10, 9, 1, 2, 3, 7, 5]."

Something like the following might be better?

  • "I replicated the reference article [4] in R [6] with some auxiliary packages [8, 10, 9, 1, 2, 3, 7, 5], because other existing replications are ... "
    • HS: Thanks, I shortened it a bit.

Example 5.2:

  • The following discussion might be unnecessary if you can recover the original plot: "... One explanation is that the results reported in Figure 2 in the reference article stem from a non-typical draw. This is unlikely, as upon examining the evolutionary paths of Undecided alleles in all simulation runs, they all show a strong downward trend already in the first 50 generations, differing only in the exact generation when this decrease onsets."
    • HS: See my response to your first comment.

Thank you very much for spending the time to read my thoughts. I hope some of them are useful. And feel free to implement them as you see fit!

@hstojic

This comment has been minimized.

hstojic commented Sep 5, 2017

Dear @rougier @anne-urai and @qihongl,

thanks again for all the comments, I have updated the article reflecting all your suggestions, let me know if you are satisfied with the changes and the final article and the code.

Best,
Hrvoje

@rougier

This comment has been minimized.

Member

rougier commented Sep 6, 2017

Thanks. @anne-urai @qihongl are you satisfied with all the modifications by @hstojic ?
If this is the case, can you post your decision here?

@qihongl

This comment has been minimized.

Member

qihongl commented Sep 7, 2017

Thanks! @hstojic @rougier

I am traveling for a conference this week and I will get back to you ASAP!

@anne-urai

This comment has been minimized.

anne-urai commented Sep 8, 2017

Hi @hstojic,

Unfortunately your changes to package management seem to have caused the code to malfunction in my installation of R. devtools should be loaded also when already installed, and the packages should only be installed if not already present (now, it seems like they will be re-installed at each run of the code? e.g. https://gist.github.com/stevenworthington/3178163).

If I manually set the packages and uncomment this part of the code, it runs smoothly. It would be great if you could still check this, so that future users can run the script easily.

@hstojic

This comment has been minimized.

hstojic commented Sep 9, 2017

Hi @anne-urai,

thanks, I have modified the code so that now it first checks whether a package exists before installing. Nevertheless, it still check whether a correct version exists and installs it even if it finds a different version. I think this is important as packages do change over time and the code might not work in the future otherwise.

Best,
Hrvoje

@anne-urai

This comment has been minimized.

anne-urai commented Sep 10, 2017

Hi @hstojic, @rougier,

thanks for the quick update. The code now runs smoothly, and all my comments have been addressed. I recommend publication of the paper. Thank you for making this code available to the community!

@rougier

This comment has been minimized.

Member

rougier commented Sep 11, 2017

Thanks @anne-urai. @qihongl Are your back from your conference? Did you make your decision ?

@qihongl

This comment has been minimized.

Member

qihongl commented Sep 11, 2017

Sorry for the delayed response. I'm overall happy with all of the modifications. I was just slightly concerned about the differences between the replication and the original result.

Since the referenced paper is highly cited, to claim that the original implementation is flawed, I think we're responsible for providing a crystal clear explanation. And I think this replication would be more impactful if we can understand exactly what was going on. I do understand that it might be hard to recover the original result. In this case, it might be useful provide a thorough mathematical analysis of why the number of undecided alleles should decrease continuously over time.

What do you folks think? @anne-urai @hstojic @rougier If people agree that a more precise explanation of the deviation from the original result is not necessary, I would recommend this paper for publication. The current result does show learning can act as "2nd order mutation" that guide exploration over certain optimization landscapes, which was the main point of the original paper.

By the way, overriding user's packages might not be a good idea. I guess a better practice is to use a container (e.g. Docker) or getting a virtual machine image.

@hstojic

This comment has been minimized.

hstojic commented Sep 12, 2017

Dear @qihongl,

thanks for a prompt response.

I agree, I would be happier as well if I could explain exactly what went wrong. Nevertheless, in my opinion tracking down the source of it is beyond the scope of a replication, and same holds for doing an additional mathematical analysis. Perhaps I would be tempted to put more effort in getting at the bottom of it if it weren't for correspondence with Geoff where he agreed with the replication results. I have also checked Java and Matlab code found by @anne-urai and with those same results are obtained (with Matlab code after correcting some bugs). Finally, the main result and conclusion of the paper still holds, as you also indicated.

Indeed, overriding packages is not nice, but I thought it's necessary. Perhaps a Docker is a better idea, do @rougier and @anne-urai have opinion on this? That might be a good idea for a general journal policy, to make the code in the articles more reproducible (independent of operating systems etc).

Best,
Hrvoje

@rougier

This comment has been minimized.

Member

rougier commented Sep 12, 2017

@qihongl I see your point but in the same time it seems impossible to recover the original code/result. As for a thorough mathematical analysis, I think it may be beyond the scope of this replication (but I've no estimation of how much work this would represent). @hstojic Do you have any idea on that matter ?

Another option would be to develop such an analysis in a new paper (I mean not a replication) if it brings new interpretation.

@hstojic

This comment has been minimized.

hstojic commented Sep 12, 2017

Genetic algorithms are not really my domain of expertise, so I'm not sure how much work would it be. If I would put an effort into it, then I think the paper would go beyond a replication and I could submit it to a different journal as well.

@rougier

This comment has been minimized.

Member

rougier commented Sep 12, 2017

@hstojic Make sense. @qihongl I think we can accept the submission as it is (modulo the docker question) but I would need you to confirm your decision.

@rougier

This comment has been minimized.

Member

rougier commented Sep 12, 2017

The docker question has been raised a couple of times and we have no definitive answer. I'm not familar at all with R so I've no real opinion (I'm not sure what it means to override packages). Can't you use virtual environment of local packages ?

@qihongl

This comment has been minimized.

Member

qihongl commented Sep 12, 2017

I see! Then I would recommend this paper for publication! Thank you very much again @hstojic !

BTW @hstojic sorry about asking too much... I used to get stuck on some research project due to seemingly irrelevant differences between the replicated result vs. the original result. And I still have a little bit of PTSD I guess... lol

@rougier

This comment has been minimized.

Member

rougier commented Sep 13, 2017

I therefore accept the submission. Congratulations @hstojic and many thanks @qihongl @anne-urai for the fast and detailed reviews.

@ReScience ReScience locked and limited conversation to collaborators Sep 13, 2017

@rougier

This comment has been minimized.

Member

rougier commented Sep 13, 2017

The paper will soon be published.

@rougier

This comment has been minimized.

Member

rougier commented Sep 13, 2017

Here is the DOI: DOI

The article will be soon listed on http://rescience.github.io/read/

@rougier

This comment has been minimized.

Member

rougier commented Sep 13, 2017

Done ! 🎉

@rougier rougier closed this Sep 13, 2017

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.