New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Review Request: Bavard and Théro #47

Closed
wants to merge 21 commits into
base: master
from

Conversation

Projects
None yet
9 participants
@heloisethero

heloisethero commented Feb 9, 2018

AUTHOR

Dear @ReScience/editors,

I request a review for the following replication:

Original article

Title: Adaptive properties of differential learning rates for positive and negative outcomes
Author(s): Cazé, R. D., & van der Meer, M. A.
Journal (or Conference): Biological cybernetics
Year: 2013
DOI: 10.1007/s00422-013-0571-5
PDF: Caze&vanderMeer2013.pdf
https://link.springer.com/article/10.1007/s00422-013-0571-5

Replication

Author(s): Bavard, Sophie & Théro, Héloïse
Repository: BAVARD-THERO-2018
PDF: Bavard-Thero-2018.pdf
Keywords: Reinforcement learning, Q-learning, adaptive learning rates
Language: Python
Domain: Computational Neuroscience

Results

  • Article has been fully replicated
  • Article has been partially replicated
  • Article has not been replicated

Potential reviewers

Benoît Girard
Mehdi Khamassi
Xavier Hinaut


EDITOR

  • Editor acknowledgment (@oliviaguest February 20, 2018)
  • Reviewer 1 (@neuronalX February 21, 2018)
  • Reviewer 2 (@benoit-girard March 2, 2018)
  • Review 1 decision [accept May 29, 2018]
  • Review 2 decision [accept June 11, 2018]
  • Editor decision [accept June 14, 2018]

Heloise and others added some commits Feb 9, 2018

@rougier

This comment has been minimized.

Member

rougier commented Feb 9, 2018

Thanks for your submission. An editor will be assigned soon.

@rougier

This comment has been minimized.

Member

rougier commented Feb 9, 2018

@oliviaguest Can you edit this submission ?

@oliviaguest

This comment has been minimized.

oliviaguest commented Feb 9, 2018

@rougier I can do it after the 20th of this month, is that too late?

@rougier rougier changed the title from Sophie Bavard and Héloïse Théro to Review Request: Bavard and Théro Feb 9, 2018

@rougier

This comment has been minimized.

Member

rougier commented Feb 9, 2018

@oliviaguest Yes
@heloisethero Can you add a link to your repository (you should be able to edit your first post)

@rougier

This comment has been minimized.

Member

rougier commented Feb 9, 2018

@heloisethero Can you also add a link to the PDF? There was a link but it was pointing to the article that seem to be stored on GitHub. Since it is a copyrighted article you cannot add it to GitHub. BEst would be to have an open access version. Else, you can add the official entry from Biol. Cybernetics.

@heloisethero

This comment has been minimized.

heloisethero commented Feb 9, 2018

@rougier I have now added a link to my repository at the right place, sorry about that. I have also added a link to the first author's ResearchGate article pdf. The figures in it are identical to the ones published in Biol. Cybernetics, although the formatting is different. I have also added to the official entry from Biol. Cybernetics just in case.

@rougier

This comment has been minimized.

Member

rougier commented Feb 9, 2018

@heloisethero Perfect. As you may have read in the thread @oliviaguest will handling your submission after February 20th. If this is a problem just tell us, we'll try to find another editor.

@heloisethero

This comment has been minimized.

heloisethero commented Feb 12, 2018

@rougier and @oliviaguest, there is no problem, Sophie (@sophiebavard) and I accept your decision.

@rougier

This comment has been minimized.

Member

rougier commented Feb 21, 2018

@oliviaguest Reminder

@oliviaguest

This comment has been minimized.

oliviaguest commented Feb 21, 2018

@benoit-girard @MehdiKhamassi @neuronalX any of you have time to review this?

@neuronalX

This comment has been minimized.

neuronalX commented Feb 21, 2018

@oliviaguest Yes, I can review it.

@benoit-girard

This comment has been minimized.

benoit-girard commented Feb 22, 2018

@oliviaguest Technically, I can do it.
There is however a possible conflict of interest: I am the tutor of H. Théro in her current doctoral school. I do not have any hierarchical relationship with her, I am not supervising her. In this role, I am essentially a possible channel of information when interacting with the doctoral school.
Please tell me if you consider it problematic.
You could also alternatively ask directly to @rcaze (first author of the initial study) to review the submission.

@MehdiKhamassi

This comment has been minimized.

MehdiKhamassi commented Feb 22, 2018

@oliviaguest Thank you for the invitation! I am already late with two other reviews. So I cannot currently. Best wishes

@oliviaguest

This comment has been minimized.

oliviaguest commented Feb 22, 2018

@MehdiKhamassi @benoit-girard @neuronalX Thank you all for the speedy replies! 😄

@rougier What do you think/what are the rules vis-à-vis the potential conflict?

@rcaze

This comment has been minimized.

rcaze commented Feb 22, 2018

I meant to write to the authors. The python code generating all the figures are on my github repo. They were mislead because they look at Matt Van Der Meer lab repo and could not find the code. I thought the link to the repo was in the original article.

@rcaze

This comment has been minimized.

rcaze commented Feb 22, 2018

I am going to alert also Matt. https://github.com/rcaze/13_02CaVa

@heloisethero

This comment has been minimized.

heloisethero commented Feb 26, 2018

@rcaze we are sorry for this mistake. As your article was really influential in our research team, we have tried on several occasions to replicate parts of it. We have been looking for your code, but never found it. Therefore we have recoded everything from scratch.
@rougier @oliviaguest we are sorry for this situation. @sophiebavard and I are ready to withdraw our review request if you think this is the right thing to do.

@oliviaguest

This comment has been minimized.

oliviaguest commented Feb 26, 2018

@heloisethero The fact you can replicate the original code using the published papers without looking at the original code is extremely important. It means the specification is actually complete given the journal articles and other writing. This is a massive boon to the original authors' work, their framework, etc. For more see, e.g., Cooper, R. P. & Guest, O. (2014). Implementations are not specifications: specification, replication and experimentation in computational cognitive modeling. Cognitive Systems Research. doi: 10.1016/j.cogsys.2013.05.001.

@rcaze

This comment has been minimized.

rcaze commented Feb 26, 2018

@heloisethero I agree with @oliviaguest. I am honoured by this attention and think that the independent reproduction gives strength to this result. Notably, the original code lacks a documentation. One member of our lab (@frct) is currently working on this topic and could give a serious review of the work. If the editors wish I can also review the paper even if I think that I am a little biased :).

@rougier

This comment has been minimized.

Member

rougier commented Feb 26, 2018

Yes, it is perfectly fine to replicate the code even if the original code is available as long as your replication is not a simple translation from one language to another and is not a copy. So this submission is fine for me.

@heloisethero

This comment has been minimized.

heloisethero commented Mar 1, 2018

@oliviaguest hello, @sophiebavard and I were wondering if you have chosen the second reviewer?

@oliviaguest

This comment has been minimized.

oliviaguest commented Mar 1, 2018

@heloisethero: As soon as @rougier tells me if there's a conflict (see my question above), I'll act accordingly. Understandably, these things can take time.

@benoit-girard

This comment has been minimized.

benoit-girard commented May 24, 2018

Fig 2 is perfect as soon as these "" are removed...

@heloisethero

This comment has been minimized.

heloisethero commented May 24, 2018

@benoit-girard you are right, thank you. Sorry for that, I have corrected it :)

@benoit-girard

This comment has been minimized.

benoit-girard commented May 29, 2018

I am now satisfied with the paper.
@oliviaguest : my review is over, and I recommend acceptation.

@heloisethero

This comment has been minimized.

heloisethero commented Jun 4, 2018

@neuronalX Did you have any chance to check our simulations? :)

@rougier

This comment has been minimized.

Member

rougier commented Jun 4, 2018

@oliviaguest @neuronalX Gentle reminder.

@neuronalX

This comment has been minimized.

neuronalX commented Jun 5, 2018

No, sorry, didn't find the time so far to have a complete look at all the comments since my review.

@rougier

This comment has been minimized.

Member

rougier commented Jun 11, 2018

@neuronalX Can you give back your comments by the end of the week please.

@neuronalX

This comment has been minimized.

neuronalX commented Jun 11, 2018

Thank you for the changes you made.

We have now redefined the colors with Hex triplets, and we have managed to run the code with both Python 2 and 3 without problem. Can you check with the new version of our code?

Yes, it works fine for me now.

Concerning the Figure 4 A, maybe you could just add a sentence saying that some similar figure could be obtained by dividing the number of simulations by 10 for instance (so this info does not stay in the review only).

@oliviaguest @rougier I recommend acceptation.

@oliviaguest

This comment has been minimized.

oliviaguest commented Jun 11, 2018

Excellent!

@heloisethero do you what to address the small comment above on your figure 4 A?

heloisethero added some commits Jun 13, 2018

Add files via upload
We added one sentence on the results section according to one of the reviewers' advice.
Add files via upload
We added one sentence on the results section according to one of the reviewers' advice.
@heloisethero

This comment has been minimized.

heloisethero commented Jun 13, 2018

@neuronalX thank you for your comment. We added this sentence at the end of the results section :

However, we obtained similar noise when dividing the number of simulations by ten, suggesting the original authors used around 500 iterations for this part of the figure.

@oliviaguest let us know if we need to do anything else?

@oliviaguest

This comment has been minimized.

oliviaguest commented Jun 14, 2018

@heloisethero I cannot see anything else that needs doing from your end — so I'll reply on this thread when I am done publishing this, assigning a DOI, etc.

@ReScience ReScience locked as resolved and limited conversation to collaborators Jun 14, 2018

@oliviaguest

This comment has been minimized.

oliviaguest commented Jun 15, 2018

This submission has been accepted for publication! It should appear soon on the journal website (here) with DOI: DOI

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.