Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Review of submission 24ferrari #19

Closed
neilernst opened this issue Jul 3, 2019 · 15 comments

Comments

@neilernst
Copy link
Collaborator

commented Jul 3, 2019

No description provided.

@data-track-reviewer

This comment has been minimized.

Copy link
Collaborator

commented Jul 4, 2019

I believe I should be assigned to this issue, but appear to lack rights. @neilernst , could you please help?

@neilernst

This comment has been minimized.

Copy link
Collaborator Author

commented Jul 4, 2019

you will have to accept the collaborator invite, then can self-assign

@data-track-reviewer

This comment has been minimized.

Copy link
Collaborator

commented Jul 9, 2019

Summary

The authors present a data set for their RE submission Learning Requirements Elicitation Interviews
with Role-playing, Self-assessment and Peer-review
. The data set includes all material to rerun the training as well as the evaluation reported on in the paper. It therefore provides value in two aspects: 1. an educational resource for RE teachers, 2. an empirical resource for replication.

Evaluation

The authors apply here for availability. Thus, the data set must be checked against functionality and availability criteria.

  • Functional: I find the data set well documented and complete. The associated paper does provide "appropriate evidence of verification and validation", it is however not linked in the repository (likely because submission is pending). Perhaps a gist of this evidence and a placeholder for a link to the paper to be provided after publication could be added?
  • Available: The data set is available on github and zenodo, likely with zenodo being the leading repository. Thus, I find the criteria for this badge fulfilled.

In summary, I am in favor of the availability badge, but suggest to provide links to the paper as soon as these become available.

Further remarks

One could consider the reusable badge for this data set as well. For that, one of the following should be provided:

  • Resuable for empirical evaluation: Rational and plan for participant selection as well as experimental setup should be provided in form of guidelines. The goal of this would be to make it easy to repeat the setup in a different course setting, thus being "very carefully documented and well-structured to the extent that reuse and repurposing is facilitated".
  • Reusable for educational purposes: Relation to learning objectives and assessment should be provided as well as guidelines for teachers on how to include this module.

The authors however do not apply for this badge, thus this might be ideas for future work.

@alessioferrari

This comment has been minimized.

Copy link
Collaborator

commented Jul 9, 2019

@data-track-reviewer Thanks a lot for the very constructive feedback.

  • We provided a placeholder for the link to the paper in Zenodo, to be filled when the paper is published. To this end, we updated the Zenodo entry by adding the sentence:

More information is available in the publication "Learning Requirements Elicitation Interviews with Role-playing, Self-assessment and Peer-review", to appear in the proceedings of the 27th IEEE International Requirements Engineering Conference, 2019.

Please check: https://doi.org/10.5281/zenodo.2625705

  • Thank you for the suggestion about the reusable badge. At the moment, we would consider the option of "reusable for educational purposes", as, at this stage, we are more interested in this.

In your opinion, would it make sense to apply for the reusable badge, by including guidelines for teachers, and learning objectives, but without the guidelines for experimental evaluation? If @data-track-reviewer, @timm and @neilernst think it makes sense, we'll provide the material by tomorrow, end of the day.

Thank you again.

@timm

This comment has been minimized.

Copy link
Contributor

commented Jul 9, 2019

@data-track-reviewer : please advise, when that new new material arrives from @alessioferrari , if this artifact now available and reusable

@data-track-reviewer

This comment has been minimized.

Copy link
Collaborator

commented Jul 10, 2019

Sure, I believe that would be a great addition. I will gladly check tomorrow and provide feedback/recommendation. I guess it is the decision of @timm and @neilernst , but feel encouraged by @timm 's comment...

One additional thought about the evidence: It is nicely summarized in the paper, so I am happy with the link. It could have been super cool to get a spreadsheet with the data from your evaluation. But, well, one can always wish for more. I do not see the availability badge in question.

@alessioferrari

This comment has been minimized.

Copy link
Collaborator

commented Jul 10, 2019

Thanks a million for the fast reply, we'll provide additional information on how to include the material in a course by today.

Concerning the spreadsheets: thanks a lot for the suggestion, we thought about that, but at the moment providing a full description of all the raw data in a way that can be easily reused would require a non negligible time, so at the moment we prefer to focus on the other material. As we're thinking about extending the work, we'll take this into account in the follow-up, thanks!

@alessioferrari

This comment has been minimized.

Copy link
Collaborator

commented Jul 10, 2019

@data-track-reviewer we have added the suggested information in the README file, and also updated the Zenodo record: https://zenodo.org/record/2625706#.XSXmRy17E60

We have also modified the status file, to notify that we apply also for the reusable badge.

We have also included a link to the YouTube video lectures corresponding to the slides.

Thanks again

@neilernst

This comment has been minimized.

Copy link
Collaborator Author

commented Jul 12, 2019

@data-track-reviewer please confirm the latest changes are acceptable

@timm

This comment has been minimized.

Copy link
Contributor

commented Jul 14, 2019

@neilernst : i'm a little confused. i understand our criteria for "available" and "reusable" but i think @alessioferrari is proposing a new a badge ""reusable for educational purposes"

@data-track-reviewer : as i understand your comments, this can be available and you are debating resuable. is that correct? if so, when will you decide.

@alessioferrari , we cannot invent a badge ""reusable for educational purposes** since our goal is inclusion to ACMportal and IEEExplorer and they only support the badges as we have defined the,

@neilernst

This comment has been minimized.

Copy link
Collaborator Author

commented Jul 15, 2019

@timm I reached out to the secret identity of our reviewer by email as well.

@alessioferrari

This comment has been minimized.

Copy link
Collaborator

commented Jul 15, 2019

@timm

This comment has been minimized.

Copy link
Contributor

commented Jul 16, 2019

I have reviewed this material and I agree with the authors that a standard reusable badge applies here. Therefore I state that badges of this artifact are

  • reusable
  • available

@neilernst : do you concur?

@timm timm added Available Reusable and removed 2.UnderReview labels Jul 16, 2019

@data-track-reviewer

This comment has been minimized.

Copy link
Collaborator

commented Jul 17, 2019

Sorry, I was traveling without internet. But I see that the discussion has developed according to my view. Also, I like the additions.

The badges I recommend are reusable and available.

@alessioferrari

This comment has been minimized.

Copy link
Collaborator

commented Jul 17, 2019

@timm timm closed this Jul 19, 2019

@timm timm reopened this Jul 19, 2019

@timm timm closed this Jul 19, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
5 participants
You can’t perform that action at this time.