Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Anonymous reviews #346

Closed
rixx opened this Issue Feb 5, 2018 · 12 comments

Comments

Projects
None yet
5 participants
@rixx
Copy link
Member

rixx commented Feb 5, 2018

While sometimes it may be useful to see speakers to know how knowledgeable a submitter is, it doesn't fit the workflow of many conferences requiring anonymitiy to avoid discrimination and talk choices based on speakers rather then content. Hiding the speakers from the reviewers should be a review setting, and I'd argue that anonymisation should be the default.

@MacLemon

This comment has been minimized.

Copy link
Contributor

MacLemon commented Mar 28, 2018

I've seen conferences greatly increase diversity in speakers and also accept more newcomers than before when anonymizing submissions for review. I support the motion to make this the default.

This would also mean that the reviewers must not be part of the orga team doing speaker communication to prevent any bias. (Somewhere between unlikely and impossible for smaller teams.)

@rixx

This comment has been minimized.

Copy link
Member Author

rixx commented Mar 28, 2018

At the same time, this needs to be optional, since in some communities you want to see the speaker to find out how knowledgeable that person is, or how well-spoken.

Submitters also tend to give unprompted identification in their talk notes, such as links to previous talks, or slides. These things are immensely helpful when assessing a submission, and at the same time prevent anonymisation.

@MacLemon

This comment has been minimized.

Copy link
Contributor

MacLemon commented Mar 28, 2018

Clearly requires double-blind-talk-reviews. :-P Just kidding!

@knarrff

This comment has been minimized.

Copy link

knarrff commented Nov 22, 2018

We are currently reviewing pretalx for use for the deRSE19 conference next year. One of the requirements is double-blind reviews. Thus:

  • This ticket suggests that double-blind reviews are currently not possible. Is this (still) the case?
  • If it is still a current issue, how likely/possible would it be to add double-blind reviews within the next 1-2 months (to allow for testing)?
@rixx

This comment has been minimized.

Copy link
Member Author

rixx commented Nov 22, 2018

I'm sorry, but as for all unpaid pretalx features, I can't put a timetable on this one. As I'm currently busy with organiser work for a conference in the end of December, I wouldn't expect this feature before January. But even then – no guarantees here, I'm afraid.

@knarrff

This comment has been minimized.

Copy link

knarrff commented Nov 24, 2018

I'm sorry. I am aware of the unpaid open-source nature. I didn't intend to ask for a timetable, more for an estimate of effort - possibly assuming someone not familiar with the code right now trying to do this.

@rixx

This comment has been minimized.

Copy link
Member Author

rixx commented Nov 28, 2018

Most of the effort will be in maintaining both modes of review: blind and regular, separated by a setting, across all methods of access, including the API, and figuring out what to do with people with both organiser and reviewer access. I'd probably take about 1–2 days of sponsored work to do this – as non-sponsored work tends to be broken up between free evenings and weekends, it would take about that much longer for me. I honestly can't say what it would be like for somebody who is not familiar with the code.

@vchuravy

This comment has been minimized.

Copy link

vchuravy commented Jan 10, 2019

@knarrff did you end up implementing double-blind reviews? I noticed that deRSE19 is using pretalx and I am evaluating it for JuliaCon.

@knarrff

This comment has been minimized.

Copy link

knarrff commented Jan 14, 2019

We decided for single-blind reviews, but not because of pretalx.

@laf0rge

This comment has been minimized.

Copy link

laf0rge commented Feb 5, 2019

One concept that I recently encountered is to do two rounds of reviews: The first one anonymous (to ensure there's no bias) and to rate the content of the talks. Then another round in "non-anonymous" mode to rate the speakers themselves (as some might have a known record of not being good speakers, or not being able to master the required language well enough, ...).

I just wanted to share this as input for a known existing usecase. I guess a per-event boolean flag would be sufficient, as one could then set that anonymous flag for the first round of reviews of the programme committee and later unckeck that flag for the second round.

@rixx

This comment has been minimized.

Copy link
Member Author

rixx commented Feb 5, 2019

Thank you! We're currently in the process of implementing the first part of proper double-blind review: just hiding every speaker information while this flag is set. Later stages should then include an editing mode allowing organisers to black out/remove personal information from submission texts.

I like the multi-round way of handling things a lot, but I'm not clear on a good user interface for switching rounds, presenting the reviews from different rounds, etc. Review rounds would be nice to have regardless of blind review, to allow for more selection stages.

@rixx

This comment has been minimized.

Copy link
Member Author

rixx commented Feb 9, 2019

Pretalx now is capable of blind reviews – reviewers can be restricted to be unable to view speaker names. People who have additional organiser permissions will still be able to see speaker names, though.

@rixx rixx closed this in 9fb2d6e Feb 9, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.