JOINDIN-756: Re-requested: Being able to anonymize feedback [DISCUSSION] #722
Jira issue originally created by user liam-wiltshire:
This has come up again a few times recently - people requesting the ability to anonymize feedback submissions.
Until now it's mainly come from individuals, however we've now had it requested from an event organiser, so perhaps it's something we need to re-address?
Essentially, they've found that talks that are not as strong are getting fewer feedbacks, and as such those speakers who could probably benefit the most from the feedback are not getting it. Because they actively promote leaving feedback (running prize draws, not opening the free bar until a certain amount of feedback has been received etc), they suspect it's down to people not wanting to leave negative feedback with their name published.
As previously discussed, the main reason for requiring details is to help reduce the amount of inappropriate feedback, and also to (hopefully) guide people to provide more constructive feedback.
I've spoken to Beth about this, and she agrees it's probably needed from a conference/UG organiser point of view. The potential solution we thought of was to require people to have an account to post feedback, but to give them an option to hide their name when leaving feedback, making it clear that the event organisers and joind.in admins will still be able to see who left the feedback, but that no-one else would.
That way, the person posting the feedback knows they can still be held accountable (including the event organiser being able to deal with any CoC issues at their event that arose from feedback), but would hopefully still feel more comfortable in perhaps leaving feedback that wasn't overly positive as the speaker and other people reading it wouldn't know who it had come from.
The text was updated successfully, but these errors were encountered:
Comment created by dstockto:
liam - I think this was functionality that existed in the original site that maybe didn't make it over. Indeed if I go to the legacy site, and log in and go to make a comment I have the option of leaving my feedback either anonymously and/or leaving it marked as private so only the speaker can see it.
Comment created by liam-wiltshire:
david - yes, from past conversations my understand is that it was omitted from the new site intentionally for the above reasons.
I'm not a fan of private feedback, however I do think some form of anon. (built while making it clear that the event organiser and site admins will know who posted it, so you can still be held to account in terms of CoC etc) would be reasonable.
Comment created by e3betht:
As someone who leaves feedback for people, is a speaker, and is a conference organizer, I can see many sides of this issue. Here are my thoughts:
Honestly, though, I think having it published anonymously, but making it very clear that their identity would not be hidden from the Joind.in admins and the Event organizers (perhaps list them so the person knows their identities) would solve pretty much all issues presented here.
As someone who there is a non-zero chance was the original person that @liam-wiltshire 's comment may have been (request from conf organizer).... And someone who brought this up with @svpernova09 at one point as well.
I'll speak up. Right now I'm honestly unsure if joind.in is useful to me as a conference organizer. Because we essentially for the last year+, don't "actually" get reviews. I mean, yes. We do. You can look at some past events we've had, and every talk has 1-3 reviews, and randomly a few (keynotes) will have a couple dozen.
But they are 95% all fellow speakers rating other speakers. And 99% overwhelmingly positive. This change, and drop in usefulness started specifically back when Anonymous Reviews were removed from the website. "For a reason" as was mentioned above. I know that from past discussions, the reason was that some of the team at the time decided that anonymous reviews were bad, because they couldn't know who said something bad, because they couldn't reach out to that person, etc. Happy to chat a bit more in private about this to anyone that wants.
However, from talking with our attendees and trying to figure out why we don't get the piles of reviews we used to, it comes down to:
... So yes, overall I do feel that some solution is needed here. Whether accounts are required but anonymous ratings come back. Or if just open ratings get allowed. Or something. But the barrier needs lowered, and people need to be able to rate a talk without fearing their name is tied to it ... EVEN IF that means some features/benefits are lost.
To quickly respond to Beth's comments above:
RE: (1) Agreed completely, as you can tell.
RE: (2) I agree it's less useful (marginally). But it's better to have the feedback than not. I'm OK as speaker knowing I can't respond to people, but that I'll get more honest feedback. If someone doesn't want to be tied to their review at all. Then don't tie them to it.
RE: (3) While I understand the sentiment, I think that's a bad path to head down. As conference organizer you should have the right to remove any comments, especially if it violates the COC. However, again, I fear that any amount of concern that the organizer might contact you, or out you to the person you are leaving feedback on, immediately invalidates the benefits, and in fact can lead to reverse CoC issues. Where someone was anonymous because they were worried, and the speaker finds out and causes a CoC issue.
I really think that for the system to work properly, that 100% completely anonymous reviews need to be allowed. And the side effects of that be understood & handled separately (via moderation, etc)
Would discrete or private feedback be better? Feedback that is not publicly visible but only visible to the affected speaker, the conference organizer and the feedback giver?
The new account shouldn't be an issue by now as one can create a new account when logging in with twitter. Fast and easy solution that can (and will be) extended to more Oauth providers.
Fully anonymous feedback is not something we are planning, on the contrary the last discussion of the topic was very clear.
I don't think that helps. If you are saying that the speaker & organizer would see who the feedback is from. The problem isn't that the feedback itself is public (the rating, or the text). The problem is having that feedback be traced back to you.
Imagine that you are someone who has safety concerns (for many valid reasons). You want to leave honest feedback on the conference, but fear that the conference organizer might retaliate. You want to leave honest feedback on a talk, but have had bad interactions with that speaker in the past. In both cases, you want to be able to leave honest feedback, without having your name tied to it.
Insert many other reasons as well. (Being in a position where your negative feedback could lead to losing your job).
I do realize that yes, most online rating systems like this now-a-days require things to be public. But they, like Amazon, or iTunes for example, are setup where you are one of millions of people, leaving a comment on something that is produced by a company that has no idea who you are, or could care.
Conferences are much more intimate situations, and with small communities. At a conference of 200, people know who you are much easier.
Again, FWIW, just giving the feedback here based upon past experience. As a conf org, joind.in was much better at getting real-honest reviews of talks, and getting lots of reviews, when people were able to leave anonymous ratings.
When the change happened, fewer and fewer reviews started happening. To where now the audiences eyes glass over when I mention using joind.in. And all(mostly) reviews are 4-5* reviews by fellow speakers. Which isn't actually useful feedback.
I've had talks that based on private feedback that should have gotten a 1* ... that has 3-4* because of people being polite in public.
#2cents - And totally up to the joind.in new team to figure out. I dunno how you've set up yourselves at the moment for decision making :). I've always loved joind.in as speaker&conference organizer. But currently, I'm sitting here with an event 2 weeks away, and debating the idea of putting little paper talk rating slips in chairs instead of using joind.in . :(
That's awesome to hear! However the website doesn't reflect that reality if so and needs some verbiage update? The homepage says you can login if you already have an account (and gives user/pass, Twitter & Facebook options). And then says if you don't have one to click 'Register Now'.
That page goes through a full account setup process.
Nowhere does it suggest that you can just 'login with Twitter' without first having an account setup.
UPDATE: I went to test this. And it didn't work. I attempted to log in with Twitter, to a different twitter account than my usual, without going through registration first. And it gave me a 'Failed to Log In'. Is this a new feature being rolled out at the moment, instead of an existing one?
I was not involved in any last discussion. So I was unaware of the details there, and bringing up some points towards it. :)
If fully anonymous feedback isn't an option. Then it may not be safe in some ways for people to use joind.in, and that may lean towards more events not being able to use it.
IMO, would be awesome to have it be an option. Even if a toggle for organizers to choose. Probably with moderation intact regardless. IE: Purely anonymous feedback & ratings would not be automatically shown or added to a talk. Until reviewed by the organizer to make sure that it's not spam/CoC-violation/etc.
(Honestly, having a 'moderation' toggle in general could be an interesting feature)
The thing is that joind.in shall be safe for the speakers getting reviews. And sadly not all people giving "honest anonymous feedback" were playing by the rulebook. So sadly we had to turn off that feature due to some people misbehaving (why can't we have nice things?). If I want to give honest feedback to the speaker I am able to defend it. Otherwise it's not honest feedback. If it's feedback for a speaker I already have a brawl with, I either give it and stand up to it or do not give it. I can understand that people do not want their feedback to be visible publicly but at least the speaker I'm criticising needs to know whom to contact to get more information especially about constructive feedback. And I used this word very carefully. Because there is no negative feedback. "The talk was shit" is not negative feedback. It's no feedback at all because as a speaker I don't know what part was shit, what the person thought was shit, and most important of all, how to improve. Because that's what joind.in is about (as far as I am aware of). It's not about giving 5 stars, it's about telling the speaker how to make the talk better for the next iteration. Which brings me to the topic of "rating without comment". That's not what the speaker need. At least not in it's current form. We've already talked about that in the team whether it would be a possibility to differentiate that more. Bunt in general this applies...
I do like the idea with the moderation step though! When the event organizer steps in and makes sure there are no CoC-violations in the comments that might make it possible for us to allow anonymous feedback. We will have to discuss that. It still doesn't help the speaker in getting more details on how to improve from people that gave anonymous feedback...
Regarding the Twitter/Facebook login: Sorry, I had something different in mind there and seem to have confused that with something else. We should be able to make the "profile creation" process as painless and unintrusive as possible to allow easier adoption. So having a "one click login via social network and account creation" would be preferred here. We seem to have to do some work there. But this is already talked about in #388
Thank you for your feedback!
Happy to keep giving feedback, and to potentially jump in and help provide code as well in future. Feel free to also just say at some point "Thanks Eli, we've had enough feedback from you, you aren't our target audience", and I'll move on.
I think that we have a major mental disconnect here. Which may lead to some of our back and forth.
You say "should be safe for the speakers". I'd like to hear from you exactly what aspect you feel is unsafe here for the speakers?
I assume, based upon past conversations/experiences. That the un-safe aspect for speakers, is purely a "Someone who didn't like me personally went and rated all my talks as 1*". IE: 'Unfair' ratings. Which is completely different from 'Unsafe'.
Does getting unfair ratings suck? Sure does. Can that be handled by admin moderation? Sure can. Is it possible for that 1* to 'feel' unfair when in fact it was warranted? Definitely.
The only truly 'unsafe' aspect I can see here right now for the speaker, would be someone attacking them / CoC violations. Which can happen anonymous or not. And that comment/review should be allowed to be deleted/moderated, anonymous or not.
95% of my own comments above have been focused on Rater Safety. And all of those concerns are 100% real safety and/or life-quality concerns. Concern over retaliation. Concern over job security. Concern over community interactions.
To be extremely specific, you said above:
And that, is 100% the problem. If the speaker knows whom to contact to get more information. And if I, as the theoretical reviewer, am worried that the speaker could cause me to lose my job, or ruin my life, or literally show up at my doorstep with a gun. Then there is no way that honest review can be left.
Yes, I'm speaking in extremes, but these are real safety concerns here, that in turn leave people to not leaving reviews. Now take it down to the non-extreme ...
I attend a talk. The speaker was horrible, the presentation was horrible. There were multiple issues. However I don't know the speaker. I've had zero interactions with the speaker. Maybe I got bad vibes from the speaker. How do I know that if I leave a scathing review, because these reviews ARE personal. Not like "This toothbrush was cheap and broke". That it's not going to be taken personally, and therefore, that I'm going to find myself an hour later with that speaker breathing down my neck and screaming "Fuck you, how dare you rate my talk like that!" And causing a situation.
Why should I risk that situation? Why should I, just to give honest feedback to the speaker, and especially to the conference organizer. Have to potentially expose myself to a situation.
That's what's happening. And so instead. You see positive feedback only, or negative feedback only in the cases where it's someone in a positive of privilege, where they feel safe leaving the negative.
(FWIW: I'm arguing the bigger points at the moment, and in absolutes. But none of this is that. Having, for example, the old method of "Anonymous but Conf Organizer & joind.in stuff" can still see who it is. Is of course better than not having any anonymous options at all. If compromise is needed).
That being said. I think if the bigger issues of removing the barriers to entry, to just like people 'click rate move on' aren't solved. It may not matter. If only 3% of the attendees can be bothered to make an account to leave a rating. Then it's not going to be super-useful :(
I can perfectly understand your POV Eli. But why am I so obsessed with speaker safety? For one thing, bad things have already happened and the speaker can't do anything against it. Currently the review is not moderated (no resources apart from AKISMET) so an unqualified review hits directly through to the speaker without them being able to do anything against it. And that is – from what I've heard – not a hypothetical issue but already happened. Yes, you are right, no one hinders you to give that feedback as a known user. But the chance for people to do so when they know that they are known is much less likely and then they can be banned after the first time. Not perfect but OK.
The rater on the other hand can avoid having bad things happen by not rating. So they have it in their own hands. Yes. at the cost of not giving a review at all and therefore reducing the number of reviews. So I'm absolutely happy to find ways to bring both views together. And IMO your idea with the moderated reviews is a really good one!
On the other hand, educating people how to give constructive feedback might help them in not writing a scathing review. Because such reviews are not the idea of joind.in as far as I see it. It's about helping the speaker to become better. So pointing out what can be improved and rating with one star is helpful and still shows your frustration. Yes, that does not hinder a frustrated speaker to breathe down your neck and cause a situation. And that's why I think the idea with the moderated review is a good idea. Perhaps even a flag for logged in reviewers "I want to stay anonymous to the speaker" so when the event-organizer accepts the comment the speaker will not be able to know who gave that comment. At least as long as they don't bribe the event organizer to tell them the name....
All in all the main thing is to realize that negative feedback is not helpful to the speaker and therefore won't help anyone. And just to be clear: "I didn't like the talk" is no negative feedback whereas "the talk was shit" is. The former is a personal statement whereas the later is a generalization that very rarely is true.
First of all. Totally get the differences between negative vs not, and constructive vs not. I agree that getting better constructive feedback is much more useful to the speaker. Getting a "1* this was shit", is not at all.
However I still believe that we have two fundamental differences here in what we are looking at.
Back on Speaker Safety
I completely understand what you are saying. Yes, a speaker can get a bad review, that is unwarranted. And that sucks for the speaker. But that pales in comparison to the reviewer side of "If I leave this rating, my life might be in danger".
Those are very very different ends of the spectrum. I'm focused on making it safe for people to leave reviews. You seem focused on ensuring the speakers feelings aren't hurt. Now I may be completely missing some speaker-safety aspect here. I'd love to hear about it if so!
Back on Purpose
I believe we also are looking at joind.in from different POVs as users as well. Given the "rating without constructive feedback isn't helpful to the speaker" type comments, and how if the speaker can't reach out to have a conversation, how will it help the speaker ... I totally agree with that!
However, that's totally looking at this from a speaker pov. But as a speaker, you'll get feedback via whatever method the conference organizer chooses to use.
To that end, if joind.in doesn't give conference organizers the tools they need, in methods that work for them. Then conferences won't use joind.in for talk reviews. And it doesn't matter at all to the speaker.
From my POV as a conference organizer. Joind.in isn't giving me useful feedback. As a conference organizer, having 'non-constructive bad reviews' ARE still important reviews. If a talk was really bad. I want to see dozens of 1's and 2's. I want to see what people have to say, regardless of how harsh. That's all important feedback.
Having every talk be a 4-5, and only positive reviews. Doesn't help me. As I have to hear through the grapevine that there was a problem. I might even have people asking for refunds because a situation was so bad. And yet, the talk is 4.5* with 'Great talk!' reviews on it.
So yeah. Very different priorities. You are focused (for good reasons) on making joind.in a great speaker feedback tool to let speakers become better speakers.
I agree with those goals. But right now I'm trying to suggest items that will actually get more conferences to want to use joind.in, instead of leaving it.
:) Thanks again for the chats
I'm on the fence on this one. I believe at some point that someone has to be able to see the user, even if that's just Joind.in admins to prevent abuse (e.g. we might need to take action against a particularly abusive user). So that's a sticking point for me, but those people should be trustworthy enough not to abuse the use of that information.
Another point to consider would be how would we deal with a user who just follows a speaker around leaving bad feedback? Currently there is almost validation in that they are public it would become quite obvious to anyone looking to see what was going on and they could even be called out that they weren't at certian events.
On the otherhand, I entirely agree with @EliW's points about at the moment Joind.in is a tool that is not very helpful because the honest and negative feedback is rarely posted. Other events like DrupalCon collect feedback on sessions and numbers but that data is never given raw to the speakers. It's used by the event organisers who might sometimes aggregate it for speakers (depending on the event).
So perhaps we could do something along the lines of anonymity as well as perhaps something along the lines of posting 'private' comments which can only be read by event organisers but the ratings (count, average and median) would be included in the overall public aggregate. If then a speaker wants further feedback on if there were things to improve in private comments, they can always speak to the event organiser who can view all the comments, which may or may not also be anonymised.
I think I also agree that we should enable ratings without comments to lower the barrier to entry. We can always encourage comments with a note on the comment page.
As for authentication, I think the solution there is make it simpler (e.g. adding other openID logins e.g. github and google ) and merge login + registration into one flow so they just have to click a button and it's done.
@EliW While I can completely see and understand your POV on the rater safety, I think all of those same points you are making apply to the speaker too, except that the speaker has no way to keep themselves safe from this because the conference posts their talk (with or without them claiming the talk or even without their permission most times), and then they are at the mercy of the conference organizer or finding a Joind.in staffer to try to help them if something goes wrong. And ratings can affect a speaker's safety, job, personal reputation, etc. perhaps even moreso than a commenter because future employers are not checking what joind.in feedback potential hires have given, but they are checking what future hires have received for their talks. It also affects a speaker's ability to be selected for future conferences. I also hope you understand that this situation was already happening. I have had people leave completely inappropriate feedback on my talks, some threatening, some commenting on my physical appearance, some sexual comments, people rating me 1's because women shouldn't be allowed to speak from stage, etc. I have been lucky in that I have had organizers who cared about these situations and were quick to act on them, but not everyone is that lucky. Racist, sexist, threatening comments discourage speakers from speaking again in the future, and I can attest to the fact that they make us feel very unsafe even continuing to attend the conference where these comments occurred. These comments have overwhelmingly been anonymous because people feel free to be trolls when they have no skin in the game.
@michaelcullum - I agree with lowering the barrier of entry, but getting more feedback is only helpful if it is useful feedback. What if we compromise and allow 4 and 5 star ratings with no comments, but if you rate 3 or less, you have to leave a comment? Low ratings with no comments are completely useless and just damage the speaker's reputation without them having anything they can fix or improve or even respond to.
On your notes about 'These comments have overwhelmingly been anonymous because people feel free to be trolls when they have no skin in the game.' I feel this is where we should have (a) a 'report abuse' button and (b) this is why I think it's essential that someone (aka joind.in staff) can see the poster and login would still be required, even if their name was not publicised.
After some discussions on discord we're going to split this into a few different tickets as we feel there are a series of different features here to help solve the engagement and value problems being highlighted. I'll create them over this weekend, in summary, though they are:
To clarify 4 and 6 (although I'll go into more detail in the ticket), the idea might be to do something vaguely along the lines of someone does a 1 star but doesn't leave a comment we'd show them this (obviously specific causes to be discussed and it would be worded a lot better):
Attempting to do lots of replies in one post. Apologies if this method fails but trying to do quickly:
Thanks @michaelcullum - I do think this would help.
That would be wonderful
You are correct @e3betht, and that sucks. But there is some truth in the fact that the speaker choose to speak, and has exposed themselves somewhat more publicly to begin with. They made the decision to speak. However, looking at your other points:
This is all very true. From my POV, this however was part of the original point of Joind.in. To make that information public. Basically the 'Facebooking of Conference Reviews'. That before joind.in this information was all privately held, and so didn't help anyone, positive or negatively. And that the goal was 'free the data'!
If there is a concern from some speakers, that having negative reviews on joind.in may negatively affect them. Then honestly joind.in shouldn't exist IMO. Since the entire point of joind.in is that it does in fact make reviews public. If they aren't, then it doesn't help future conference organizers know if someone was a good speaker or not in accepting them for example.
Perhaps joind.in should go to being a non-saas if this is a concern. And just be a rating tool for conferences to install/use themselves. But then, there are already solutions for that. It's definitely a double-edged sword. You can't have the benefit of showing off the awesome speakers. Without the drawback of showing off where a speaker stumbles. (since that is a benefit to other people)
That is in fact horrible, and is something that has to be stopped. My argument is simply that stopping anonymous reviews, just because these happen while being anonymous - is not IMO the right answer. That's like banning all cars, because people have car crashes. Instead my argument is that we still need the benefits of anonymous comments.
But yes, we need moderation, and/or other tools for conf organizers & joind.in admins, to handle these cases. Removing the offending comments (and potentially reporting the offenders in whatever way is possible). Anonymous comments don't have to mean 'un-deletable'. Nor do they have to be 100% anonymous, as they could still be visible to the site admins, and potentially the conference organizers.
It's not useful to the speaker. No. But it's useful to the conference organizer. Extremely so. As yes, Michael correctly interpreted.
@EliW - I’m not talking about speakers needing to only get positive feedback. I think there is somehow a disconnect in the kind of feedback we are discussing. I think it is essential to get constructive negative feedback, but we have solid, data-driven evidence that when we allowed anonymous comments, it caused a big problem with harassment and threats. On the other side of this issue, we have a hope that allowing anonymous feedback would encourage more people to leave constructive feedback. As a conference organizer, I completely and fully understand your need to get feedback on your event and your speakers. However, we have no proof that allowing anonymous ratings will help this problem, and we have concrete evidence that it causes major issues. As a conference organizer, I’d also disagree that getting a one rating with no comments is still helpful. Without context, that rating is useless. Did the the attendee give them a 1 because the talk was bad? Or because they hate WordPress? Or because the speaker was a non-native speaker and the attendee is racist? Or because there were AV issues and they couldn’t hear the speaker well enough? Or because the talk was way over their head? Or because they thought the talk was way too basic? Without this kind of context, the only option is to just assume the speaker is bad at what they do, but in many of these instances, that is not the truth of the situation.
One last note - I should clarify that anonymous commenting means that no one knows who the commenter is. In your most recent message, you mention that anonymous comments could be non-anonymous to site admins and potentially conf organizers. For most of this thread, we were basing our responses on our earlier comment that you wanted completely anonymous commenting, where site admins and conf organizers would not be able to tell who they were. I am fine with someone not wanting to post their name publicly with the post, as long as site admins and conference organizers are still able to view the comments and handle harassment and discrimination issues directly with the commenter.