New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Global idea: Improve detection of reviews that really need more comments #20

Closed
sebastientromp opened this Issue Oct 13, 2016 · 4 comments

Comments

Projects
None yet
2 participants
@sebastientromp
Contributor

sebastientromp commented Oct 13, 2016

Context: there is (and always will be) more people posting reviews than reviewers, so anything we can do to help reviewers focus their efforts on the reviews that need it the most will benefit everyone (the players because they'll have comparatively more reviewers available, and reviewers because they'll increase their chance of their comments being useful and appreciated)

More concretely, when going through the list of reviews I haven't looked at yet, there are two categories of reviews where I'm never sure if I should add my inputs on:

  • reviews where someone else already commented on and who seemed to give good advice. Should I look through the whole review myself to check the advice? Or should I wait until the OP reacts to bring another perspective? And if the OP never reacts, is there a point of spending time to add my opinion at all?
  • reviews where I see someone posted several reviews in a row, without much detail (which means they haven't spend much time walking themselves through the review first). Will my comments be read / appreciated?

Update: I wanted to start detailing more in details what the feature looks like. The idea would be to give each review a "score", that indicates how desirable (what this means exactly needs to be defined) it is to comment on the review.
The criteria I have in mind for now (in no particular order, and the numbers are used to show bonus/penalty, the actual value doesn't matter for now):

  • How old is the review? Score could increase for a while until a given time has passed, then decrease slowly until the end of time (assumption is that if no one commented on a review for some time, maybe the review just isn't interesting)
  • Has the OP done a pre-review? Could use the number of comments as an indicator
  • Are all fields filled? Use the presence of a decklist and tags for instance (small bonus)
  • Is it a win/loss? Maybe losses should be favored over victories?
  • Has there been other comments? If yes, maybe decrease the score, as there might be less need for comments?
  • Has the OP already responded on existing comments (either with a comment or with upvote/marked as helpful)? If no, it might not be urgent to add anything until we hear back from the OP
  • How many other reviews from the OP are open? We don't want someone to flood the front page with too many reviews
  • Has the review been closed to indicate no further comment is necessary? In that case, global score is 0
  • Has the OP good karma? Here karma is supposed to reflect the OP's "respect" towards the reviewers and how helpful he is to the community (ie upvote comments, mark them as helpful, answers / generate discussion, closes outdated reviews, and also use the global reputation score)
@sebastientromp

This comment has been minimized.

Contributor

sebastientromp commented Oct 13, 2016

Related: hide the entries that don't need more comments (or conversely showcase the ones that need more). Criteria to decide whether or not they need more comments could be:

  • If no sign from OP since the last comment (no upvote, no comment, no marked as helpful): the review doesn't need more comments
  • OP can flag the review as "resolved" -> no more comment needed. Also, need to send reminder emails to OP to flag a review as "resolved" or "still looking for advice" after some time
  • Anything with number of helpful comments?

Rationale: there is (and always will be) more people posting reviews than reviewers, so anything we can do to help reviewers focus their efforts on the reviews that need it the most will benefit everyone (the players because they'll have comparatively more reviewers available, and reviewers because they'll increase their chance of their comments being useful and appreciated)

@sebastientromp sebastientromp changed the title from Send reminder emails if OP has not upvoted / marked as helpful / commented on a review after N days to Global idea: Improve detection of reviews that really need more comments Oct 13, 2016

@sebastientromp

This comment has been minimized.

Contributor

sebastientromp commented Oct 13, 2016

Related: give incentives to OP to close / comment / vote on the reviews to help keep the review list as clean as possible

@sebastientromp

This comment has been minimized.

Contributor

sebastientromp commented Oct 13, 2016

Related: Send reminder emails if OP has not upvoted / marked as helpful / commented on a review after N days

@pjbingham

This comment has been minimized.

pjbingham commented Oct 13, 2016

Context: there is (and always will be) more people posting reviews than reviewers,

Indeed. Been pondering this one - without conclusion yet - for a while now myself. I continue to ponder...

Bascially, there needs to be a way to focus the volunteer attention on the most (1) worthy (e.g. becaue the OP has put the effort in too) and (2) valuable (to the OP, to the commentator, to the community in general, to Z2H, etc.) games posted for review. Guaranteeing attention is the function of paid coaching, so I think it is fine to have an approch to being selective and generating focus for the rest. Quality over quantity: critical, as quality of review is the site's current USP I think.

(This underlying thinking is what will be guiding my thoughts on what to do and form the success criteria to test against)

so anything we can do to help reviewers focus their efforts on the reviews that need it the most will benefit everyone (the players because they'll have comparatively more reviewers available, and reviewers because they'll increase their chance of their comments being useful and appreciated)

Yup - totally agree. Plus, attempt to define "need it most" in more detail is above.

More concretely, when going through the list of reviews I haven't looked at yet, there are two categories of reviews where I'm never sure if I should add my inputs on:

reviews where someone else already commented on and who seemed to give good advice. Should I look through the whole review myself to check the advice? Or should I wait until the OP reacts to bring another perspective? And if the OP never reacts, is there a point of spending time to add my opinion at all?

I agree this is a tricky one (and one I've wondered about frequently myself). We're balancing two conflicting goals: (1) ensuring as many reviews as possible get a response that helped (2) a tendency for reviews that have multiple participants to produce the best and most informative debate.

Is the answer simply to divide the time in a structured way? Thus:

  • 50% of the time finding and reviewing (worthy?) games that have escaped attention somehow
  • 50% of the time generating an actual conversation on games that have (say) currently only one reviewer involved

Not sure.

reviews where I see someone posted several reviews in a row, without much detail (which means they haven't spend much time walking themselves through the review first). Will my comments be read / appreciated?

I think it's fine to be hard-nosed about this. I myself review games where the reviewer went through it first before going looking for any others. Particularly with the automated uploader being a thing now, there's a lot more of this, so we need to focus on the ones where there's evidence the OP themselves is also focussing on it.

A useful (new) distinction to make might be between "upload" and "post". With the HDT plugin, there now exists (many) games that could be argued to be uploaded, but not actually posted. "Posting" a game, to me, means filling in all the fields, being explicit about any specific questions I have, adding the decklist, etc. (i.e. all the things that were true of what constituted a "good post" with the old comments system still apply) then finally making it public. Thus "posting date" is the point in time the game was set to "public".

So... given all this... I think we want to try to find games that are:

  • public (obviously)
  • have all the fields completed
  • have comments on the RHS (the reviewing section) by the OP themselves (indicating they've stepped through their own game)
  • have changes to the title - from the default one assigned on upload (a weighting, rather than criterion)
  • any other indicators that I've managed to forget right now that the review has had time spent on it by the OP - what others could there be?

Some (or perhaps all) of these are probably better implemented as prioritising factors rather than strict inclusion/exclusion criteria... indeed, it's probably better to develop a "need" algorithm rather than exclude anything (anything with a low "need" score would be implicitly left off simply by virtue of ending up at the bottom of a long list).

Regarding what the OP can give to help this process:

Related: hide the entries that don't need more comments (or conversely showcase the ones that need more). Criteria to decide whether or not they need more comments could be:

I agree this would be... essential. I think the aforementioned prioritised list should only include those that - in theory anyway - need attention.

If no sign from OP since the last comment (no upvote, no comment, no marked as helpful): the review doesn't need more comments

Not sure about this one... or at least it should take a while before reviews "expire" and drop off the list.

OP can flag the review as "resolved" -> no more comment needed. Also, need to send reminder emails to OP to flag a review as "resolved" or "still looking for advice" after some time

I think this is a good idea and useful addition to the review tools (including regardless of what else is ultimately decided on to implement)

Anything with number of helpful comments?

Or... both: "resolved" is defaulted to true if the OP marks comments as "helpful" (but he/she has the option to set it back to "looking for further input"), and the OP can set "resolved" to true even if they don't make any other changes.

(use case: I tend to mark comments as "helped me" if I learnt something actually new, whereas I tend to use the upvote feature for comments that are otherwise good (i.e. confirmed my thinking, generally positive, expanded on a point, etc. etc. So, it is possible for a review to not actually teach me anything new, but be good/quality/satisfying nonetheless - and it would be nice to be able to indicate so)

Related: give incentives to OP to close / comment / vote on the reviews to help keep the review list as clean as possible

Definitely. Further reputation points?

btw, is the reputation point system still fit for purpose after all the ways-of-working changes on the site?
(I think it is, but just checking...)

Related: Send reminder emails if OP has not upvoted / marked as helpful / commented on a review after N days

Need to avoid nagging ofc, but perhaps one reminder per game might make sense. I suggest it follows the same conventions (and respects the same settings) as the notification system in general however (i.e. it just becomes another type of notif).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment