Join GitHub today
GitHub is home to over 40 million developers working together to host and review code, manage projects, and build software together.Sign up
Modify default Discourse weights #1681
This commit modifies the default weights in the Discourse plugin. The
Most significantly, we move over to fully like-minted cred, instead
Based on local testing on a few forums I'm familiar with, I feel
This commit modifies the default weights in the Discourse plugin. The overall theme is to make the plugin flow less cred to "raw activity", in favor of only flowing cred to posts where there is some explicit signal that they were valuable. Most significantly, we move over to fully [like-minted cred], instead of minting cred directly to posts and topics. Also, we remove the edges that tend to flow cred to posts indiscriminately. For example, topics no longer flow cred to every post within the topic. Based on local testing on a few forums I'm familiar with, I feel confident that these cred scores are an improvement over the current defaults, as we now have a few "real life test cases" of high-noise, high-activity users, and these weights reduce the amount of cred that accrues to such "stress testers". With these changes, I feel that we can start cautiously using the Discourse plugin in [Trust Level 2] communities. Test plan: `yarn flow` and code inspection are sufficient to verify that the new weights are technically valid. Because my calibration process for validating these changes involves subjective judgements about contributions from real people, I'm declining to publicly post any specifics; reviewers are welcome to reach out to my offline for further discussion. : https://discourse.sourcecred.io/t/minting-discourse-cred-on-likes-not-posts/603 : sourcecred/docs#24
Yeah, my main goal here was to remove all the activity-minted cred, and make the edge weights more spam-resilient. If Discourse is used in isolation, then the absolute cred minting per like is irrelevant; and I plan to come up with "cross plugin relative weight norms" later (after #1682).
Beanow left a comment •
We've debated why minting cred on explicit value signals are an improvement. So
I would definitely like to see a better explanation of the edge weights, or to split this to a second PR.
Significantly changing the cred flow topology by removing all of these backward edges, seems like an entire separate discussion and I'm not comfortable approving or blocking, as I don't have enough insight into what this will do.
wchargin left a comment
Most of the edge weight changes mitigate posts accruing “un-reviewed”
If I was asked to review: should we run this as an experiment? I could have given an answer.
Since that's not the question, I'm going with the PR is under-explained in this regard.
etc. etc. Worst of all, a lot of this seems like it can't be considered in isolation.
@Beanow, I appreciate your pushback and think you raise reasonable points. The underlying issue here is that right now, we don't have tools or processes for assessing the quality of cred scores, and thus we're flying blind when making changes to them.
This means that we should invest in cred analysis tooling, in repeatable experiments we can run on cred quality, etc. However, I don't think we should block changes to weights on developing this tooling.
The current discourse weights were chosen with very little forethought (the PR that added them (#1292) had 0 discussion of weight choices, and was just focused on unblocking the CredSperiment. These updated weights are more grounded, since they came about from a practical need to mitigate certain issues we're seeing in practice. We should avoid status quo bias, especially in cases where the status quo came about from a mostly random/arbitrary process.