Guide users to issue triage by automatically locking resolved issues and PRs after a set period of time

[ChristianMurphy]

Context

A source of frustration and friction in the contributing process to unified often comes from back and forth on issues and PRs, usually centered on understanding:

• what the author wants to achieve
• how to replicate the issue

Having as much of this context in the conversation up front, makes conversations more productive and pleasant overall.

The contributing guides, issue forms, and PR templates surface more of those guidelines earlier in the contributing experience.

Issue

One area which is missing this experience is existing resolved issues.
People find the issues though search, either external search engines or GitHub search.
They find issues which match keywords in an issue they are experiencing, it may or may not be the same underlying issue (more often not).
For existing threads, the GitHub user interface does not provide any of the context of the Contributing guide, Issue form, or Issue template.
Without that context users often post "same", without any context, leading to a lengthy and at times frustrating back and forth to get to the root of the issue.
Often a misunderstanding of the documentation or a new issue, rather than a regression of the original issue.

Potential Solution

After an issue (or PR) has been closed/resolved, wait a set amount of time, then leave a comment that future reports should create a new issue with the needed context and lock the resolved issue.
The set amount of time could probably be something between a day and a month to give some leeway for further discussion by the original issue reporter(s), two weeks feels reasonable.

This would:

• Still allow the original reporter(s) to follow up with questions or concerns
• Direct people experiencing an issue much later to go through the new issue triage process, to gather the needed context for a productive discussion.

Alternatives

1. Leave a comment, but don't lock the issue.
   • This would offer guidance for people who read the thread
   • This would do nothing for people who comment, without reading any of the prior discussion
2. Nudge GitHub towards a UI update
   • On PRs, Issue, and Discussions Github has a "It looks like this is your first time contributing in this repository!" modal, with links out to support and contributing documentation, it would be nice if something similar appeared for comments
   • Comment templates, allow maintainers to write comments or notes to the site to give context to new contributors
   • Finer-grained issue locking, like allowing previous commenters (even without write access) to continue to comment on the thread

Relevant Technologies

• Locking and commenting could be done through a GitHub action, such as https://github.com/marketplace/actions/lock-threads
• Locking and commenting could be done though an automation bot, such as https://github.com/unifiedjs/beep-boop-beta

[wooorm]

Yep, sounds good.

Why wait, say, a week? What would happen between that and a month?
Or even, 3 days? 1 day? 1 hour? Immediately?
Particularly as we are talking about issues (actual problems) and PRs here,
with phase/solved or phase/no applied? (and maintainers can still respond).
I think can could lock immediately, it’s done. Issues can always be unlocked/reopened/relabeled when needed

[ChristianMurphy]

This is asked by BB already: the release should be immediate or a maintainer should say when to expect it.
There is no need to ask for this

Gotcha

question on how to apply/test a solution, etc

Do you have a link to an example where someone asks this?

I'd look for some links.
What I'm thinking of, are IIRC there were some cases where features were added behind a flag and there was a follow up question of how to enable/apply the option.

The downside of locking far away is that locking changes the timestamp of the issue, I believe.

How so?
Looking at a recently locked issue remarkjs/react-markdown#696 the timestamps appear to still accurately reflect most of the discussion happened in 2022.

cron-tools or so runs daily to fetch all our repos, with issues/PRs with that label,
presumably we can know when that label was applied/comment was posted, and if that’s longer than X ago, remove label & lock.

That's roughly what I was thinking too, that it would need to be a new bot.
Either a global cron tools. or another scheduled action applied to each repo.

lock old issue in a repo such as remarkjs/remark as an experiment

Yep

evaluate, discuss, come up with how to improve things

Sounds good

The ancient remarkjs/react-markdown, or even all 4+ year old issues, might benefit from more exhaustive messaging:

Some inspiration from other projects:

• Compilation error when using plan-only mode hashicorp/terraform-cdk-action#2 (comment)
• How to use macOS built-in spelling dictionary? streetsidesoftware/vscode-spell-checker#2284 (comment)

🪦 Legacy: this issue is ancient. The advise here will probably not work.

Maybe after a few years, but if we lock immediately, maybe not calling it legacy quite so fast 🙂

[wooorm]

How so?
Looking at a recently locked issue remarkjs/react-markdown#696 the timestamps appear to still accurately reflect most of the discussion happened in 2022.

With the word “timestamp” I meant the particular timestamp associated with an issue/PR of when it was updated.
To illustrate, I went to issue 1 on reactjs/react-markdown and locked it. That moves it to the top of https://github.com/remarkjs/react-markdown/issues?q=is%3Aissue+sort%3Aupdated-desc.
This isn’t as big of a problem in 3 days. But it’s in my opinion a problem with 1 month.

Some inspiration from other projects:
Maybe after a few years, but if we lock immediately, maybe not calling it legacy quite so fast 🙂

These examples are good when we are locking new issues after 3 days, 1 week, 1 month, yes.
And so is that second point.
But this section was about 4 year old issues, which I hoped to discuss separately, as I believe they should be treated differently.
I think there should be a message about out of date info added to those ancient issues.

[ChristianMurphy]

With the word “timestamp” I meant the particular timestamp associated with an issue/PR of when it was updated.

Ah last updated, gotcha.

This isn’t as big of a problem in 3 days. But it’s in my opinion a problem with 1 month.

Could you expand on why it is a problem?

I could see if the issue was pushed out of the fold/off the first page of results that could be confusing.
But, taking a look at https://github.com/remarkjs/react-markdown/issues?q=is%3Aissue+sort%3Aupdated-desc as an example.
The first page of issues covers over a month of time, so even if there is some movement in the order it wouldn't hide the issue, and it would resolve itself in less than 31 days.

But this section was about 4 year old issues, which I hoped to discuss separately, as I believe they should be treated differently.
I think there should be a message about out of date info added to those ancient issues.

Gotcha that may be workable.
https://github.com/marketplace/actions/lock-threads for example has a configuration for exclude-issue-created-before which we could use to treat "ancient issues", however we end up defining that, different.

[wooorm]

Could you expand on why it is a problem?

It shuffles issues around. It makes the issues tab a bit harder to use as a human. It makes that date less useful.
I don’t worry about this being a problem if it’s in 3 days, 7 days.
I do imagine it to getting annoying at a 1 month. And more after that.

Another point is the comment + locking after X days. I think getting another notification for every issue after a month is a bit annoying.

I don’t really see a need for further questions / saying thanks after 30 days. Or 2 weeks. But like, 3 days should be enough for that?

The first page of issues covers over a month of time, so even if there is some movement in the order it wouldn't hide the issue, and it would resolve itself in less than 31 days.

Yeah, mostly good 👍 But we might also get more issues if everything old is locked!

github.com/marketplace/actions/lock-threads for example has a configuration for exclude-issue-created-before which we could use to treat "ancient issues", however we end up defining that, different.

👍

My idea on solving this currently is:a label bb/lock-candidate (?) being applied on phase/{solved,no} by BB from now on. And some cron-tools project to check for that label on all repos every day, get all association issues, and either a) if the label was applied 3+ days ago, or b) if the issue did not receive comments i 3+ days, then removing the label, commenting, and locking.

…hich would also not do anything to ancient closed issues, so we can treat those differently!

[ChristianMurphy]

I don’t really see a need for further questions / saying thanks after 30 days. Or 2 weeks. But like, 3 days should be enough for that?

Again, I'm good with most anything from a few days to a month.
If you feel that three days is a good starting point, I'm good with trying that.

we might also get more issues if everything old is locked!

Locking or no, the number of issues has been growing over the years.
Locking may add slightly accelerate the total number of new issues being opened, but the better quality in details, should reduce triage time and improve the overall discussion time in a way that leads to a net reduction of time spent.
I also don't see it accelerating to the point we have more than 25 issues in a month, for any given repo, for a while longer.

My idea on solving this currently is:a label bb/lock-candidate (?) being applied on phase/{solved,no} by BB from now on. And some cron-tools project to check for that label on all repos every day, get all association issues, and either a) if the label was applied 3+ days ago, or b) if the issue did not receive comments i 3+ days, then removing the label, commenting, and locking.

which would also not do anything to ancient closed issues, so we can treat those differently!

Sounds good to me

[remcohaszing]

Sometimes I find an old unsolved issue on a random repository and I want to post insights based on new developments. For such cases it’s always unfortunate if conversation is blocked. However, I don’t think this weighs against the annoying unrelated comments we get on some issues in the unified ecosystem.

[wooorm]

unsolved issue

Christian’s proposal is talking about resolved issues

[ChristianMurphy]

💯 agree @remcohaszing.
I don't want to close off conversation on unresolved issues and PRs either.
GitHub's tools for on-topic-ness of comments in an open conversation are limited.

Which is why this is scoped down to just resolved/closed issues and PRs.

[ChristianMurphy]

Newly announced but may be worth considering in the context of this.
Discussions can also be closed now: as resolved, outdated, or duplicated.
https://github.blog/changelog/2023-03-14-adding-the-ability-to-close-github-discussions/

If we opt to close discussions, would we want a similar locking mechanism?

My initial thoughts.
Discussions feel a bit different.
They are intentionally a bit more open ended, and can have more back and forth.
More often than on issues, I've seen the discussion authors post snippets of their solution, or other adopters reaching out to the discussion author for more information.
Those feel like fairly health functions of the community I don't want to squash those.

That said, the eldest discussions, particularly in MDX are starting to get outdated.
Referencing quirks and limitations of MDX 1, which don't impact MDX 2, and could potential confuse those who aren't familiar with that context.

Maybe after a year or two discussions get locked, and labeled as outdated?

Interested to hear others thoughts.

[wooorm]

I think that, being able to have very strict issues/PRs with forms, bots, and locking, is in part possible because discussions are very loose, chaotic.

We could try and regulate that chaos, perhaps. But I think that, as you say, will squash some of that.

Or leave it more alone. Which is currently where I am at!

[karlhorky]

I think that locking issues and PRs is a common practice in the industry, and resolves some surface-level issues for maintainers, but shuts down the possibility of some open discussion:

1. Reactions to useful comments in the thread
2. Links to useful new code or discussions related to a topic
3. Discussions related to the original code (if it is the current code in the main branch or the published package)

I've personally experienced wanting to contribute these things to a PR, and have seen others contribute in useful ways long after an issue or PR has been closed as both resolved or unresolved.

Making users create new issues for this (or, in many cases, just silencing their contribution) does improve the maintainability of open source repositories, but it has the potentially unwanted effects of:

1. Old issues / PRs becoming static and not having the capability of also including new information
2. A feeling by users that they are being blocked from contributing
3. Duplication of information in multiple places, making it harder to find + contribute to canonical sources

I think I don't have as much experience as any of the Unified.js maintainers in maintaining open source repositories, and I can't say how dramatic the downsides are with keeping issues + PRs open at scale. However, I could imagine that better muting, spam control and potentially banning could be used more effectively without any shutting of open discussion.

[wooorm]

There’s a time sink introduced for maintainers because people post the 1st step of rubber duck debugging in our forums.
E.g. https://github.com/orgs/rehypejs/discussions/63#discussioncomment-8499841 “how to make it work?” on a 3 year old issue that describes something that doesn’t happen anymore.
Or https://github.com/orgs/remarkjs/discussions/1051#discussioncomment-8491915, where people don’t spend the time to fill out the support form.
Or just now this: remarkjs/remark-gfm#9 (comment), completely unrelated comment.
These are things that people can mostly solve themselves if they fill out the form.
When they fill out the form, it becomes very easy to answer them. Without it, it’s a back and forth of “why” 5 times.

[karlhorky]

Agreed on "unstructured / invalid questions" - we deal also with this a lot, and even try to teach our students to follow a form of asking the questions: https://twitter.com/karlhorky/status/1754178237050048681

I think structured questions are good and necessary! I would not want to in any way harm that.

I'm not sure locking old resolved + unresolved issues helps much with people not following a structure though - I guess with an ideal solution:

1. Maintainers don't need to spend time on dealing with messages not following the format
2. Users post in the structured format
3. Users can get what they need

So maybe something like automated replies + muting + redirecting to new issues on "low info messages" would be the ideal solution - although I'm not aware of a solution for this out of the box.

To be clear though, these are not the types of positive interactions that I would want to avoid silencing.

[ChristianMurphy]

Thanks for chiming in @karlhorky!
Your perspective is appreciated.

Reactions to useful comments in the thread

hypothetically yes.
I haven't really seen this happen on issues or PRs.
This is more common on discussions, which we aren't considering locking https://github.com/orgs/unifiedjs/discussions/215#discussioncomment-5319089

Links to useful new code or discussions related to a topic

Again hypothetically yes.
I've seen references from new issues back to old ones, but haven't really seen the reverse in practice.

Duplication of information in multiple places, making it harder to find + contribute to canonical sources

I agree generally.
The way unified uses issues and PRs isn't really intended as a form of documentation.
And I'd like to keep it that way, scrolling through comments and code snippets of varying quality with only chronological sorting isn't a great developer experience.
The project readme and unified site are the most canonical sources.
After that, GitHub discussions offers an experience where answers can be upvoted and marked as correct to bubble useful information up.

I'm not sure locking old resolved + unresolved issues helps much with people not following a structure though

Nudging adopters towards opening a new issue brings them through the issue form, which is a more guided process.
It also gives the beep boop bot enough metadata to help with some initial triage https://github.com/unifiedjs/beep-boop-beta

So maybe something like automated replies + muting + redirecting to new issues on "low info messages" would be the ideal solution - although I'm not aware of a solution for this out of the box.

We do some automated replies (to new issues and PRs, not yet to comments) with https://github.com/unifiedjs/beep-boop-beta.
Determining "low info messages" and guiding them towards a positive outcome is a tricky business, which today requires a human.
Maybe eventually chat bots will get there, but I'm not aware of one which has enough understanding and safe guards that I would feel comfortable having it directly interact with contributors.

these are not the types of positive interactions that I would want to avoid silencing.

No one is trying to silence the community.
We're aiming to guide the community to positive interactions.
Right now, comments on old issues are a reoccurring and frequent source of poor developer experience.

Definitely open to practical alternatives which offer an even better experience.

[karlhorky]

Sounds good overall, I appreciate the thought that is put into making the processes and workflows better for both maintainers and the community. Thanks for taking my points into consideration.

Maybe there are possible middle ways in future to gain the DX benefits above while still avoiding locking.

[wooorm]

Yeah, thanks for the input. This is still an open ended discussion. I’m not 100% on where and how to go.

But there is a very real finite time I (and others) can spend on things. Most days there are a couple times I spend 5 minutes on an “I have the same problem, how make it work?”, which I can’t spend doing something useful or fun. It’s an annoying experience that adds up over months, years. Folks often mean well and sure deserve a nice response but it’s too much. A lot of it for me boils down to “do I want to spend the next 10 years dealing with these off topic posts or would I enjoy it more to live in a forest?”