Proposal: Issue commenting period

[nzakas]

Problem

Our open issue list is growing faster than we're able to close issues. We used to try to keep open issues under 100, and then we passed 200, and now we're well on our way to 300. I don't think keeping such a long issue list is of benefit to anyone -- the older issues rarely, if ever, get resolved, and they never get any other type of attention. Most of the open issues are stuck in the "evaluating" stage and only sometimes do we ever fully evaluate the request and decide what to do.

Proposal

I'd like to propose a formal "commenting period" for all issues (maybe 14 days?). The idea is that someone on the team needs to support an issue within the commenting period in order for it to be considered. Any issues that are still labeled "evaluating" after 14 days, and haven't otherwise been committed to (for instance, we've committed to the JSCS tasks even though many are still "evaluating"), we will just automatically close.

Why I think this will work:

1. If no one is willing to step forward for an idea within two weeks, it's unlikely that it will happen because the issue ends up pushed down the issue list and ends up out-of-sight (and therefore, out-of-mind).
2. We can give people quicker feedback on their ideas by closing things that haven't gained interest, rather than leaving them in limbo wondering what's happening.
3. We all have too many issues to review, and while I can't speak to everyone else's experience, I tend to focus on the newer ones because they tend to trigger notifications.

In the end, the open issues would end up as a list of things that have a very good chance of being implemented rather than a backlog of issues that are mostly ignored.

[ilyavolodin]

I agree. I would, however, like to propose maybe 3 week period of time. People go on vacation, or just don't have time to pay attention to issues every day, so 3 weeks seems like a reasonable amount of time to wait for feedback. I would also suggest that we go through and evaluate all issues that are currently in evaluating stage. It would not be fare to close them in bulk due to the fact that we didn't need +1s on the issues before.

[mysticatea]

👍

GitHub's reaction button does not trigger notifications, but for an example, 👍 reactions are intentions of support. I'm glad that the bot will handle such reactions.

[alberto]

I agree, the growing number of issues makes me a bit nervous. Something between 2 weeks and 1 month seems fine.

[nzakas]

My only concern about three weeks is that it's a long time to wait for such feedback (although, compared to forever, like now, it's definitely better).

Can people comment on what they feel is the best waiting period?

And agree @ilyavolodin about going through old issues. That's why I wanted to get this policy in place, so we can go close a bunch of old issues and point back to this discussion.

[alberto]

I mentioned 1 month because 2 weeks sounded a bit aggressive (not against it, though) and it's easier to do the math with 1 month than 3 weeks :). Also because it's the same timespan we have (in theory) for accepted issues in our maintainer guide.

For reference, the current timeframe in that policy to close an issue is 1 year.

[nzakas]

We don't have to do math, the days open is listed at the top of the issue. :) We removed the time-based constraints here because we never followed them: e7b1e1c

The 30 day waiting period was for accepted issues (which we can definitely do). What I'm talking about is for unaccepted issues.

My goal is to be as aggressive as possible so we all stay sane. :)

[vitorbal]

In my opinion, 3 weeks is an acceptable middle ground. I agree with @ilyavolodin that 2 weeks may be a tad too aggressive.
We could re-evaluate after some time, if we still see a lot of "forgotten" issues.

[IanVS]

👎

I've seen other projects do something similar, closing out issues after a certain period of inactivity. As a previous user of Sails.js, I felt disappointed and a little abandoned when they implemented their bot and closed out a bunch of old issues which were valid problems but hadn't had much activity (because at some point, once an issue is determined to be a bug, there's not much useful activity that can happen until it's worked on or fixed).

Personally, I don't see the problem with having old valid issues in the stack, and I would rather we lean more towards the side of keeping unnecessary issues around than prematurely closing out valid issues and therefore loosing visibility on them.

If we do go down this road, I think we need to be very careful and thoughtful about how we implement it, or we will quickly garner the ill will of many users and potential contributors.

Maybe a more gentle approach would be for our bot to periodically ping the OP and most active commenters to remind them the issue is still open, and ask them to close it if possible, leaving the final decision to a human. It could also add a label to indicate the issue is old and inactive. But, if an issue is still an issue, I think it does a disservice to everyone to automatically close it out.

[kaicataldo]

I agree with @IanVS that we need to be careful about how we go about this, as it could easily make our users contributing issues feel like their voices aren't being heard. I'd also be worried about closing viable issues that we just haven't gotten around to fixing yet.

That being said, it's hard to imagine that all our older issues will be resolved - I tried to go through and close out some of the older issues when I first started contributing, but at this point it's hard to know if the issues still need to be worked on and can be difficult to get the information we need to actually work on the issue when they were filed so long ago.

I really like the idea of a bot pinging on some of the older issues. What if the bot pinged after 2 weeks of inactivity as a reminder to take a look and then pinged once again after 3 weeks of inactivity to suggest closing the issue?

I feel like that could be a good way to make the process transparent and give fair warning to our users while still bringing our attention to issues that might get lost along the way and making sure all issues are not static and are moving towards a resolution.

[nzakas]

First, I think there's been a misunderstanding here: the bot isn't going to do any closing, we are. This is explicitly not a discussion about what the bot should do, it's about our overall policy. We can have a separate issue to talk about what changes, if any, the bot should have to support our overall policy at a later point in time.

@IanVS I think you missed some of my earlier points. The commenting period is for unaccepted issues - that means we haven't been able to make any progress on it. If it's a verified bug, it will be marked as accepted and wouldn't fall under this policy (though we should also discuss what to do about long-lived accepted issues as well). Bugs tend to be verified fairly quickly, so I'm not too worried about important bugs falling through the cracks. What we're really focusing on here are enhancement and feature requests that no one is paying attention to and are otherwise a distraction.

@kaicataldo

That being said, it's hard to imagine that all our older issues will be resolved - I tried to go through and close out some of the older issues when I first started contributing, but at this point it's hard to know if the issues still need to be worked on and can be difficult to get the information we need to actually work on the issue when they were filed so long ago.

This is precisely the point. No one feels like going back and triaging old issues, or tracking down the people who originally opened them, so they just sit there and gather dust. We're not doing anyone a favor by keeping old issues around when we have no intention of working on them.

---

All that said, the question we need to answer is:

What is the correct commenting period for unaccepted issues?

I proposed 2 weeks, @ilyavolodin countered with 3 weeks. It seems like there is more support for 3 weeks, so I'm happy to go along with that. Other thoughts?

[kaicataldo]

Understood - sorry for going off on a tangent. I'm 👍 for 3 weeks.

[alberto]

I was wrong about when relative dates become absolute so 👍 to 3 weeks

[IanVS]

Ah yes, I didn't read carefully enough and got spooked. Three weeks seems like a reasonable timeframe after which to close an unaccepted feature request or enhancement which hasn't had movement (or a bug that hasn't been reproducible or whose original poster hasn't been responding). 👍

[platinumazure]

👍 to three weeks. 👎 to any automated closing (but @nzakas has clarified that's not the case, just making my opposition to that clear).

[markelog]

I also had pretty frustrating experience with sails bot, although their team is not as active as ESLint but I think we should be very careful on how we want to handle this, specifically on who to phrase bot messages

[nzakas]

@markelog please see my note earlier that this is not about bots. :)

[ilyavolodin]

Quick question, what if the team couldn't reach consensus originally and the discussion is still ongoing after 21 days? Should we put something like after 21 days since opening and 5-7 days of inactivity?

[alberto]

Good point. It's not mentioned in the PR, but I assumed from the first comment that this was only to close issues that showed no support.

[platinumazure]

@nzakas Slightly beyond the original scope of the issue, but do we want to add anything to the policy regarding the reopening of closed issues (i.e., if there seems to be traction or if an ESLint team member wants to champion it)?

This may address @ilyavolodin's question, if we decide that the issue will be closed by date only but that an ESLint team member can reopen the issue if it seems like the process can move forward again.

[kaicataldo]

@platinumazure Mind expanding on what you think the benefit is of having another policy to have to remember when someone can just open a new issue (referencing the old issue if necessary)? We already have a policy for closing duplicate issues, which I feel like covers this situation:

Team members may close an issue immediately if:

1. The issue is a duplicate of an existing issue.

Totally understand your concern, but I think we already have a lot of policies in place and this just doesn't feel necessary to me. Just my 2 cents!

[platinumazure]

@kaicataldo Good point. However, I'm wondering if the intent of the duplicate issue policy should really be about any duplicate issue, or just a subset? For example, we should obviously close duplicate issues when the other issue is still open (user didn't bother searching), or when the other issue is closed and it's clear we're not going to implement the issue.

I'm not trying to add policies per se-- I'm just trying to make sure we don't permanently reject an idea that might get traction later. There might be other ways to do it. For example, we could use a "wontfix" label for issues we will clearly never do, and then either relax our duplicate issue closing policy (to allow new issues to come in) or allow reopening of issues that were not closed as "wontfix".

Perhaps a better idea might be to use a label for issues we closed due to lack of discussion (e.g., "inconclusive"), and then allow for either reopening or creating of new issues in those cases while rigorously applying the duplicate issue policy in all other cases.

I'm just throwing ideas out here-- not attached to any particular idea at this point.

[kaicataldo]

Gotcha, thanks for explaining. I think it's a valid discussion, though perhaps we could make a new issue for it?

Doesn't seem to me like pruning current issues due to lack of interest/activity has to mean we wouldn't be willing to look at a new similar issue that comes in at a later time. We're just trying to avoid a giant backlog of inactive issues that we'll most likely never get to.