Proposal: Alloy proposal process

[rfratto]

A design document for this proposal can be found at #909. All discussion about this proposal should be done on the proposal PR.

Background

Today, there is no formalized process for proposing changes to Alloy, and no formalized process for reviewing proposed changes. This has led to governance members being unsure how to get a proposal reviewed. This becomes exacerbated for external contributors who do not have a direct line of communication to governance members. The result of a lack of process can be seen by the number of open proposals in an unknown state; 47 at the time of writing.

At the same time, more and more proposals for Alloy have been performed offline. This has gradually drifted away from the design in the open mentality previously accepted by Grafana Agent. This has made it more difficult for external contributors to understand and influence the direction of the Alloy project.

Formalizing a process for creating and reviewing proposals will help ensure that:

• All contributors can contribute to the direction of Alloy.
• All existing and new proposals are guaranteed to be evaluated and decided on.
• A publicly accessible single source of truth is used for discussing all proposals.

Proposal

At a high-level, proposals go through the following stages:

1. Issue: A brief [issue][new-proposal] is created for the proposal.

2. Discuss: Public discussion on the proposal issue drives the proposal towards one of three outcomes:

   • Accept the proposal, or
   • decline the proposal, or
   • ask for a design document.

   If the proposal is accepted or declined, the proposal process ends here.

3. Design document: If requested, a design document is written.

4. Consensus: Once comments and changes have slowed down, a final public discussion aims to reach consensus for one of two outcomes:

   • Accept the proposal, or
   • decline the proposal.

A fully detailed design document for this proposal is available at #909, following the process that the design document proposes. As such, please keep conversation about this process here so there's a single source of truth for discussion.

[rfratto]

cc @grafana/grafana-alloy-team

[wildum]

Once comments on PRs for design documents have slowed down, the PR is merged
regardless of the current state of the proposal. More PRs to update the design
document may be warranted based on ongoing discussions on the proposal issue.

Just to clarify, that means that even if the proposal goes in Likely Declined following the discussions on the design doc PR, we would still merge it?

[wildum]

Comments on PRs for design documents should be restricted to grammar,
spelling, or proposal procedural errors related to the preparation of the
design document itself. All other comments should be made on the proposal
issue.

Makes sense but as I'm doing it, it's not a very practical workflow to quote things from the design doc in the proposal. Also we don't have the "start review" button where the comments are all pending till you finish the review. One way to work around this is to start a review in the design doc and then to copy over the comments in the proposal issue and delete the review in the design doc pr. That works but that's also not great.

I know that you mentioned this problem in "Conversation toil" but I would suggest developing further mitigation strategies:

Authors of proposals can start with offline discussions and use alternative tools (such as Google Docs) to prepare proposals for public presentation.

What about proposals started with github issues that spawn a lot of discussions? Should some part of it be extracted into google docs to be refined and added back to the proposal afterward? Should we link the google doc in the github issue?

The proposal review group keeps discussions on track.

What does this mean concretely? Should the review group make comments like "this topic is done, this one should not be discussed further, this one requires attention?" Comments on issues cannot be resolved, should the review group delete comments?

[thampiotr]

Every week, a subset of Alloy governance members review open and pending proposals as part of the proposal review process. The proposal review process may be performed asynchronously or synchronously, up to the discretion of the governance members.

Is every week necessary? Should it maybe be fortnightly or monthly?

[thampiotr]

Proposals are in one of the following states:

What if the proposal is on an issue that is not immediately urgent and there are more important decisions / work to do at the time. We may not want to rush the decision on Accepted/Declined status, but there is no way to "park" the proposal for future consideration in the process right now.

Similarly, if a proposal has been Accepted but not acted on for a long time, it may no longer be valid or useful and we should perhaps have a way to freeze it in some way too, so it can be revisited? Or should it go to Declined as Obsolete?

[thampiotr]

However, GitHub's issue system is not well-suited for long-form discussions. This may lead to long threads that are difficult to follow.

This is a very big issue IMO. Not only it's not well-suited for long-form discussions, it's also not suited at all for multiple threads of discussions on different topics. How do you resolve a discussion thread? How can a person revive a discusison thread? How many are currently open and need resolving before we conclude?

What makes it even harder is having a design doc in a separate PR, but mandating that discussion about it should go back to the original issue. I think this problem deserves its own paragraph in the "Rationale" section (which maybe should be renamed to trade-offs).

[thampiotr]

Overall, I'm not in favour of adopting this, with the main reasons being:

• It's quite a heavy and formal process. I believe that proposals can be made more open and reviewed regularly without the need for this extra friction, keeping it still informal while addressing the main problems.
• It's a lot of process to introduce all in one go. I'd prefer to see a more iterative approach where we make smaller changes and review how it's been working for us later.
• The lack of isolated discussion threads on GH issues makes it really hard to have a productive discussion and track what still needs to be addressed. Not to mention the historical record in the right context. Online docs (like Google Docs) are the gold standard IMO and I would prefer to use them instead.
• Separating the Design Doc from a discussion about the Design Doc is going to make things hard and messy IMO. This will ultimately not achieve the goal of greater openness, because it will become too hard to track these discussions and fewer people will be able to participate.
• Providing timelines on decisions - depending on the proposal, it may not be even necessary to rush the decision. We may not be able to meet the timelines or push ourselves to make a less informed decision as a result. I don't think it's a good idea in general.

On the other hand, the things I like about this initiative are:

• Having some simple review stages is a good idea.
• A board where we can see where the proposals are in the review stages is a great idea.
• Regular review of proposals is also a good idea - but I don't think we should be making it too formal or providing guarantees. At least not yet.

[rfratto]

I've updated the design document to address some of the comments here. I've left each change as a separate commit to hopefully make it easier to review the differences in state.

I've been reflecting on how we can optimize discussions on an issue to make them easier to follow. Alternative tools (PR reviews, Google Docs) typically encourage discussions to focus on individual lines or paragraphs of the proposal. I think we're seeing that approach not work for a long-form discussion on a single thread. Instead of having many sequential comments, we may find it easier to consolidate thoughts into as few sequential messages as possible; this can also identify opportunities to combine thought streams into a larger, cohesive piece of feedback to act on.

I've tried to follow a consolidated approach with this comment to address all the feedback above, though I found it useful to still categorize my thoughts for readability.

Proposal discussions

The main theme of the proposed process is for contributors to engage in discussion and work together to iterate on the initial proposal to identify an agreeable solution to the described problem.

If there is consensus about the problem statement being valid, discussion should lead towards consensus on how the problem gets solved. Under this approach, it's completely valid for the proposed implementation to completely change following discussion.

If there is not consensus about the problem being solved, or the consensus feels that the problem is not worthy of solving, then the proposal should move towards the Declined state.

Discussion will generally make the opinion of a commenter clear, and discussion slowing down becomes the measure of whether consensus has been achieved or whether a discussion should be revived so it can be resolved. New commenters can always revive a discussion if they have something new to add that would change the current consensus of a particular thought.

Proposal definition

One detail that I feel may have been missed is when the proposal process should be applied:

The proposal process is intended for all user-facing changes or any largely scoped internal change. Other changes typically do not need a proposal, such as minor code refactors. If a contributor is uncertain about whether a proposal is needed, it is preferable to create a proposal instead of jumping into a change that may be declined.

"All user-facing changes" includes requests that typically wouldn't be considered a proposal today, such as someone asking for inclusion of a new upstream otelcol component.

This is partly why the original proposal issue can be brief, but I do agree with @thampiotr that all proposals should include a summary of the background/reasoning for why we should do something. This is also why not every proposal needs a design document; wanting to include a new otelcol component doesn't need much more detail beyond brief what/why statements.

Subjecting all user-facing changes to the proposal process fixes some issues we've seen today: it's not always clear whether we would welcome contributions for something, it's not always clear when approval is necessary, and it's not clear how long someone should need to wait to get a green light on a request.

The number of proposals under this system (even if they're very small) is also why proposal review is weekly; weekly review will help keep the number of in-flight proposals small. Personally, I think weekly is too infrequent given our six-week release cycle, but I don't think shorter is feasible at the moment and if it were any longer I would worry about whether it's possible for a proposal to be approved and implemented within a single release cycle; this would encourage making constant exceptions to fast-track approvals.

Consensus

Another subtle detail from the design document is around the definition of consensus:

The goal of the proposal process is to reach general public consensus about the outcome in a timely manner.

This is fully open design, where we aim for general consensus from everyone involved in a discussion, not just those with commit access to the repository. Aiming for public engagement and consensus will help us build better software as we have more opinions and varying backgrounds to fuel our proposals.

Note that consensus doesn't mean unanimity; trolls and rare dissenting opinions do not necessarily prevent consensus from being reached. I would expect unanimity to be quite rare.

Public consensus also means that governance members don't have to always be deeply involved in conversations. Public discussion may reach consensus on its own (in either the approval or decline direction) without any time investment from the governance team or other maintainers.

In terms of priority, it's really up to the proposal review group whether there is capacity to move a proposal from Incoming to Active to signal the proposal review group getting involved in something. However, "capacity" should only consider the capacity to engage in a proposal; a proposal may be approved without being scheduled for implementation any time soon, making implementation capacity out of scope.

Other replies

Here are some other replies that I couldn't fit into the themes above:

Just to clarify, that means that even if the proposal goes in Likely Declined following the discussions on the design doc PR, we would still merge it?

That's right. Merging design documents regardless of the proposal helps keep a public record of what was considered. I'm open to changing this if you have specific concerns or thoughts about why the alternative (only merging design documents for approved proposals) is better.

Similarly, if a proposal has been Accepted but not acted on for a long time, it may no longer be valid or useful and we should perhaps have a way to freeze it in some way too, so it can be revisited? Or should it go to Declined as Obsolete?

That's a good point. I think it would be challenging to identify a strategy for this right now, and it's not clear how likely it is to happen. If this process gets approved, we can always keep an eye out for this happening and discuss/refine our proposal process as necessary.

It's a lot of process to introduce all in one go. I'd prefer to see a more iterative approach where we make smaller changes and review how it's been working for us later.

Sure, can you suggest what you're thinking for an iterative approach?

[rfratto]

That's right. Merging design documents regardless of the proposal helps keep a public record of what was considered. I'm open to changing this if you have specific concerns or thoughts about why the alternative (only merging design documents for approved proposals) is better.

@wildum Thinking this through some more, I feel inclined to change this slightly to either:

• PRs for design documents only get merged once the proposal has reached a final state (i.e., Accepted or Declined), or
• PRs for design documents only get merged for Accepted proposals.

While I think having a public record is what was considered is useful, the proposal issue already serves as a public record. I'm also worried that merging design documents before a proposal reaches a final state can lead to the author having to create multiple PRs to update the design document as discussion continues.

Only merging the design document PR once the proposal has reached the final state avoids authors from having to create more than one PR for their design document (when requested).

I'm still deliberating whether design documents for declined proposals should be merged; it might give people the wrong impression, and I can't identify any reasons why merging these PRs makes sense.

WDYT?

[wildum]

I like the idea of only merging the design docs for accepted proposals. When the code is not clear, it can be useful to search for a related design doc that might give some context. Declined design docs might bring confusion more than anything.

Overall I'm in favor of trying this process.
If we identify some pain points we can always refine it or go back to google docs.

One big pro that I did not see mentioned yet was the possibility of linking implementation PRs to the proposal. This ensures that the PRs follow what was accepted in the proposal.

This is fully open design, where we aim for general consensus from everyone involved in a discussion, not just those with commit access to the repository. Aiming for public engagement and consensus will help us build better software as we have more opinions and varying backgrounds to fuel our proposals.

I agree with this point. We have a small squad and a lot of ideas. It's not feasible to get fully involved in all proposals, especially when it requires a lot of background on a specific tech. Involving the community on this is a big +. It can also help to prioritize some ideas. I can imagine that some proposals in google docs that were abandoned could have been implemented if it would have had a lot of interests from users.

Some additional cons:

• Google docs has more options in terms of formatting and enriching the content (manipulating images, inserting chips...).
• You can share the google docs to a small circle before sharing it in the open.

[erikbaranowski]

Having to contain all discussion in a github issue like we are doing here is a blocker for me. I'm finding it hard to follow all the discussion above and I don't think it is a reasonable format. I would be a yes if we could have discussions in the design doc PR and resolve conversations like any other PR. I think linking to the design doc from this issue is enough of a trail for someone to find the discussions and edits.

[rfratto]

@erikbaranowski I'm open to changing the process so that proposals that need design documents are moved completely to PRs, so that all new discussion takes place on the PR and the original proposal issue is closed in favor of the PR. But because not all proposals need design documents, there would still be some purely issue-based discussions for smaller proposals. Would that solve enough of your concerns?

[tpaschalis]

After thinking back at how proposals were handled in the last couple of years, I'd also be in favor of trying this out.

Things I liked:

• Switching focus back towards the "design in the open" philosophy
• Having a frequent, public forum for triaging proposals. I've seen this at work nicely at the Go project (with its much bigger scale and complexity), and I think that it can help us get through proposals faster without that much overhead
• Having a separate home for design documents; design docs are not always required, and when they do
• I feel that the design-docs-as GitHub PRs (after potentially maturing in a Google doc) might work well

There are a couple points I'm not 100% sure about though

First off,

The proposal process is intended for all user-facing changes

I think that all is a strong word here and I'm worried that this might dissuade small changes that indeed would not benefit that much from going through the process or drive-by fixes by the community (eg. a new config field, fixing a typo or a small bug). I think that proposals should be requested from maintainers on a case-by-case basis, and that we should leave the room to fine-tune the process in the future, depending on how much toil it puts on the team.

Secondly, a proposal might have multiple potential solutions. It wasn't clear to me if they were intended to be presented on a single design doc each, if they should be separated from one another or if we decide on a case-by-case basis.

[rfratto]

I think that all is a strong word here and I'm worried that this might dissuade small changes that indeed would not benefit that much from going through the process or drive-by fixes by the community (eg. a new config field, fixing a typo or a small bug).

I see your concerns here, and I agree that bug fixes should not be subjected to the proposal process.

I think the proposal process solves one big issue we have today: it's not clear when we're interested in code for something, and it's not clear when you have to ask for permission. Forcing all features and enhancements (that is: anything that becomes subject to backward compatibility) to go through the proposal process would help fix this, but if that's too dramatic of a change I think we can do something more gradual.

As an alternative, we can make the following changes:

1. The proposal process is primarily intended for changes that would be (eventually) subject to backward compatibility, but it is not a requirement that such changes are originally introduced as a proposal.

2. Any existing issue can be turned into a proposal by adding a "proposal" label. This is something the Go team does that I didn't copy into our process originally, but I see the value now.

With this, we can continue to refine what we think should be a proposal, and add a proposal label after the fact. This will also help signal to readers when a request requires consensus and approval.

[mattdurham]

Is a proposal separate from an issue request, is it a replacement, or can a proposal happen within an issue?

Is a proposal an expectation that the proposee will do the change or is more of the ability to get an enhancement/issue promoted/finalized?

[erikbaranowski]

@erikbaranowski I'm open to changing the process so that proposals that need design documents are moved completely to PRs, so that all new discussion takes place on the PR and the original proposal issue is closed in favor of the PR. But because not all proposals need design documents, there would still be some purely issue-based discussions for smaller proposals. Would that solve enough of your concerns?

I'm comfortable moving forward as long as proposals that do require a design doc can be discussed on the design doc PR. I don't think it is necessary to close the parent issue if there is a design doc PR but am comfortable moving forward in either case. It is common in github that an issue may have some higher-level discussion and then a PR has more detailed discussion which is naturally organized when people decide where best to comment.

[rfratto]

Secondly, a proposal might have multiple potential solutions. It wasn't clear to me if they were intended to be presented on a single design doc each, if they should be separated from one another or if we decide on a case-by-case basis.

@tpaschalis This is a good call out, and I'll make this more explicit in my PR. I would a proposal with multiple potentially solutions to be handled in the same proposal issue or design document from the same author. In rare cases there may be two competing proposals from two different authors; we'll figure out how to best handle that when the time comes.

Ideally a proposal author does the legwork to discount alternatives and show their math in the proposal, but it's acceptable if a proposal author shows multiple solutions and discussion leads towards consensus on a single approach.

I think it's worth reiterating that the proposal process does two things:

1. Work towards consensus that a problem is worth solving, and then
2. work towards consensus on a solution towards that problem.

---

Is a proposal separate from an issue request, is it a replacement, or can a proposal happen within an issue?

@mattdurham Typically proposals include a suggested solution while requests only include the proposal. However, any issue can be turned into a proposal by adding a proposal label, so someone creating an issue doesn't have to start with the intent of it being a formal proposal.

Is a proposal an expectation that the proposee will do the change or is more of the ability to get an enhancement/issue promoted/finalized?

A proposal is a way to get consensus and approval on a specific solution to a problem; we don't make any assumptions about who will implement the proposal or when it will be implemented.

---

I'm comfortable moving forward as long as proposals that do require a design doc can be discussed on the design doc PR. I don't think it is necessary to close the parent issue if there is a design doc PR but am comfortable moving forward in either case. It is common in github that an issue may have some higher-level discussion and then a PR has more detailed discussion which is naturally organized when people decide where best to comment.

@erikbaranowski I'll make the change as I suggested; I am worried about fragmenting discussion and making consensus harder to measure, so I would currently prefer to move things to design documents once a design document is requested.

I'll make the change to my design document to mention this, and we can always revisit later if that turns out to not be necessary.

[rfratto]

To continue this proposal being an example of the proposed process, the discussion for this proposal is now moving to #909. Closing in favor of the design document PR.