proposals: add release-approval-process

[philips]

This is a proposed process for approval of new releases of
specifications and projects from the OCI.

The creation of this process is designed to clarify how a release gets
created and who needs to sign off.

[wking]

”e..g” → ”e.g.”

[crosbymichael]

lets chill out on the nits until we look at content

[mrunalp]

Maybe a different criteria for a major vs minor release will help. For e.g. 0.1 should require LGTM from half but 1.0 should require 2/3 LGTMs.

[wking]

The last sentence reads a bit strangely around the MUST to me. How about replacing the last two sentences with:

A release is approved a week after the announcement if at least two-thirds of maintainers LGTM the release. Minor and point releases MAY be approved earlier if all maintainers LGTM the release.

And that also gives space for @mrunalp's different thresholds.

[wking]

On Thu, Jun 09, 2016 at 01:48:30PM -0700, Mrunal Patel wrote:

Maybe a different criteria for a major vs minor release will
help. For e.g. 0.1 should require LGTM from half but 1.0 should
require 2/3 LGTMs.

What about 1.1.0? The charter's only Member out for “I don't like
that release” is “then you can quit” 1. If that applies to minor
and point releases (and in the absence of wording to the contrary I
expect it does), then I think we want to be cautious about them as
well.

 §8.d

[stevvooe]

A release should be preceded with a release proposal, giving reasoning about how the release meets the requirements of the release. If any requirements for the release are not met, but the proposal is to proceed despite the lack of met requirements, this should be called out.

[anuthan]

I would also request that we get a LGTM from one of the platform specific parties for "Making a release". As in atleast one LGTM for Linux , one from Windows and one from Solaris or whoever is contributing to the specified project. We can keep a time limit for this response as well.

[wking]

On Fri, Jun 10, 2016 at 12:17:22PM -0700, Abhijeeth Nuthan wrote:

I would also request that we get a LGTM from one of the platform
specific parties for "Making a release". As in atleast one LGTM for
Linux , one from Windows and one from Solaris or whoever, is
contributing to the specified project.

I don't think there's a need to call that out. Platforms who wish to
be represented should have maintainers they trust (and/or employ) on
the project, and expect the maintainers to respect any per-platform
concerns raised (just as concerns from anybody should be respected,
#17).

[philips]

I agree we don't need to call this out. Maintainers should be trusted to make the right call, and platform maintainers should make their case to the relevant channels. It is pretty clear how this stuff should work based on the OCI gov. docs 5.d: "The primary means for any organization to influence the technical direction of the OCI Projects is via contribution or service as maintainers. "

[crosbymichael]

Content looks good so far but it is missing any type of quantitative criteria that tells when an release is ready or not. It does not talk about testing phases, feedback, resolving outstanding issues, etc.

We need something that we can measure because if a maintainers that rejects a release have to state their reasons why then the maintainers whom approve must also have valid reasons to cut a release.

i.e. "we have solved the large schema issues for the specification, no other schema changes are required and there are no outstanding bugs based on feedback from implementations, its safe for a release."

[wking]

So if a project with seven maintainers gets 5 LGTMs and 2 REJECTs, it's rejected? If so, I think you want to reference this rejection exception in the “Making a release” section.

[stevvooe]

There are lot of problems with restricting this to GitHub issues. What state should they be in? It calls into question how they are closed or why they were closed. The other issue is that GitHub issues can be edited and comments can be deleted. I am not sure if it is healthy to politicize the state of a GitHub issue.

Allowing the medium of rejection to affect the validity increases the chance of deflection, especially when we have multiple communication channels. A rejection should be a rejection and the reasoning can be examined afterwards.

[wking]

On Thu, Jun 09, 2016 at 05:07:44PM -0700, Stephen Day wrote:

+Rejecting a release: A project maintainer MAY choose to reply with REJECT and MUST include a list of concerns filed as GitHub issues…

There are lot of problems with restricting this to GitHub
issues. What state should they be in? It calls into question how
they are closed or why they were closed. The other issue is that
GitHub issues can be edited and comments can be deleted. I am not
sure if it is healthy to politicize the state of a GitHub issue.

I agree, but I don't think the current wording politicizes the state
(it just says concerns must be filed as issues). The current
wording has the vote for tagging as the only gatekeeper, and that
makes sense to me. The issues and milestones are just for building
consensus before a vote (possibly the next vote).

[wking]

On Thu, Jun 09, 2016 at 01:52:06PM -0700, Michael Crosby wrote:

Content looks good so far but it is missing any type of quantitative
criteria that tells when an release is ready or not. It does not
talk about testing phases, feedback, resolving outstanding issues,
etc.

Do you have to talk about that? Isn't it up to the maintainers to
have a sufficent handle on the pulse and REJECT if they feel
concerned?

We need something that we can measure because if a maintainers that
rejects a release have to state their reasons why then the
maintainers whom approve must also have valid reasons to cut a
release.

I expect this will be covered by the release notes / ChangeLog entry.
Maybe just require that to be posted with the initial
release-intention announcement?

[wking]

On Thu, Jun 09, 2016 at 02:11:07PM -0700, W. Trevor King wrote:

I expect this will be covered by the release notes / ChangeLog
entry. Maybe just require that to be posted with the initial
release-intention announcement?

This is not a big ask, because until you've compiled the release notes
/ ChangeLog entry it's hard to tell if you should be making a point,
minor, or major bump.

[caniszczyk]

@crosbymichael want to add a section around release criteria then? we can iterate from there... I assume you want to include things such as at least two implementations and so on... but one way to look at it is release criteria is essentially everything defined in a milestone (e.g., https://github.com/opencontainers/runtime-spec/milestones) but if you want it to be more concrete than that, go for it! :)

[crosbymichael]

@caniszczyk yep

[philips]

@crosbymichael @caniszczyk I didn't put i criteria because I think they should be developed through discussions on the dev@opencontainers.org posts about releases and meetings. It should be the responsibilities of the maintainers to define their criteria through those discussions and reflect them in issues tied to milestones.

[philips]

@crosbymichael @caniszczyk if that makes sense I am happy to add some additional language into this doc.

[wking]

The content and rules in this document are basically an extention of
the current maintainer's guide PR-landing rules 1 to milestone
assignment and tagging. Should this text get moved over to a
project-template PR updating the maintainer's guide? Should all/some
of the maintainer's move over to this repo? If we split rules about
maintainer workflow between the two repositories, what are the rules
for what goes in each repository?

Of those choices, I prefer shifting this PR to project-template,
because that sets up the per-repo maintainer's guides that have worked
well in the past, and you don't have to give new maintainers a way to
discover this repo and its docs. It doesn't really seem like
something that the TOB will be interested in, and the current charter
gives sole responsibility to approving releases to “the TDC” (although
that language should probably be updated to reflect whatever the
current opinions are on whether TDCs are one, many, per-project, …
3).

[2]: https://www.opencontainers.org/about/governance §5.b.vii

[philips]

@wking Good idea, I am fine with putting this into project-template once we finish the discussion.

[crosbymichael]

I think we have to have some type of quantitative rules or nothing has really changed. There is no way to measure when a release should be cut right now.

If we leave it up to milestones, it is still arbitrary about what is placed or removed from a milestone. Some of the project wide criteria can be things like, having a functional test suite to ensure the release is backwards compat with the previous release in it's major. Having compliance tests for implementations available for the version.

I think one of the most important things about working on a spec is stability and we need release criteria to help ensure that.

[stevvooe]

We are trying to build a specification here, not a webapp. These timelines are way too tight. Larger companies are going to take time to allocate resources and examine the specification. Time needs to be given to take appropriate community feedback.

[philips]

@stevvooe This sets up a 3 week minimum guideline assuming zero changes. Community feedback requiring changes adds additional time automatically. And the RC releases will come off of what are already regularly scheduled pre-releases leading up to the RC.

Can you be a bit more constructive? What do you think the minimum time from RC to release should be assuming the community has already seen many earlier releases?

[wking]

On Thu, Jun 09, 2016 at 05:26:41PM -0700, Brandon Philips wrote:

Can you be a bit more constructive? What do you think the minimum
time from RC to release should be assuming the community has already
seen many earlier releases?

It didn't make it into the minutes, but I vaguely remember @RobDolinMS
floating a week and a half to turn around review at Microsoft 1. I
think that means a “we're about to cut $MAJOR.$MINOR.$PATCH” post (as
currently suggested here), and then at least 30 days (to mirror 2?)
for feedback/voting. I'm fine with that 30 days being divided over 4
or 5 weekly release candidates, as long as the clock starts over if
anything more than a patch-level fix lands. Linux's 4.6 release went
through 7 candidates before a final 4.6, with 4.6-rc1 on 2016-03-26,
4.6-rc7 on 2016-05-08, and 4.6 on 2016-05-15, so a month-ish of
candidates doesn't seem unreasonable. With an approach like this, the
Microsoft team might only be chiming in on every other release
candidate, but they'd still have at least two rounds of review before
anything went out (which is enough time to raise alarms if anything
suspicious was in the release).

However, it is a long time to freeze development, so I suggest
starting a per-release branch as soon as the proposal goes out. Then
master (or other releases, e.g. a bugfix on an older line) can keep
chugging along while folks decide what to do about the proposed
release. The parallel branches would also let you freeze and propose
1.1.0-rc1 immediately after 1.0.0 lands (merging development that had
happend in another branch while 1.0.0 was under review for release).

 §8.d has a 30 day limit for Members to quit the OCI after a
 release they don't like was cut.

[cyphar]

I agree with @stevvooe, these timelines are a bit tight even for software projects. A specification bump is useless without running code, so we have to accommodate the fact that we need some cook time with an implementation. One week for an rc is alright for software, but it usually takes more than a week to review and update an implementation -- let alone test it.

This section is about specifications, but it reads like a tight timeline for OCI applications. Maybe make each rc a 2 week window with the final release being a 4 week window.

[vishh]

It will be useful to identify a few key stakeholders beyond just maintainers to validate a release candidate. Just having one or two implementations doesn't seem to satisfy the requirements of a standard. Would it be possible to come up with a list of partners who have a stake in this effort and get their feedback before making a final release?

If we do end up identifying some partners, then we will have to give them sufficient time to validate a release candidate. For large organizations, I would not be surprised if this process takes a month.

[wking]

On Fri, Jun 10, 2016 at 12:12:29PM -0700, Vish Kannan wrote:

Would it be possible to come up with a list of partners who have a
stake in this effort and get their feedback before making a final
release?

Isn't there already a list of pretty icons for that 1? I think the
goal is to make it clear where folks who have an interest should watch
for announcements, and to tell them how to chime in with feedback, but
I don't think we need to go around and knock on doors.

If we do end up identifying some partners, then we will have to give
them sufficient time to validate a release candidate. For large
organizations, I would not be surprised if this process takes a
month.

A month (4 weekly rcs + a final release) works for me 2, but in that
case you'd expect the slow responders to be showing up at the last
minute with any feedback. Still, the charter only gives Members 30
days to quit after a release 2, so I think it's safe to assume
everyone involved can turn things around in that time frame.

[philips]

@stevvooe @RobDolinMS Are y'all OK with a 1 month minimum time to make a major release? That would be a minimum of 1 week between releases (I am highly doubtful we will ever hit this best case) and an rc1, rc2, and rc3, and final. If someone shows up three weeks into the process, in rc3, with a show stopper everything would start over with rc4, rc5, rc6, and final. Making it 7 weeks.

Ideally we would learn of show stoppers early in the months leading up to the RC. And the point of making regular releases is to build consensus early in the process instead of creating an explosion of activity at the end.

[stevvooe]

@philips A 1 month minimum sounds okay. I would still like to hear from the larger companies on this.

[philips]

@crosbymichael I think a reason that people feel what is being placed in a milestone feels arbitrary is because they aren't following all of the individual discussions. Today most of those discussions happen on GitHub threads or on the weekly calls. That is why there is a part of this process that requires reporting out to the mailing list about the weekly progress.

Now, I totally agree there are some base line quantitative things should be true across our projects as well, here are some of my ideas:

• Tests that cover all optional and required features of the schemas
• Tests for backwards compatibility with all post-v1.0.0 versions
• A change log outlining all of the major changes and features from the previous release
• Documentation to help an implementor through any breaking change

I think it would be helpful for people to enumerate other quantitative baseline things to make a release and I can add them to this policy. What other ideas do people have?

[philips]

cc @opencontainers/image-spec-maintainers @opencontainers/runtime-spec-maintainers @opencontainers/runc-maintainers

[wking]

On Thu, Jun 09, 2016 at 05:21:29PM -0700, Brandon Philips wrote:

• Tests that cover all optional and required features of the schemas
• Tests for backwards compatibility with all post-v1.0.0 versions

These are not going to be possible if the spec does not bundle it's
test suites (and I'm in favor of not bundling them [1,2]).

• A change log outlining all of the major changes and features from
  the previous release
• Documentation to help an implementor through any breaking change

The first of these sounds critical to me 3, and the second would
certainly be nice (and I'm fine requiring it).

[diogomonica]

I just found out about this PR from my GitHub change feed, did I miss any emails/meetings?

Feels to me that a big change like this is exactly the kind of thing that should be discussed at the TOB level, right?

[wking]

On Thu, Jun 09, 2016 at 09:51:46PM -0700, Diogo Mónica wrote:

I just found out about this PR from my GitHub change feed, did I
miss any emails/meetings?

The triggering dev@ thread is 1, and the dev@ thread announcing this
PR is 2.

Feels to me that a big change like this is exactly the kind of thing
that should be discussed at the TOB level, right?

I didn't see anything in the charter about TOB oversight over releases
or the release process 3 outside of the general project
add/remove/reorg power from §6.a 4. I'd expect the TOB to be
involved in conflict resolution (again via §6.a) if a maintainer
wanted to promote a release dispute to the TOB, but I don't see a need
for the TOB to specify release procedures (outside of conflict
resolution). Can you explain why you feel the TOB has an interest
before a maintainer reports a conflict?

 Subject: Criteria for 1.0 image-spec Release
 Date: Thu, 2 Jun 2016 15:15:04 -0700 (PDT)
 Message-Id: <1f0bef4e-c786-4e64-b2b3-1634a557f4ca@opencontainers.org>

 Subject: Proposal: OCI Project Release Approval Process
 Date: Thu, 09 Jun 2016 20:45:20 +0000
 Message-ID: <CAD2oYtO-N2aEg+_F9BcSCZD4AG0veEFSp-6d1rqb2v0nC05WSA@mail.gmail.com>

[diogomonica]

@wking so what you're saying is that changes in the opencontainers/tob repository aren't of the concern of the TOB?

[cyphar]

This section is much shorter than the specification section. We should definitely codify rc releases -- this is quite important for several reasons. Currently we have runc 1.0.0-rc1 but I feel like there's quite a few things left to do and I'm not sure how many rcs other maintainers plan to have. I do like the fact that alerts on dev@opencontainers.org is required (I actually didn't know about the rc1 bump until I did git pull).

Personally, I'd like it if we specified something like this:

• Any maintainer (with 2 sponsors) must announce intentions to release on dev@opencontainers.org.
• The announcement must clearly say what the changelog is (so that other maintainers can argue whether or not each thing in the changelog is "ready"), and what the conditions for release are (no major changes, etc).
• They also need to specify how many rcs they want to have (minimum for a major release is 3, minor release is 1 and patchlevel is 0). Each rc must last at least a week.
• The maintainers also should make it clear what the commit hash they wish to tag (we really should move to stable and unstable branches IMO).

[wking]

On Thu, Jun 09, 2016 at 10:11:28PM -0700, Aleksa Sarai wrote:

This section is much shorter than the specification section. We
should definitely codify rc releases -- this is quite important for
several reasons.

For example…? The reason why it's shorter/simpler is that OCI Members
aren't locked into supporting runC, while they are locked into
supporting OCI specs 1. I'm not clear what sort of obligations a
runtime-spec release would impose on a Member, but I don't see how a
runC release would impose any obligations.

[cyphar]

I'm confused -- why is this in here at all then? runC is an OCI project, so we should be also making the rules clear for runC as well as the *-spec projects. If we don't want to make any rules for runC then we should remove the "Applications" from the title. No rules are better than half-baked rules.

[wking]

On Thu, Jun 09, 2016 at 10:30:22PM -0700, Aleksa Sarai wrote:

+## Conformance/Testing and Applications Releases

I'm confused -- why is this in here at all then? … No rules are
better than half-baked rules.

I'm all in favor of fully baking the rules. I'm just trying to
explain why they may not need to be as strict as the spec rules. I
agree that runC rcs would be nice if cutting a runC release meant
committing to bug-level backwards compat. But if the compatibility is
covered on the one side by runtime-spec and the other by a
command-line API spec (or similar), then the only remaining consumer
exposure is out-of-spec runC extensions [edit: and bugs or incomplete
specifications]. If those get the same OCI weight as in-spec features,
then yeah, runC extensions are basically another spec repository. If
those get less weight than in-spec features, then you can probably cut
a few corners.

[wking]

On Thu, Jun 09, 2016 at 10:08:16PM -0700, Diogo Mónica wrote:

@wking so what you're saying is that changes in the
opencontainers/tob repository aren't of the concern of the TOB?

They absolutely are. I'm just saying this PR was misdirected 1, and
@philips agrees it should be moved to project-tempate after discussion
settles 2. If the discussion is too spammy for TOB members, I'm fine
moving the PR before discussion settles (we can always link back).

[mrunalp]

s/We want do/We want to/

[philips]

done.

[tianon]

Minor typo? "... release candidates and SHOULD add restart the three-candidate count when a breaking ..."

[philips]

fixed

[wking]

I think the best way to address Jason's (paraphrased) “how do we
amend this?” is by separating project governance (with generic
voting rules and such) from release-specific rules. I've taken a stab
at that in philips#2, if folks want to look it over.

[wking]

I tried to flesh this out a bit in philips#2, but I'm concerned about two things:

• If you pull in the TOB, you have to explain how the TOB and maintainers will interact (i.e. what does “guide” mean in your sentence?). I'd rather just have {project}+security@ addresses (e.g. runtime-spec+security@opencontainers.org), and the maintainers can pull in the TOB as they see fit (like always).
• Who gets advanced notification of security fixes? Do folks who want advanced access apply by mailing the security address? E.g. “please let us know before pushing security fixes so we can patch our system. We have $N users, and you can trust us not to leak/abuse the information because $REASONS”. Where is the list of approved early-contacts maintained (probably not in the public project repo)?

[cyphar]

Something that isn't mentioned is signed tags of releases. This is something we really need to get a handle on, with each maintainer having a GPG key and they must sign releases. I personally have shifted to signing each individual commit, but that's optional.

[caniszczyk]

@philips what's left on this to finalize this?

[philips]

@caniszczyk As I think we are getting close to having a policy everyone agrees with. So, I just put up a new PR against the project-template repo and will post to the mailing list for a new vote.

[philips]

Vote has been sent to project maintainers: https://groups.google.com/a/opencontainers.org/forum/#!topic/dev/x-Oh3PDz1Y8 and the TOB has been cc'd. Closing this PR out.