Discussion: What is the optimal length of Code Freeze?

[jberkus]

Currently Code Freeze lasts around 4 weeks. In 2019, Code Freeze length was 1 to 1.5 weeks. The change from 1 to 4 weeks happened largely by accident and hasn't been reviewed by the community, so it's time to do so. Note that, should we decide to change the length, that's unlikely to take effect before the 1.24 cycle.

This issue is to discuss what the optimal length of Code Freeze would be, given all the interests and schedules that have to be balanced for a release. Among the considerations are:

• The set of deadlines that happen before and after Code Freeze starts. How many of them must happen during a Code Freeze period? How far apart are they?
• Docs deadlines were moved to after Code Freeze, allowing folks to focus completely on docs after turning in their code.
• Conformance soak time must be taken into consideration
• If we want to shorten Code Freeze, we have the option of either starting it later or ending it earlier. The latter would involve an earlier thaw, with more cherry-picks from main to the release branch.
• The relationship between Code Freeze and release quality
• How does Code Freeze hinder development while it's in effect?

See also the Release Retro.

/area release-team
/area release-eng

[justaugustus]

(Converted to a GH discussion)

[saschagrunert]

Thank you for the input @jberkus! I'm starting my thoughts with 1.23 (because it's the current cycle and already planned):

https://github.com/kubernetes/sig-release/tree/master/releases/release-1.23

• Conformance soak time must be taken into consideration

Can you elaborate on that?

• If we want to shorten Code Freeze, we have the option of either starting it later or ending it earlier. The latter would involve an earlier thaw, with more cherry-picks from main to the release branch.

We could move code freeze to test freeze, if we can assume that everything is merged before code freeze. I think one of the main reasons for the separation is to give tide some time to merge the huge amount of PRs.

• The relationship between Code Freeze and release quality

Coupling deadlines to real-world events is always good. So we moved code freeze to beta.0. This makes an alpha release less stable than a beta, which is a satisfying alignment from my point of view.

• How does Code Freeze hinder development while it's in effect?

Only PRs in the milestone can be merged. So in it's also fine to put a bunch of approved PRs into the milestone before code freeze and let them merge after the deadline. Adding the milestone after code freeze will be handled as an exception.

[jberkus]

All of those points are from yesterday's retro; I'd suggest reading the notes and watching the video if you want to capture the discussion around them. They're not my points, I was taking notes.

[saschagrunert]

Thanks for the heads up, looked at the notes and still think I’ll leave my comment as it is. Do you have a link to the recording available yet?

[jberkus]

Was just asking about that in contribex. But for conformance, you need to ask @justaugustus, not me.

[jberkus]

Only PRs in the milestone can be merged. So in it's also fine to put a bunch of approved PRs into the milestone before code freeze and let them merge after the deadline. Adding the milestone after code freeze will be handled as an exception.

FWIW, there were only a few dozen PRs pending code thaw this time, the lowest I've ever seen the post-code-freeze queue. Not sure if that's just to it being August, or whether not much dev really is happening during CF.

[jberkus]

Tagging @reylejano to supply commentary for 1.23

[reylejano]

Data points from releases:

Release	week of Code Freeze	week of Code Thaw	# of weeks of Code Freeze	# of weeks in release cycle
1.16	9	11	1 week, 5 days	12
1.17	8	10	1 week, 1 day	12
1.18	9	11	1 week, 5 days	12
1.19	13	delayed after release	10 weeks, 5 days	20
1.20	9	13	3 weeks, 5 days	13
1.21	9	13	4 weeks, 2 days	13
1.22	11	15	3 weeks, 6 days	15
1.23	13	16	3	16

• more on the delay of code thaw for 1.19
  • releng: Remove milestone restrictions from kubernetes/kubernetes test-infra#19277
  • https://groups.google.com/g/kubernetes-dev/c/YXGBa6pxLzo/m/SPb6yhNBBAAJ
  • https://groups.google.com/a/kubernetes.io/g/leads/c/BZZk0tG2cJU

[jberkus]

So, how well did the 3 weeks work this time?

[reylejano]

I believe we should shorten code freeze if the length hinders development. For 1.23, 3 weeks was appropriate. More on why...

For some background on coupling events and deadlines for 1.23, the start of code freeze was coupled with the first beta cut and code thaw was coupled with the release date. Test freeze was coupled with the first rc cut.
References:

• release-1.23: update branch cut deadlines #1688
• Schedule for the 1.23 release #1645 (comment)

It seems we have coupled code thaw with the release date from releases 1.20 to 1.23 -- I'm trying to find the background details on this and will update but I believe coupling thaw on release dates make possible fast-forwards easier even though currently we ask authors to create a cherry-pick PR.

I'm going to use data to determine if the length of code freeze hinders development (I'm open to opinions if there is better set of data to use), looking at stats for merged PRs on k/k master on the day of thaw and the day after:

• For 1.23, after Code Thaw there were 0 merged PRs after thaw and 53 the day after
• For 1.22, after Code Thaw there was 1 merged PR after thaw and 89 the day after
• For 1.21, after Code Thaw there were 2 merged PRs after thaw and 48 the day after
• For 1.20, after Code Thaw there was 1 merged PR after thaw and 48 the day after
• For 1.18, after Code Thaw there were 26 merged PRs after thaw and 92 the day after
• For 1.17, after Code Thaw there were 2 merged PRs after thaw and 21 the day after
• For 1.16, after Code Thaw there were 16 merged PRs after thaw and 52 the day after

For releases with 3+ weeks of code freeze, the average is 1 merged PR on the day of thaw and 59.5 PRs the day after
For releases with ~1.5 weeks of code freeze, the average is 14.67 merged PRs on the day of thaw and 53.7 PRs the day after

This could have been that development may have slowed during the start of the pandemic or we're at a certain maturity level with the project.

My conclusion is that a 3 week code freeze was appropriate for 1.23, I did not hear feedback that contributors were waiting for k/k to open. I favor to keep 3 weeks of code freeze if it is better for conformance soak, continue to couple deadlines with events, and is easier for branch management.

[reylejano]

I'm going back to previous retros on the discussions about shortening code freeze. I see the discussion started with release 1.10 and shortening code freeze was a success in 1.11. I'll add a comment if my thoughts have changed from reading the previous retros.

[jberkus]

For context, we're not seeing the kind of push from SIG leads to shorten code freeze the way we did back in 1.9. So it's possible that 3 weeks is fine with most folks; that's one of the things I was hoping we could determine with this thread.

[reylejano]

Related issue #1792
Simplify the release process by removing Code Thaw and open the master branch on Test Freeze.

[JamesLaverack]

My current draft of the schedule for 1.24 is the same in shape as 1.23 and has code freeze in week 12 and thaw in week 15 for a three week freeze.