Skip to content


Subversion checkout URL

You can clone with
Download ZIP
Fetching contributors…
Cannot retrieve contributors at this time
337 lines (245 sloc) 11.8 KB
The Stable Linux Kernel Tree, Delivering a Stable Platform on a
Constantly Moving Base
The Linux kernel development process has had a "stable" kernel branch
for much of its lifespan. But, for the past 5 years, that branch has
looked much different from what people consider "stable".
This talk will go into how the different branches of the Linux kernel
are developed, the reasoning and rules that govern the stable kernel
branches, and why companies can, and should, rely and contribute to them
in order to build and support their products.
Hi, I'm Greg, and I'm a Linux kernel developer.
I'm here to talk about the stable kernel tree, and how it fits into the
rapidly changing Linux kernel development cycle. I'm going to cover how
the kernel used to be developed, how it currently works, and how you can
help out.
First off lets look at some numbers for what the current kernel
development process is creating:
The kernel as of the 2.6.35 release, currently contains:
33,315 files
13,470,000 lines
This is a big project.
Working on this, over the past year, was:
2,478 developers
342 companies
Which shows how large the community of developers is.
Now the fun numbers:
11,600 lines added
5,000 lines removed
2,300 lines modified
That's not bad, until you realize the time unit involved here:
per day for all of 2009
that works out to:
5.51 changes per hour
That's 24 hours a day, 7 days a week.
That's a lot of change. Linus is the fastest moving software project
ever known in this history of computing.
And this rapidly changing body of code is used by all of the different
companies and users that Jim spoke about the first day here. That's a
huge change for how most companies viewed software and the traditional
model of the "don't change it" type of engineering that used to go on.
But then again, Linux has succeeded in ways that no other operating
system ever has, so it makes sense that it has done so in a manner
different than anything before.
Before getting into how the kernel development process works today, it's
good to look back and see how it used to work, which will show how we
got to what we now do.
Let's start with the 2.0 release, way back in June of 1996. After that
version came out, in June of 1996, Linus released 21 releases before
starting a "development" branch, called 2.1.
2.1 was where the new development happened. Over the next 2 1/3 years,
releases happened every couple months, until finally, 2.2.0 was released
after 141 releases.
Then it all started over again. 2.2 was worked on for a bit, 28
releases and 4 months later, 2.3.0 was started and it continued on for a
bit shorter, just over 1 1/2 years and 58 releases.
And again, it started over.
The kernel branched and 2.5.0 was started 10 months and 15 kernels
2.5 was hard. Real hard. We had all these ideas of things we wanted to
do, we adopted new tools and were figuring out how to use them, we did
all sorts of crazy development as a large number of us were now being
paid to do this development, so we had more time to do it.
And, because we all had more time to do this work, it took even longer.
2.6.0 didn't come out until almost 3 years later, taking 86 releases.
Because of the long development cycle, companies really wanted to use
parts of the 2.5 kernel in their "stable" releases. So they backported
things from the 2.5 tree into the 2.4 trees, and started calling them
"enterprise" kernel releases. These releases were _hard_ to do by the
distros and the developers in charge of them. Usually the same
developer had to make the changes in the 2.5 kernel and then turn around
and try to figure out how to backport the changes to the 2.4 kernel for
a distro to use. It was a nightmare for the developers, something that
none of them who lived through it ever wanted to do again.
With the start of the 2.6 releases, we all settled into a cycle of bug
fixes and waiting to see when Linus would open up the new development
tree. But, as we were waiting, a funny thing happened. We realized
that we had learned our tools and proceedures quite well, forcing a new
development model on ourselves without even realizing it.
Here's a list of some of the things we had done:
- one patch does one thing
- no patch breaks the build
- complete description of each patch.
Because of this, we were now able to slowly start to change and evolve
the kernel without breaking things. Linus started releacing the -rc
kernels in a predictable manner, and after one kernel summit, we
codified all of this saying we would stick with what we had been doing
over the past year, there would not be any more development branch at
all, only releases with small release cycles.
We set up some rules:
- new changes in the -rc1 kernel
- bugfixes afterwards
- a few -rc releases and then a new kernel comes out.
Thie happened every 2-3 months like clockwork, and has for the past 7
But, one of the objections to this development model was the fact that
for that 2-3 months, there was no "stable" tree. If a security problem
happened, or a bugfix that was really needed, users had to wait for the
three months, or brave using a development kernel.
So, some people proposed a "backport" kernel that might track the last
version, but Linus quickly said that there was no way this was going to
work out, and anyone who tried to do so was insane:
From: Linus Torvalds <torvalds <at>>
Subject: Re: RFD: Kernel release numbering
Newsgroups: gmane.linux.kernel
Date: 2005-03-03 16:23:39 GMT (5 years, 29 weeks, 3 days, 21 hours and 1 minute ago)
I'll tell you what the problem is: I don't think you'll find anybody to do
the parallell "only trivial patches" tree. They'll go crazy in a couple of
weeks. Why? Because it's a _damn_ hard problem. Where do you draw the
line? What's an acceptable patch? And if you get it wrong, people will
complain _very_ loudly, since by now you've "promised" them a kernel that
is better than the mainline. In other words: there's almost zero glory,
there are no interesting problems, and there will absolutely be people who
claim that you're a dick-head and worse, probably on a weekly basis.
From: Greg KH <>
Subject: Re: RFD: Kernel release numbering
Date: 2005-03-03 16:43:53 GMT
> Anybody?
Well, I'm one person who has said that this would be a very tough
problem to solve. And hey, I like tough problems, so I'll volunteer to
start this. If I burn out, I'll take the responsibility of finding
someone else to take it over.
From: Chris Wright <>
Subject: Re: RFD: Kernel release numbering
Date: 2005-03-03 16:55:33 GMT
> Anybody?
Andres Salomon (-as patches) and I have been talking about that at least
regarding security fixes. It's worth trying in a more complete and
formalized way. Guess I can be branded a sucker :)
So that started the "stable" tree.
It would last only as long as the development cycle of the previous
[Show slides that describe how the kernel now is developed]
The rules were set pretty strict:
- It must be obviously correct and tested.
- It cannot be bigger than 100 lines, with context.
- It must fix only one thing.
- It must fix a real bug that bothers people (not a, "This could be a
problem..." type thing).
- It must fix a problem that causes a build error (but not for things
marked CONFIG_BROKEN), an oops, a hang, data corruption, a real
security issue, or some "oh, that's not good" issue. In short,
something critical.
- New device IDs and quirks are also accepted.
- No "theoretical race condition" issues, unless an explanation of how
the race can be exploited is also provided.
- It cannot contain any "trivial" fixes in it (spelling changes,
whitespace cleanups, etc).
- It must follow the Documentation/SubmittingPatches rules.
- It or an equivalent fix must already exist in Linus' tree (upstream).
Since this has been set up, we've done over 360 kernel releases for the
past 5 1/2 years for the stable team.
The stable kernel update history (since the stable kernel process was
introduced after the 2.6.11 release) looks like this:
2.6.11 12 79
2.6.12 6 53
2.6.13 5 44
2.6.14 7 96
2.6.15 7 110
2.6.16 62 1053
2.6.17 14 191
2.6.18 8 240
2.6.19 7 189
2.6.20 21 447
2.6.21 7 162
2.6.22 19 379
2.6.23 16 302
2.6.24 7 246
2.6.25 20 492
2.6.26 8 321
2.6.27 54 1584
2.6.28 10 613
2.6.29 6 383
2.6.30 10 419
2.6.31 14 826
2.6.32 23 2003
2.6.33 7 883
2.6.34 7 601
2.6.35 6 442
If you note, a few kernels seem to contain a lot more releases than
others. After a few years of doing this work, it turned out that the
distros were starting to rely on the stable releases to get updates from
the kernel development team.
The 2.6.16 kernel was the first experiment in keeping the tree open for
longer than just a few release cycles. It lived for a bit over 2 years.
The 2.6.27 kernel was the second one of these, and it's still alive,
getting a release last week.
About last year, a number of the developers who worked for the distros
got together and figured out that if they all picked one kernel version,
it would make their lives much easier. This resulted in the 2.6.32
kernel release being picked up by Debian, Red Hat, and SuSE for their
"long-term enterprise" kernel releases. Canonical and Oracle have since
also picked up on this and have based releases on this version as well.
This makes the management and support of the kernel for these releases
much easier. The Debian, RedHat, SuSE and Oracle developers have been
working to put the fixes that they have found are needed for the .32
releases into the upstream -stable release as well. This lowers the
number of patches that they have to maintain, and offers the fixes to
all other users as well, making everyone's lives easier.
As you can see, the .32 kernel also has the most number of individual
patches for a stable release as well. This is due to the large number
of patches that the distros have been digging up and passing on to me.
I'd really like to thank the Debian and Red Hat and Novell developers
for doing this work, as it has been a great help and it shows how strong
the community is, even when working for different companies that
compete, they work together to create something better overall that they
couldn't have done individually.
So, how do you get a patch into the stable tree.
Once again, look at the stable_kernel_rules.txt file:
- add "Cc: <>" to the Signed-off-by: area of your
- send the git commit id of the patch to and state
which stable kernel tree you want it applied to.
It's that simple. If you are a subsystem maintainer, you should start
thinking about this when you add patches to your trees for merging. If
you notice that the patch fixes a bug, or adds a new device id, mark it
with a Cc: tag. Some subsystem maintainers are very good about this,
and others, very bad. Networking has a wonderful maintainer, David
Miller, who actually creates a separate patch queue that he tests out
before sending the patches on to me, which is a great help. KVM also
creates a git tree to help out with this, and they do a lot of testing
as well. Other subsystems don't do anything at all, and I'm reduced to
digging through git commit logs to see if I can find stuff when people
report problems {cough SCSI}
Review cycle:
- patches go out for review, copying the authors of the patch and the
people who signed-off on the patch
- 48-72 hours to NAK the patch
- new stable release happens
Get Involved
- tag your patches
Jump to Line
Something went wrong with that request. Please try again.