Drop support for RHEL6/CentOS6

[icecrime]

Description

Current state of things:

• Those distros are supported as per http://docs.docker.com/installation/rhel/and http://docs.docker.com/installation/centos/
• RedHat themselves only support Docker on RHEL7 (as stated by https://access.redhat.com/solutions/1378023, and confirmed by several people)
• 1.7.0 is broken on these (most notably because of libnetwork bridge creation code: docker#14024)

We shipped 1.7.0 without mentioning we would break compatibility, so this will be fixed in 1.7.1, but we will drop support in 1.8.0.

How to

• Documentation update
• Consider making --userland-proxy=false a default?
• Drop older kernel compatibility code (cf. bridge creation fallback, there's probably more)

References

• #14174
• RedHat support ticket

Ping @moxiegirl @thaJeztah to anticipate documentation change and spread the words on related issues / PRs!

[visualphoenix]

👎 would prefer if we could deprecate 6.x in the future but many institutions haven't migrated to systemd-capable systems yet.

[visualphoenix]

1.8.0 is so soon. could we schedule dropping support for 6.x in, say, 6 months?

[unclejack]

@visualphoenix No, it's been on life support ever since RHEL7 and CentOS7 were released.

[crosbymichael]

@visualphoenix you could still run docker 1.6 on those distros and 1.7.1 when it is released.

[visualphoenix]

can docker support the 1.7.x series for 6.x for a while going forward? it seems the 1.7.x series has a bunch of regressions

[visualphoenix]

@crosbymichael the networking features and volume support for 1.8 was pretty huge for 6.x to get. also there have been so many new things baking for v2 registry which haven't stabilized.

[LK4D4]

@visualphoenix Yes, 1.7 will be supported on RHEL6 until they introduce bug to kernel, which we won't be able to fix on our side.

[visualphoenix]

@LK4D4 i mean, will there be future 1.7.x releases improving stability. Docker hasnt been great in the past about backporting fixes to earlier release series.

[icecrime]

@visualphoenix We stick to the project rules of supporting our latest version, and backport security fixes (where applicable) to version N-1. Right now 1.7.0 is our focus: we are fixing regressions in 1.7.1, and we'll fix more in 1.7.2 if necessary.

[visualphoenix]

👎 this sucks.

[icecrime]

@visualphoenix I don't think it does: as I said, we will make 1.7.x solid, even on RHEL6. However, we simply don't have the capacity to do this later on once 1.8.0 will be out.

[unclejack]

Red Hat has been doing a lot of work on the 2.6.32 kernel in order to make it work with Docker. As far as I'm concerned, this was mostly work done to make Docker work on RHEL 6 / CentOS 6 until RHEL 7 / CentOS 7 would get released and the customers could use RHEL 7 / CentOS 7 to run containers.

Unfortunately, it seems that the 2.6.32 kernel requires too much work to backport code and keep it fully functional when it comes to containers. The 2.6.32 kernel had only some of the features containers use nowadays on modern kernels. Expecting this very old kernel to be stable when running containers isn't reasonable, even if a lot of kernel developers work on it and if they've put a lot of time into making it better. Since this is a very old kernel, backporting some of the latest kernel features and fixes takes considerable effort, and is an ever increasing effort with diminishing returns (more effort needed as the kernel's code changes, people going straight to RHEL 7 / CentOS 7 for new deployments).

There are a lot of companies and people out there still running RHEL 6 / CentOS 6, but this is not a good environment for running containers.

Moving forward, the Docker project can't support RHEL 6 / CentOS 6 because those two no longer support it. That means that those two distributions shouldn't be used for new deployments where containers need to be used. The reason behind this is that they'd have to put too much effort into making the kernel work properly with containers. Since they're not going to patch such kernel bugs, we wouldn't be able to fix 99.9% of the reported issues which only occur on those systems, thus not being able to properly support those systems.

This isn't a matter of not wanting to help or being mean to someone by dropping support for RHEL 6 / CentOS 6. It's simply about allowing Red Hat and the CentOS project to focus on making RHEL 7 / CentOS 7 better for running containers. The Docker project will also benefit from not having to support the only family of distributions with such an old kernel and not having to hold back on making it more stable everywhere else. The user land proxy had to be kept around because of RHEL 6 / CentOS 6.

Long story short: supporting containers on RHEL 6 / CentOS 6 was increasingly challenging for Red Hat and they've decided to stop supporting it; since they're no longer going to make changes to the kernel and to the associated user land components, we can't support it any further because we can't fix the kernel bugs. We can only try to make sure Docker still runs on it and that's about it. Anyone who runs into a kernel panic on those systems while running containers will have a hard time getting help.

As a Docker maintainer who's put a few hundred hours into supporting people who run into problems with Docker on RHEL 6/CentOS 6 and who has tested Docker on these distros, I can only recommend the following:

1. upgrade to RHEL 7 / CentOS 7
2. run the software which requires RHEL 6 / CentOS 6 in RHEL 6 / CentOS 6 containers
3. keep your systems fully updated with yum update
4. do not use custom kernels (including kernels from elrepo, mainline kernels, custom rpms, etc)

[logicrime]

Visualphoenix hit the nail on the head, this DOES suck.

[visualphoenix]

@icecrime i'm confused when you say:

we will make 1.7.x solid, even on RHEL6.

By solid, that would mean fixing a bunch of 1.7.0 bugs which you guys currently have set to fix for 1.8.0, like: #14337 #13681 #12080 #11394 #11065 #9570 to name a few.

[crosbymichael]

@visualphoenix Docker is opensource so you are free to help out the project and yourself by submitting patches for various bugs that you care about. The maintainers are always available to answer any questions if you get stuck.

I think everyone would be very happy to do another point release with a batch of bug fixes.

[fcntl]

Congrats for breaking your software on the most widely used OS distro on planet Earth.

[icecrime]

@fcntl Again, the most widely used OS distro on the planet dropped support for our software. We cannot fix this alone.

[visualphoenix]

@icecrime actually the support document says:

Red Hat does not recommend running any version of Docker on any RHEL 6 releases. Docker feature developments are tied to RHEL7 infrastructure components for kernel, devicemapper (thin provisioning, direct lvm), sVirt and systemd.

Docker features are only tied to whatever components Docker wants to support. Right now, docker works on the upcoming 6.7 kernel. I could see making the minimum supported kernel version be the 6.7 kernel, but does not recommend is not the same as does not support.

[lostsnow]

Bad news 💔
Most of my servers are running CentOS 6.x

[bhuvan]

@icecrime @unclejack @crosbymichael To be clear, it's more to do with kernel version than the OS version. By saying Drop support for RHEL6/CentOS6 you claim you wont support kernel =<2.6.x. right?

if users use redhat certified 3.x kernel with RHEL6/CentOS6, they can still use docker 1.8+. correct me if i'm wrong.

[FomalhautWeisszwerg]

@visualphoenix Just fork and do yourself. Docker is opensource. Why don't you do?

[jamshid]

+1 Thanks for not trying to support RHEL6/CentOS6 through another release, and instead spending that time on general, forward-moving fixes and features.
Heartbroken users, the pain of upgrading your OS is less than the pain of trying to run docker in an environment that really doesn't want to run it.

[leafmulch]

+1 for @bhuvan's question.

[thaJeztah]

@bhuvan @leafmulch see the earlier comment; docker#14365 (comment)

As a Docker maintainer who's put a few hundred hours into supporting people who run into problems with Docker on RHEL 6/CentOS 6 and who has tested Docker on these distros, I can only recommend the following:

1. upgrade to RHEL 7 / CentOS 7
2. run the software which requires RHEL 6 / CentOS 6 in RHEL 6 / CentOS 6 containers
3. keep your systems fully updated with yum update
4. do not use custom kernels (including kernels from elrepo, mainline kernels, custom rpms, etc)

We don't support using custom kernels, only the kernel that ships with a distribution. It could work, but it's asking for trouble and you're fully at your own risk there. The steps above are really the better approach if you want something stable and care about your data.

[leafmulch]

@thaJeztah understood, but here's my current situation: https://docs.docker.com/installation/oracle/

Docker requires the use of the Unbreakable Enterprise Kernel Release 3 (3.8.13) or higher on Oracle
Linux. This kernel supports the Docker btrfs storage engine on both Oracle Linux 6 and 7.

Thanks.

[thaJeztah]

@leafmulch thanks for pointing that out, not sure what the situation will be there. That docs may need to be updated as well? @unclejack

[loveai88]

If I use 3.10.80-1.el6.elrepo.x86_64 kernel in centos 6 ,can I continue use docker 1.8?
Thanks

[unclejack]

@leafmulch @loveai88 You can try that, but you will probably run into bugs if the devicemapper libraries are in any way incompatible with your kernel or if there's any kind of incompatibility between that newer kernel and the user land components.

The user land libraries are more adequate for kernel 2.6.32 and it wouldn't be a surprise to see that there are problems caused by switching to a newer kernel.

Please keep in mind that kernel 3.8 is only supported by Oracle Linux today and that might mean you will also run into unique bugs.

Running at least kernel 3.10 on RHEL 6 / CentOS 6 might fix the problems you were seeing, but that's not something we (the people of the Docker project) can support. The reason behind that is that the vendors of those distributions don't support those custom kernels on the distributions and we can't do that either - we can't provide support for something we've got no way to fix bugs on and we can't ask the distribution vendors to fix problems in their distribution which have been encountered in unsupported setups so that Docker can be used.

Please consider upgrading to RHEL 7 / CentOS 7, instead of installing custom kernels.

[mgomersbach]

@visualphoenix This does suck, but how is not having systemd related to docker?

[visualphoenix]

Hey mark, for me the relation to systemd is that my company hasn't yet
fully ported its deployment and host level configuration and metric
gathering management to systemd. Likewise, the ongoing effort to port
thousands of systems to containers isn't complete yet. From comments I've
seen on other tickets, there are others in this situation also. I surmise
those companies who require it will likely wind up purchasing official
support from docker if they require support on these platforms. What with
1.8 marking the transition of docker to spinning its core out for reference
use for the Open Container Project initiative for its driver, this would
probably mean that OCP won't support 6.6+.

Also related to systemd/OCP/docker: systemd nspawn is one avenue for
running containers. Rkt stage1 only supports systemd. So it's possible that
future docker versions will require aspects of systemd.

Just my 2c.

On Friday, July 3, 2015, Mark Gomersbach notifications@github.com wrote:

@visualphoenix https://github.com/visualphoenix This does suck, but how
is not having systemd related to docker?

—
Reply to this email directly or view it on GitHub
docker#14365 (comment).

Raymond "VisualPhoenix" Barbiero

[pdericson]

We shipped 1.7.0 without mentioning we would break compatibility, so this will be fixed in 1.7.1, but we will drop support in 1.8.0.

Was the break in compatibility deliberate?

[thaJeztah]

Was the break in compatibility deliberate?

No, it wasn't, just unfortunate it was discovered too late

[unclejack]

@pdericson No, it was most certainly not deliberate. It was a regression.

@visualphoenix Please read the reply below carefully.

I understand that dropping support for RHEL 6 / CentOS 6 can be frustrating, annoying and even difficult for some people and companies. I understand that resources need to be put into this migration and that this makes some people angry.

You should also keep in mind the fact that you didn't have Docker the day when you or your company deployed RHEL 6 / CentOS 6. Expecting your LTS operating system to have perfect compatibility with software which requires an up to date kernel isn't reasonable. The same applies to software which didn't even exist when that version of the OS was originally released.

Docker Inc has given away Docker for free so that everyone may use it. There's no "commercial" or "enterprise" version of Docker, only other products and services.
The amount of time the paid developers from Docker Inc have spent helping members of the community and users in all kinds of places is somewhere in the range of 10,000-30,000 hours. These users don't owe any money to Docker Inc, nor were they required to sign some kind of contract to benefit from all of this.

Whenever someone opens an issue on GitHub, they aren't asked "Do you have a commercial support contract with Docker Inc?", they're only asked to provide detailed information so that the community and the maintainers may help with the issue they've reported.

Please don't make comments regarding "paid support contracts" and speculate that Docker Inc will ask you to pay a support contract when it comes to supporting a version of a distribution which has officially dropped support for Docker. This is offensive because all that Docker Inc has given away for free wasn't free for Docker Inc, they're actually paying employees, and because this is an open source project before anything else. It's also offensive to those who contribute to the project on their own time. Everyone spends time fixing bugs, writing docs, implementing new features, trying to reproduce bugs and helping others to make the project better for everyone. Contributors sometimes don't get an extra hour of sleep, don't do something else they'd enjoy or stay up late at night to improve Docker.

Docker will not require systemd. Docker got some new features to make it work with systemd, but systemd won't become a requirement for running Docker. boot2docker, RancherOS and others don't use systemd. Since boot2docker is also an official project and doesn't use systemd, I hope it's crystal clear that systemd won't become a dependency for Docker.

Kernel 2.6.32 had a lot of problems with containers, even on Debian Squeeze with lxc. Docker on RHEL 6 / CentOS 6 wasn't future proof.

[Djelibeybi]

@leafmulch we will continue to support Docker, including 1.8+, on Oracle Linux 6. We have a newer kernel and newer device-mapper libs on that platform than our upstream vendor. Hence why we require UEK3 for Docker support.

[thaJeztah]

Thanks for adding that info, @Djelibeybi

[icecrime]

If you are concerned with the quality of 1.7.x on RHEL6/CentOS6, please help us test 1.7.1-rc2 (RPMs coming): docker#14264 (comment)

[lsm5]

Anyone reading this, I'm NOT speaking for paid RHEL6 users/support, but if you're on CentOS6 and can try out the xen4centos kernel, it might take you past the initial problems and docker will hopefully work fine after devmapper etc. are updated as well, still WIP though. @jasonbrooks is working on this atm, he'll post more details soon.

I'm quite certain we can make newer kernels and related packages available for CentOS6

[barkerd427]

This is a huge blow to those enterprises who have not even been able to complete the testing of rhel 7. At least we'll be able to upgrade to 1.7 when rhel approves of it, but it's disappointing this was announced so late relative to how fast rhel 6 users probably move.

[thaJeztah]

@barkerd427 I feel your pain, but this was effectively out of our control, once RHEL upstream ended support. Recent RHEL 6 kernels resulted in panics and strange behavior when using Docker (even on older versions of Docker) . The docker maintainers decided to drop support, because we cannot provide a stable solution without the backing of the distribution upstream, and decided it was better to not support a platform, than to "support" a system that is really instable.

[barkerd427]

@thaJeztah I know all too well the pains of rhel6 and Docker! I want to move as quickly as I can to rhel7, but we just aren't ready as it had so many changes that have to be tested. I certainly understand the requirement to move away from it. I wish Redhat would have handled this better with their customers. You guys have done an awesome job with this project and supporting outdated kernels. Thank you.

[pdericson]

RHEL / CentOS 6.7 was released on 2015-07-22. The base kernel is now kernel-2.6.32-573.el6.x86_64. This kernel works well with Docker.

Will Docker Inc consider un-dropping support from RHEL / CentOS 6.7 with the 573 kernel?

[rtoma]

@thaJeztah would you please answer @pdericson 's question?

We are considering upgrading from RHEL 6.6 to 6.7. Which is so much easier than upgrading to RHEL 7.

If Docker Inc would/could support the 6.7 kernel, that would make a lot of RHEL 6 users very happy.

[thaJeztah]

@rtoma @pdericson

I don't think it's wise to revert this decision; without the backing of RHEL upstream, we cannot guarantee a stable solution (which, I think, would defeat the purpose of running a long term support OS)

In addition, having to keep the 2.6 kernel into account when developing new Docker features would hold-back the project as a whole, affecting all other users.

You are still able to install and run docker 1.7.1 on RHEL/CenOS 6;
http://yum.dockerproject.org/repo/main/centos/6/Packages/docker-engine-1.7.1-1.el6.x86_64.rpm

But updating to docker 1.8 and above, will require an updated system, or building docker from source.

[Djelibeybi]

Or switch to Oracle Linux 6 and the UEK3, which supports Docker 1.8 and higher.

[pdericson]

I am happily running Docker 1.7.1 on RHEL / CentOS 6.7 :) Note that at night and on weekends I absolutely do not run RHEL / CentOS 6.7!

My point is, Docker Inc. needs to make the choice between meeting their customers where they are right now and pushing for the future without worrying about supporting a major segment of their install base.

I love the fact that so many cool new features are being added to Docker 1.8 and beyond but I also love the fact that Docker 1.7.1 makes getting my job done on RHEL / CentOS 6.7 so, so much easier.

Please stop using Redhat as an excuse. If Docker Inc. breaks compatibility with RHEL / CentOS 6.7 it's on Docker Inc. only and engineers stuck with RHEL / CentOS 6.7 will feel that pain and be forced to look for alternatives.

[gavinwhyte]

@pdericson Docker doesn't care about doing market research on OS in corporates that could earn potential revenue.

[gavinwhyte]

@pdericson Maybe if they hire a data scientist, do some proper analysis, maybe they might consider supporting Docker on RHEL 6, quiet possible, just don't care about revenue.

[pdericson]

Good points @gavinwhyte but also keep in mind that automated testing is hard - it's much easier just to support the kernels developers happen to have running on their dev machines

[gavinwhyte]

@pdericson We all know supporting developers is what makes this product works, not all OS are equal some are more equal than others. But if people are smart enough to figure out market share, then maybe there is an opportunity considering dockers popularity that distros within reason are supported. Mind you RedHad has fixed issues in ver 6.7. At the least Docker should reciprocate. Quiet funny when they use Redhat 6 or Centos 6 as an excuse.

[pdericson]

Alright, I'll stop being a troll and act serious.

I understand why @icecrime raised this issue. But the situation has changed. The 2.6.32-573 kernel that comes with RHEL / CentOS 6.7 is stable. We all just went through a bad time where RedHat released two broken kernels but they have fixed the situation now and it's back to smooth sailing.

Please Docker Inc, @icecrime and others - we think you guys rock, we love docker (why else would we introduce it to enterprise?) - can you please seriously consider making RHEL / CentOS 6.7 with 2.6.32-573 the new minimum support OS + kernel.

[LK4D4]

@pdericson We personally broke nothing intentionally in 1.8, we kept all code(userland proxy, bridge creation through ioctl etc.) for 2.6 kernel. I think you can use it in RHEL6. But we can't just say: all is okay, guys, we recommend you to use docker on RHEL6, because all is not okay - redhat will never again fix 2.6 kernel for docker, me or you.

[gavinwhyte]

@LK4D4 I totally agree with you broke nothing, Docker has introduced a product that is amazing. Docker depends on Operating Systems. The Operating System concerned is Centos 6. Right now, Redhat has provided fixes, meaning Docker runs on the 2.6.32-573 kernel. Meaning Docker should provide support for this kernel, Redhat has made the attempt to fix. Making a comment like "Redhat will never again fix 2.6 kernel" in rather sad. The mere fact that it works on the kernel currently Docker should support it. When it comes to a point when it does work, we should address it then. Don't assume problems will happen in the future and based on that hypothesis Docker cannot support it now. It works now, Docker 1.8 should happen now. The future has more in store.

[ocontant]

It is really unfortunate that docker is not supported on RHEL 6. It comes often that the realm of developer is often incompatible with the realm of operations. It's fun as a developer to be on the latest cutting edge version of thing as it ease development by a lot. But operation for many consideration and reason must stay away from the cutting edge version.

It is not true that majority of people will move to RHEL 7 quickly. The majority will complete to move from RHEL 5 to RHEL 6 by 2016 to be just on time before 2017 End of Extended Support and will stick wtih RHEL 6 until 2020 before moving to RHEL 7.

I don't know any bank or fortune 500 company rolling out RHEL 7 already in day to day operations. Even less migrating existing system to RHEL 7. Most company going to RHEL 7 for now are startup, PME and company that doesn't have to comply with ISO, PCI and other management consideration and constraint. They are also the type of company deploying Ubuntu for production, giving even less market for RHEL 7.

In our context, unfortunately for us, it is a complete turn off for Docker until 2018-2020. Seems like container technology is not yet mature enough for our market. (In all due respect for this technology and the people working on it, just meaning it's still too early for us to be able to adopt it. Continue your good work and in a few years we will be able to catch up). My disappointment only comes from the hope I had that we could be able to use it in our context. Can't wait until we can start deploying RHEL 7 in enough volume for our environment, to put in place a new standard using container.

Wish you the best! I'll still play with it in my lab in the mean time :)

[pdericson]

Wow has it been 4 months? I'm still running docker 1.7.1 on rhel / centos 6.7 and it's working ok.

[ocontant]

Do you have any stability issue so far like network disconnection?

Sent from my Samsung device

-------- Original message --------
From: Peter Ericson notifications@github.com
Date: 2015-12-12 17:27 (GMT-05:00)
To: docker/docker docker@noreply.github.com
Cc: Olivier Contant contant.olivier@gmail.com
Subject: Re: [docker] Drop support for RHEL6/CentOS6 (#14365)

Wow has it been 4 months? I'm still running docker 1.7.1 on rhel / centos 6.7 and it's working ok.

—
Reply to this email directly or view it on GitHub.

[pdericson]

Network has been ok, did come across docker#14738 recently but it's rare - I'm considering maintaining an internal fork of 1.7.1 to apply the fix.

[ourway]

It's simply impossible for us moving to RHEL7 and I loved docker. Sad!