Don't install CNI plugins to /opt

[Silvanoc]

What would you like to be added:

Let RPM and DEB packages install CNI plugins under /usr/lib/cni, /usr/lib/cni-plugins or similar.

Why is this needed:

Current installation path for CNI plugins (/opt/cni) is not aligned with Debian and Fedora packaging best practices (even possibly policies).

In the case of Fedora, previous versions of their packaging guidelines were explicitly stating that packages shouldn't install any files under /opt. For a documented reason that I cannot fully understand, they reformulated it 4 years ago in a way that IMHO made a bit unclearer, but the essence remains the same: Fedora packages shouldn't install files under /opt.

In the case of Debian there are no clear/explicit policies forbidding it, but the standard linter for Debian packages (lintian) reports it as an error in the package.

[Silvanoc]

@kubernetes/sig-release-proposals

[k8s-ci-robot]

@Silvanoc: Reiterating the mentions to trigger a notification:
@kubernetes/sig-release-proposals

In response to this:

@kubernetes/sig-release-proposals

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[Silvanoc]

BTW, other packages being provided install under /usr/bin. Therefore at least for the sake of consistency, this issue should be considered. No matter how much interest exists on being compliant with any packaging policies.

[Silvanoc]

Probable relevant source code:

For Fedora packaging: https://github.com/kubernetes/kubernetes/blob/4f29960cb2058c4f556bf1e5ba49c9a29a501e70/build/rpms/kubernetes-cni.spec#L19

For Debian packaging: https://github.com/kubernetes/kubernetes/blob/4f29960cb2058c4f556bf1e5ba49c9a29a501e70/build/debs/BUILD#L143 and https://github.com/kubernetes/kubernetes/blob/4f29960cb2058c4f556bf1e5ba49c9a29a501e70/build/debian-hyperkube-base/Dockerfile#L63

[BenTheElder]

you should file this issue against https://github.com/kubernetes/release for better visibility, currently releases use the specs there. that repo also has much lower volume of interaction than this one 😅

[BenTheElder]

Moving these binaries would probably require updating a lot of the ecosystem to handle this though..

I will also add that sig-release generally agrees that more broadly something needs to be done about the state of debian / rpm publishing, but they're still figuring out what direction to take it going forward and could probably use more help with that.

[justaugustus]

/area release-eng
/priority important-longterm
/milestone v1.16

[ttousai]

Hello @justaugustus ! I'm a bug triage shadow for the 1.16 release cycle and considering this issue is tagged for 1.16, I'd like to check its status. Code freeze is starting on August 29th (about 3 days from now), do you think this issue can make 1.16?

[aojea]

I think that those guidelines only apply to the distribution packages, i.e. the ones that each distribution package and consider part of its distribution.
In this case, IIUIC, these are kubernetes packages. Most of the distros follow the FHS https://en.wikipedia.org/wiki/Filesystem_Hierarchy_Standard, and based on that seem that the /opt folder seems to be a good place based on the content.

/opt is reserved for the installation of add-on application software packages.

A package to be installed in /opt must locate its static files in a separate /opt/ or /opt/ directory tree, where is a name that describes the software package and is the provider's LANANA registered name.

https://refspecs.linuxfoundation.org/FHS_3.0/fhs-3.0.html#optAddonApplicationSoftwarePackages

[justaugustus]

(transferred this issue to k/release)
/area release-eng
/milestone v1.17
/kind question
/priority important-longterm
cc: @kubernetes/release-engineering

[k8s-ci-robot]

@justaugustus: The provided milestone is not valid for this repository. Milestones in this repository: [next, v1.16]

Use /milestone clear to clear the milestone.

In response to this:

(transferred this issue to k/release)
/area release-eng
/milestone v1.17
/kind question
/priority important-longterm
cc: @kubernetes/release-engineering

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[Silvanoc]

you should file this issue against https://github.com/kubernetes/release for better visibility, currently releases use the specs there. that repo also has much lower volume of interaction than this one sweat_smile

@BenTheElder thanks for the information. Thankfully @justaugustus already did so.

[Silvanoc]

I think that those guidelines only apply to the distribution packages, i.e. the ones that each distribution package and consider part of its distribution.
In this case, IIUIC, these are kubernetes packages. Most of the distros follow the FHS https://en.wikipedia.org/wiki/Filesystem_Hierarchy_Standard, and based on that seem that the /opt folder seems to be a good place based on the content.

/opt is reserved for the installation of add-on application software packages.

A package to be installed in /opt must locate its static files in a separate /opt/ or /opt/ directory tree, where is a name that describes the software package and is the provider's LANANA registered name.

https://refspecs.linuxfoundation.org/FHS_3.0/fhs-3.0.html#optAddonApplicationSoftwarePackages

The FHS documentation is a bit too unclear, therefore the distributions are implementing it differently. In the case of Debian, the package linter (lintian) would report it as an issue.
Fedora used to have a policy similar to that of Debian, but they changed it and now it's allowed.
As I've documented in the "Why is this needed" section of my initial ticket report.

@aojea you appear to be working for SuSE. Don't you have similar policies to those of Debian or Fedora?

Anyway, even if K8s doesn't care for the distribution policies, some consistency would be desirable and installing some things to /opt and some others to /usr/bin is not consistent.

[aojea]

@aojea you appear to be working for SuSE. Don't you have similar policies to those of Debian or Fedora?

yeah, but those policies are for the packages that are part of the distribution, this is an external package that works with that distribution, but is not released as part of it. Per example, if you have a bug with this package in a specific distro, you can't open a bug in that distro, right? They really don't know who is packaging this or who is the maintainer.

These are distribution kubernetes packages, they are built and maintained by the distribution:

https://build.opensuse.org/package/show/devel:kubic/kubernetes
https://apps.fedoraproject.org/packages/kubernetes

If you want to include a new package in a distribution you have to follow a process and, of course, obey the policies.

Ref:
https://www.debian.org/doc/manuals/developers-reference/ch05.en.html#newpackage
https://en.opensuse.org/openSUSE:How_to_contribute_to_Factory
https://fedoraproject.org/wiki/New_package_process_for_existing_contributors

Anyway, even if K8s doesn't care for the distribution policies, some consistency would be desirable and installing some things to /opt and some others to /usr/bin is not consistent.

Don't get me wrong, I'm not saying we should not care about consistency and best practices, just wanted to point out the difference between a distribution package and a third party package :)

[Silvanoc]

yeah, but those policies are for the packages that are part of the distribution, this is an external package that works with that distribution, but is not released as part of it. Per example, if you have a bug with this package in a specific distro, you can't open a bug in that distro, right? They really don't know who is packaging this or who is the maintainer.

Fully agree if thinking on OpenSuSE or Fedora. But in Debian there's not a big difference between distribution packages and properly debianized external packages (you would use lintian in both cases). In fact the packages typically first get created as external packages and then move into the distribution. Therefore properly debianized packages are expected to be in a shape that can be taken over by the distribution and therefore it raises the probability of directly getting them distribution supported.

Of course, SuSE and RedHat are investing resources on K8s, therefore I can understand that the /opt installation has more backing. Debian will then have to care itself for "distribution packaging", what will take much longer. In any case, if you decide for the /opt way, then I'd at least move everything there for consistency.

Don't get me wrong, I'm not saying we should not care about consistency and best practices, just wanted to point out the difference between a distribution package and a third party package :)

I didn't get you wrong, I just oversaw that Debian is probably the distribution handling it a bit differently here and you are right WRT OpenSuSE and Fedora.

BTW, if installing to /opt, according to the FSH and since there's no LANANA provider registered for K8s, then /opt/<package> should be used. Or K8s gets registered as provider in LANANA. The last option is simply ignoring the standard and using things like /opt/kubernetes, /opt/cni, ...

[BenTheElder]

1. We should stop shipping these CNI plugins in a package because it's not really useful, you still need a CNI config and realistically whatever tooling does that is supplying it's own binaries anyhow

2. However: this path (/opt/cni is required by all of the CNI integrations afaik, so Kubernetes is still ultimately going to be putting binaries in this path, short of changes that do not involve the packaging at all).

The packages write here because that's where kubelet / container runtimes look.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[justaugustus]

Closing as I agree with @BenTheElder's assessment.

See here to track the deprecation of kubernetes-cni deb/rpm packages: #885

/close

[k8s-ci-robot]

@justaugustus: Closing this issue.

In response to this:

Closing as I agree with @BenTheElder's assessment.

See here to track the deprecation of kubernetes-cni deb/rpm packages: #885

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.