Allow binaries written in Python via a "platform python" style approach

[cgwalters]

Like https://fedoraproject.org/wiki/Changes/Platform_Python_Stack

But we can do it in even stronger way: if we wanted to make it a whole lot harder for external people to use it, we could randomize the location of the binary, e.g. /usr/libexec/platform-python-$(uuidgen).

(Walking over every python binary and rewrite the shebangs)

[cgwalters]

I could pretty easily add an rpm-ostree compose option to do this.

[bgilbert]

/usr/libexec/platform-python-$(uuidgen) is a pretty cool idea if we end up needing to ship Python. It'd still be better not to have to maintain a Python at all, though.

[ajeddeloh]

I'd really really really like to not ship python. It's a huge chunk of the dependency tree we don't need to worry about. Less to ship is less to break. I'd also like to not ship an entire language to support one package.

I really want to keep FCOS as minimal as possibe.

[dustymabe]

small nit: with the /usr/libexec/platform-python-$(uuidgen) approach it would mean files in subsequent composes have different checksums, right? that being said I'm sure the same goal could be achieved

To re-iterate some previous conversations we had I think (I'm working off of memory here) there were multiple reasons we wanted to keep python out of the base:

1. prevent users from running scripts directly on the host
2. prevent shipping/maintaining python
3. prevent issues where user's python script needs library X that isn't installed
4. prevent security issues in python requiring a respin

This proposal helps with the 1st and 3rd of those since user's wouldn't be able to run their own scripts. The others would still be issues.

[dustymabe]

Added the meeting label so we can discuss during the community meeting today.

[dustymabe]

Discussed in the community meeting today (see the log starting at 16:59:17).

We added two more smallish reasons why it would be nice to remove python from the base. The list of reasons now comes out to:

1. prevent users from running scripts directly on the host
2. prevent shipping/maintaining python
3. prevent issues where user's python script needs library X that isn't installed
4. prevent security issues in python requiring a respin
5. less space used on disk + less data transmitted for updates
6. better perception we're a minimal OS

Each reason was then analyzed and we determined

• #1 is our primary concern
• #3 is also important so we don't break users when we make changes
• #2 is mostly handled for us because we are part of Fedora, but there is still a cost associated to finding/fixing bugs related to python and tools that use it, so it won't be non-zero if we do include a system python.
• #4 we have an action item to evaluate how often python (or python related libraries) in Fedora CoreOS need security updates (can use package set in Fedora Atomic Host as the reference).
• #5/#6 minimal is good, but we don't think we'd lose a significant number of users/community members if we shipped a system python, vs no python.

I'll dig into the investigation for #4 this week and we'll revisit in the community meeting next wednesday.

[glevand]

I feel the discussion should be If python support is not included, what do we need to do?, as apposed to If python support is included, what do we need to do?.

In the meeting this week the question came up about selinux and python. CL has a python USE flag, and support for that was added to the selinux packages to either not build certain parts of the packages, or not install certain parts of the packages. See the python USE flag commits of my PR that updates CL selinux. I guess we would need to do the same for FCOS.

[dustymabe]

I feel the discussion should be If python support is not included, what do we need to do?, as apposed to If python support is included, what do we need to do?.

I don't disagree. Unfortunately we're working a bit backwards here because this discussion was born out of the firewall discussion in #26. So it was more Is firewalld an option? and then we started this conversation to see if it was an option.

[dustymabe]

OK. I tried to take a stab at looking at our current python content in Fedora CoreOS
based on the image I create using the configs in the fedora-coreos-configs
repo. I concentrated on python3 since python2 should be completely gone soon.

I gathered the list of packages by running rpm -e python3

[root@coreos ~]# rpm -e python3
error: Failed dependencies:
        python(abi) = 3.7 is needed by (installed) python3-setuptools-39.2.0-7.fc29.noarch
        python(abi) = 3.7 is needed by (installed) python3-pip-18.0-3.fc29.noarch
        python(abi) = 3.7 is needed by (installed) python3-libselinux-2.8-3.fc29.x86_64
        python(abi) = 3.7 is needed by (installed) python3-setools-4.1.1-12.1.fc29.x86_64
        python(abi) = 3.7 is needed by (installed) python3-sssdconfig-2.0.0-3.fc29.noarch
        python(abi) = 3.7 is needed by (installed) python3-libsemanage-2.8-3.fc29.x86_64
        python(abi) = 3.7 is needed by (installed) python3-pyyaml-4.2-0.1.b4.fc29.x86_64
        python(abi) = 3.7 is needed by (installed) python3-ply-3.9-8.fc29.noarch
        python(abi) = 3.7 is needed by (installed) python3-bind-32:9.11.4-5.P1.fc29.noarch
        python(abi) = 3.7 is needed by (installed) python3-pytoml-0.1.18-1.fc29.noarch
        python(abi) = 3.7 is needed by (installed) atomic-registries-1.22.1-27.gitb507039.fc29.x86_64
        python(abi) = 3.7 is needed by (installed) python3-IPy-0.81-23.fc29.noarch
        python(abi) = 3.7 is needed by (installed) python3-audit-3.0-0.2.20180808git77fbcf3.fc29.x86_64
        python(abi) = 3.7 is needed by (installed) python3-policycoreutils-2.8-6.fc29.noarch
        python(abi) = 3.7 is needed by (installed) authconfig-7.0.1-7.fc29.x86_64
        python3 is needed by (installed) python3-bind-32:9.11.4-5.P1.fc29.noarch
        python3 is needed by (installed) authconfig-7.0.1-7.fc29.x86_64
        /usr/bin/python3 is needed by (installed) python3-setuptools-39.2.0-7.fc29.noarch
        /usr/bin/python3 is needed by (installed) python3-pip-18.0-3.fc29.noarch
        /usr/bin/python3 is needed by (installed) python3-libsemanage-2.8-3.fc29.x86_64
        /usr/bin/python3 is needed by (installed) bind-utils-32:9.11.4-5.P1.fc29.x86_64
        /usr/bin/python3 is needed by (installed) atomic-registries-1.22.1-27.gitb507039.fc29.x86_64
        /usr/bin/python3 is needed by (installed) python3-policycoreutils-2.8-6.fc29.noarch
        /usr/bin/python3 is needed by (installed) policycoreutils-python-utils-2.8-6.fc29.noarch
        /usr/bin/python3 is needed by (installed) authconfig-7.0.1-7.fc29.x86_64
        /usr/bin/python3 is needed by (installed) nfs-utils-1:2.3.2-1.rc3.fc29.x86_64
        /usr/bin/python3 is needed by (installed) xfsprogs-4.17.0-3.fc29.x86_64
        /usr/bin/python3 is needed by (installed) setools-console-4.1.1-12.1.fc29.x86_64

I then looked at the bodhi updates for each one including python itself. I
grouped together security updates that seemed to be related but for several
releases of fedora at the same time (i.e. a security update for foo-2.1 for
f24, f25, f26).

python itself

• ~12 issues over past 3 years
• https://bodhi.fedoraproject.org/updates/?packages=python
• https://bodhi.fedoraproject.org/updates/?packages=python2
• https://bodhi.fedoraproject.org/updates/?packages=python3

python3-setuptools

• no listed security issues in past 3 years
• https://bodhi.fedoraproject.org/updates/?packages=python-setuptools

python3-pip

• ~4 listed security issues in past 5 years
• https://bodhi.fedoraproject.org/updates/?packages=python-pip

python3-libselinux

• 1 listed security issue in past 3 years
• https://bodhi.fedoraproject.org/updates/?packages=libselinux

python3-setools & setools-console

• no listed security issues in past 2 years
• https://bodhi.fedoraproject.org/updates/?packages=setools

python3-sssdconfig

• 2 listed security issues in past 5 years
• https://bodhi.fedoraproject.org/updates/?packages=sssd

python3-libsemanage

• 1 listed security issue in past 3 years
• https://bodhi.fedoraproject.org/updates/?packages=libsemanage

python3-pyyaml

• 2 listed security issues in past 6 years
• https://bodhi.fedoraproject.org/updates/?packages=PyYAML

python3-ply

• no listed security issues in past 5 years
• https://bodhi.fedoraproject.org/updates/?packages=python-ply

python3-bind & bind-utils

• ~13 listed security issues in the past 5 years
• https://bodhi.fedoraproject.org/updates/?packages=bind
• note that these bulletins are for bind an all subpackages. In many cases I would
  bet the "clients" wouldn't be affected.

python3-pytoml

• no listed security issues in past 2 years
• https://bodhi.fedoraproject.org/updates/?packages=python-pytoml

atomic-registries

• no listed security issues in past 3 years
• https://bodhi.fedoraproject.org/updates/?packages=atomic

python3-IPy

• no listed security issues in past 8 years
• https://bodhi.fedoraproject.org/updates/?packages=python-IPy

python3-audit

• 1 listed security issue in past 3 years
• https://bodhi.fedoraproject.org/updates/?packages=audit

python3-policycoreutils & policycoreutils-python-utils

• 1 listed security issue in past 3 years
• https://bodhi.fedoraproject.org/updates/?packages=policycoreutils

authconfig

• 1 listed security issue in past 3 years
• https://bodhi.fedoraproject.org/updates/?packages=authconfig

nfs-utils

• no listed security issues in past 4 years
• https://bodhi.fedoraproject.org/updates/?packages=nfs-utils

xfsprogs

• 1 listed security issue in past 3 years
• https://bodhi.fedoraproject.org/updates/?packages=xfsprogs

note that we do not currently have firewalld installed but I'll look
at issues for firewalld too:

• 3 listed security issue in past 3 years
• https://bodhi.fedoraproject.org/updates/?packages=firewalld

Most packages have very few security updates but there are some that have
more than 10.

[embik]

Just to add my two cents as a present Container Linux and future FCOS user: We use (self-deployed) Python on Container Linux, mainly because Ansible is our automation driver for Kubernetes clusters. With kubeadm, we need to run a few tasks on the host itself like deploying updated config files and restarting services (I don't see a way around that at the moment). I would love to see Python to be included in FCOS .

[vtolstov]

@embik +1, i'm use saltstack minion on the hosts

[dustymabe]

hey @embik @vtolstov - I think the sentiment from the group is to try to get rid of python as much as possible. For what you are doing I think is #1 and #3 from the list of reasons above of why we don't want to include python. It's mostly that we don't want to break users by adding or taking away python libraries or shipping new versions of them so let's not ship python OR let's disallow users from using python on the host to prevent that.

You might be able to convince us otherwise, but I think those are the feelings right now. sorry 😢

[jlebon]

We also discussed ansible briefly, and most agreed that Ignition should be used for any type of light config mgmt. (Ideally, you don't have that much configuration to do, esp. not in cluster mode, and only a bit in the single server mode).

mainly because Ansible is our automation driver for Kubernetes clusters

Making FCOS work well with Kubernetes is one of its major goals, so I expect that we'll definitely address this use case. How we will do this though, I'm not sure if we've discussed it much yet.

[sedlund]

OR let's disallow users from using python on the host to prevent that.

Yes, this is one area that RancherOS made more clear with separate system and user spaces. Everything is in containers, any number of versions of python can exist. I'm honestly surprised to see RPM installed on a container distro, is that going to stay?

[dustymabe]

I'm honestly surprised to see RPM installed on a container distro, is that going to stay?

rpm is there for read only operations only, inspecting installed rpms, querying dependencies etc.. For example it's the one way I got the dependency information to know what "python" related rpms we have installed in FCOS today (see rpm -e python3 output from #32 (comment)).

Write operations aren't allowed. You can't rpm -ivh foo.rpm.

[embik]

Hey @dustymabe @jlebon, thanks for your answers. As far as I understand Ignition it's great for initial configuration, but I don't see it allowing us running update procedures. It's a bummer to hear about the current opinion and I'm not sure how to convince you other than describing our use case. This is how we update Kubernetes nodes:

1. Set up updated configuration files on the nodes (which kubeadm requires at the moment for working around quirks)
2. Update our systemd unit file for kubelet to pull a new kubelet image
3. Run kubeadm upgrade on the host
4. Restart the systemd unit to update/replace the running kubelet container

The first step might be not necessary in the future as kubelet live reconfiguration becomes a thing, but I don't see any way to restart my kubelet "system containers" in a structured update procedure without Ansible or a similar tool. This is my personal opinion but prevent users from running scripts directly on the host seems like an anti-pattern for me as a user. Provisioning systems via Ignition is great, but for systems running for a longer period of time I need some kind of lifecycle management and Ansible is our current tooling for that. I'm wondering how FCOS is going to handle general lifecycle management without it.

I guess what I'm saying is that if it won't be included with FCOS by default I'll have to continue our pypy deployment to /opt which is totally fine, I just wanted to chime in so you have some user input here.

[bgilbert]

@embik Thanks for the feedback! A goal of Fedora CoreOS is to carry forward the Container Linux philosophy that hosts should generally be immutable. If you needed to change the configuration, you'd update your Ignition config and reprovision the host.

We recognize that that approach isn't always viable, and in some cases configuration management is a better fit. Ideally CM would be possible on FCOS, but without carrying large amounts of code (or compatibility constraints) in the OS to support it. I've filed #53 to track this use case; please feel free to add your thoughts over there.

[dustymabe]

Ok discussed in this weeks community meeting. Lots of discussion, but we agreed on two points:

• AGREED: we think security issues for python (#4) aren't significant enough to cause us great hardships during releases
• AGREED: we'd really really like to remove python, but if we choose that we want to keep a tool or a few tools in fedora coreos that use python then we think the system python approach given that it solves #1 and #3, which are our primary concerns, would be reasonable.

i'll write up some text for the design doc and put up a PR

[dustymabe]

PR to the design doc: #56

[dustymabe]

PR merged. closing this as decided

[cgwalters]

I was thinking about this late one night (like a normal human) and one idea I had is (in addition to moving it to /usr/libexec/python or whatever) to patch our Python to deny execution of any scripts that don't live underneath /usr (and also on the same filesystem as the interpreter).

This wouldn't be some insurmountable barrier, but...I think most people would give up trying to use it at that point.

[bgilbert]

@cgwalters Oh wow. I like it.

[jlebon]

Random: noticed this today in the output of rpm-ostree status --advisories on my Silverblue: CVE-2019-5010 python: NULL pointer dereference using a specially crafted X509 certificate.