Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fluentd does not work with hostmount-anyuid with SELinux Enforcing #89

Open
richm opened this issue Mar 10, 2016 · 15 comments

Comments

Projects
None yet
7 participants
@richm
Copy link
Contributor

commented Mar 10, 2016

With SELinux Enforcing, with the new hostmount-anyuid scc for system:serviceaccount:logging:aggregated-logging-fluentd, fluentd is not able to read/write /var/log on the host.

@richm

This comment has been minimized.

Copy link
Contributor Author

commented Mar 11, 2016

Tried doing this:

Under seLinuxOptions: you can specify a default, eg:

 seLinuxOptions:
   level: "s0:c123,c456"
   user: xxxx  #selinux username
   role: yyyy  #selinux role, not openshift roles
   type: zzz

I'm not sure what goes in xxxx and yyyy here.
but doing this:

# oc edit scc hostmount-anyuid

...
runAsUser:
  type: RunAsAny
seLinuxContext:
  seLinuxOptions:
    level: s0:c6,c1
    type: spc_t
  type: MustRunAs
supplementalGroups:
  type: RunAsAny
...

doesn't work - fluentd doesn't start - AVCs like this:

type=AVC msg=audit(1457655809.603:929): avc:  denied  { entrypoint } for  pid=8879 comm="exe" path="/pod" dev="dm-1" ino=4718595 scontext=system_u:system_r:spc_t:s0:c1,c6 tcontext=system_u:object_r:svirt_sandbox_file_t:s0:c1,c6 tclass=file
        Was caused by:
                Missing type enforcement (TE) allow rule.

            You can use audit2allow to generate a loadable module to allow this access. 
@richm

This comment has been minimized.

Copy link
Contributor Author

commented Mar 11, 2016

@rhatdan suggests this:

Save this file to /tmp

make -f /usr/share/selinux/devel/Makefile container_logger.pp
semodule -i container_logger.pp

docker run -ti -v /var/log:/var/log --security-opt label:type:container_logger_t fedora /bin/sh

How do I start the fluentd container with this argument?
Do I need to edit the scc hostmount-anyuid?

And you should be allowed to play with the logs in locked down mode.

If this works, I will write a blog on doing it.

Depending on what the app needs the policy could be a lot simpler, but I added full capabilities and full
networking.

container_logger.te

policy_module(container_logger, 1.5.0)

virt_sandbox_domain_template(container_logger)

logging_manage_all_logs(container_logger_t)

allow container_logger_t self:capability { chown dac_override fowner kill setgid setuid setpcap net_bind_service net_raw sys_chroot mknod audit_write setfcap };
dontaudit container_logger_t self:capability fsetid;
dontaudit container_logger_t self:capability2 block_suspend ;

kernel_read_network_state(container_logger_t)

allow container_logger_t self:capability { net_raw net_admin net_bind_service };

allow container_logger_t self:udp_socket create_socket_perms;
allow container_logger_t self:tcp_socket create_stream_socket_perms;
allow container_logger_t self:netlink_route_socket create_netlink_socket_perms;
allow container_logger_t self:packet_socket create_socket_perms;
allow container_logger_t self:socket create_socket_perms;
allow container_logger_t self:rawip_socket create_socket_perms;
allow container_logger_t self:netlink_kobject_uevent_socket create_socket_perms;

corenet_tcp_bind_generic_node(container_logger_t)
corenet_udp_bind_generic_node(container_logger_t)
corenet_raw_bind_generic_node(container_logger_t)
corenet_tcp_sendrecv_all_ports(container_logger_t)
corenet_udp_sendrecv_all_ports(container_logger_t)
corenet_udp_bind_all_ports(container_logger_t)
corenet_tcp_bind_all_ports(container_logger_t)
corenet_tcp_connect_all_ports(container_logger_t)

@ewolinetz

This comment has been minimized.

Copy link
Contributor

commented Mar 29, 2016

to keep it tracked in the same place, this is the blog post written for this issue: https://www.projectatomic.io/blog/2016/03/selinux-and-docker-part-2/

@elyscape

This comment has been minimized.

Copy link
Contributor

commented Mar 30, 2016

Unfortunately, the blog post is incomplete. To replicate the comment I just posted there:

Unfortunately, this isn't quite enough. It should be, as there's a docker_log_t type that inherits from var_log_t, and running logging_manage_all_logs(container_logger_t) gives us access to it. Unfortunately, Docker does not correctly set the log files to be of type docker_log_t, and they default to docker_var_lib_t. As a result, container_logger_t is blocked from accessing /var/lib/docker/containers/$ID/$ID-json.log. To work around this until Docker fixes the issue upstream, you can add the following line at the end of container_logger.te:

docker_read_lib_files(container_logger_t)

This makes things work in the interim, though it also provides wider access than would be ideal.

I'll be creating an issue upstream with Docker sometime tomorrow.

@richm

This comment has been minimized.

Copy link
Contributor Author

commented Mar 30, 2016

@elyscape

This comment has been minimized.

Copy link
Contributor

commented Mar 31, 2016

Ultimately, we're probably going to need to use a different security context constraint specifically for the fluentd service account, as hostmount-anyuid requires its containers' SELinux context to be provided by Origin/Kubernetes, whereas we'll need to provide a custom one. Something like this should work:

allowHostDirVolumePlugin: true
allowHostIPC: false
allowHostNetwork: false
allowHostPID: false
allowHostPorts: false
allowPrivilegedContainer: false
allowedCapabilities: null
apiVersion: v1
defaultAddCapabilities: null
fsGroup:
  type: RunAsAny
kind: SecurityContextConstraints
metadata:
  annotations:
    kubernetes.io/description: 'container-logger provides all the features of the
      restricted SCC but allows host mounts and any UID by a pod, which runs as the
      container_logger_t SELinux type.  This is primarily used by the log aggregation
      system. WARNING: this SCC allows host file system access as any UID, including
      UID 0.  Grant with caution.'
  creationTimestamp: null
  name: container-logger
priority: null
requiredDropCapabilities: null
runAsUser:
  type: RunAsAny
seLinuxContext:
  seLinuxOptions:
    type: container_logger_t
  type: MustRunAs
supplementalGroups:
  type: RunAsAny
users:
- system:serviceaccount:logging:aggregated-logging-fluentd

Then we just need to make sure that the container_logger module is included with the Origin node install.

@elyscape

This comment has been minimized.

Copy link
Contributor

commented Mar 31, 2016

This is, of course, dependent on either giving container_logger_t access to docker_var_lib_t files or having the issue with docker_log_t resolved in the relevant packages. I've made pull requests that fix them, and the ones relevant for Origin (on CentOS 7 and RHEL7, at least) are:

That being said, the former at least should not be merged until it is decided if its upstream counterpart (moby/moby#21673) will be merged.

@sosiouxme

This comment has been minimized.

Copy link
Member

commented Apr 4, 2016

We're already providing custom roles for logging components, so adding a custom SCC is actually not much of a burden if that proves useful.

It seems like the main issue now is how to distribute the custom SELinux policy module. For RPM-based installs we can just add a policy RPM but we also have to consider containerized installs especially on Atomic where we can't add RPMs. Would it be fair to expect Ansible to install this module on all the nodes when configuring logging? At least until the upstream PRs mentioned here are shipped.

Or, as an alternative, does everything still work out if we just keep fluentd in the privileged SCC?

@richm

This comment has been minimized.

Copy link
Contributor Author

commented Apr 4, 2016

@sosiouxme "Or, as an alternative, does everything still work out if we just keep fluentd in the privileged SCC?"

Yes, it works, it's just that "privileged" gives the fluentd container (and service account?) a lot more access than is needed.

@sosiouxme

This comment has been minimized.

Copy link
Member

commented Apr 5, 2016

Yeah, it does. But that seems like a better experience for users until we can get the SELinux policy changes in a RHEL release.

@rhatdan

This comment has been minimized.

Copy link

commented Apr 5, 2016

Sure, but as I stated above, your changes will have no effect on newly created containers. On on containers logs which have had restorecon run on them.

sosiouxme added a commit to sosiouxme/origin-aggregated-logging that referenced this issue Apr 5, 2016

README,tests: use privileged SCC for fluentd
until there is a more proper fix for
openshift#89
@elyscape

This comment has been minimized.

Copy link
Contributor

commented Apr 5, 2016

That's where a new version of docker-selinux with containers/container-selinux#8 would come in, plus a fix to Docker so that new log files have the correct SELinux type to begin with. That work is being tracked in moby/moby#21673.

@lvthillo

This comment has been minimized.

Copy link

commented Apr 6, 2016

The workaround of using privileged SCC for fluentd isn't solving the problem for me.

2016-04-06 08:45:37 +0000 [error]: unexpected error error_class=Errno::EACCES error=#<Errno::EACCES: Permission denied - /var/log/es-containers.log.pos>
  2016-04-06 08:45:37 +0000 [error]: /opt/app-root/src/gems/fluentd-0.12.22/lib/fluent/plugin/in_tail.rb:97:in `initialize'
  2016-04-06 08:45:37 +0000 [error]: /opt/app-root/src/gems/fluentd-0.12.22/lib/fluent/plugin/in_tail.rb:97:in `open'
2016-04-06 08:45:37 +0000 [error]: /opt/app-root/src/gems/fluentd-0.12.22/lib/fluent/plugin/in_tail.rb:97:in `start'

workaround (didn't solve)
$ oc edit scc hostmount-anyuid

runAsUser:
  type: RunAsAny
seLinuxContext:
  type: MustRunAs
supplementalGroups:
  type: RunAsAny
users:
- system:serviceaccount:openshift-infra:pv-recycler-controller

oc edit scc privileged

runAsUser:
  type: RunAsAny
seLinuxContext:
  type: RunAsAny
supplementalGroups:
  type: RunAsAny
users:
- system:serviceaccount:openshift-infra:build-controller
- system:serviceaccount:management-infra:management-admin
- system:serviceaccount:default:router
- system:serviceaccount:default:registry
- system:serviceaccount:jenkins:default
- system:serviceaccount:logging:aggregated-logging-fluentd

Deleted the fluentd pods but stillt he same issue.
I recreated the logging project several times. The documentation is up to date about using privileged SCC for fluentd

@sosiouxme

This comment has been minimized.

Copy link
Member

commented Apr 18, 2016

@lorenzvth7 please try it again. we recently merged a fix for this and pushed out a new deployer image. in addition to the SA being a member of the privileged SCC, the pod spec had to indicate that it's a privileged pod.

@lvthillo

This comment has been minimized.

Copy link

commented Apr 18, 2016

@sosiouxme Thanks, it works!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.