Packaging: Add stricter systemd unit options

[erdnaxe]

What this PR does / why we need it:
This increases the isolation of Grafana service on systemd systems.

Special notes for your reviewer:
I only tested these changes on NixOS.

Release notice breaking change

Potential failure to start in Ubuntu 18.04 / Debian 9 / CentOS

In Grafana v8.2.0, this change can prevent the grafana-server service from starting on older versions of systemd, present on Ubuntu 18.04 and slightly older versions of Debian. If running one of those versions, please wait until v8.2.1 is released before upgrading. If you still want to upgrade or have already ugpraded, a simple fix is available here: #40162 (comment)

[CLAassistant]

All committers have signed the CLA.

[daniellee]

@erdnaxe thanks for submitting your changes upstream and helping us improve our packaging.

Could you explain the changes and the consequences of them? Are these recommendations from systemd-analyse security tool?

[erdnaxe]

Could you explain the changes and the consequences of them? Are these recommendations from systemd-analyse security tool?

Yes, these are recommendations from systemd-analyse security tool. Some notes:

• Only Debian systemd unit was setting UMask (9e21a08), I also apply this to rpm.
• I enable ProtectHome because I believe Grafana should never look in /home/.
• I set ProtectSystem to full which make /usr/, /etc/ and /boot/ read-only. Setting it to strict might introduce regressions in Grafana as I believe there is a need to write in /var/ and it is not trivial to list all authorized folders.
• I enable ProtectKernel* because I don't want Grafana to collect information about the kernel.
• NoNewPrivileges, LockPersonality and RestrictSUIDSGID (and maybe others) might introduce regressions if Grafana needs to call a setuid/setgid executable. For example if Grafana needs to call sendmail then it will fail with these options enabled.
• RestrictAddressFamilies is restricted to IPv4/IPv6 and Unix sockets as Grafana is a web server that might also talk to local sockets.
• PrivateUsers is going to create a new user namespace, which might introduce regressions. Maybe we can keep it disabled in a first time. For example it is impossible to make Grafana server listen on port 80 with this option as the CAP_NET_BIND_SERVICE capability will not work.

These parameters help to reduce the attack surface of an attacker that would have managed to get a RCE exploit in Grafana.

[kminehart]

Just a couple comments. I think setting RestrictAddressFamilies multiple times may overwrite it. The docs aren't clear, it seems to suggest that only happens when you set it to "". Regardless I think it'd be clearer if it was all one line since there's no deny-list stuff happening.

Other than that, I think it looks good. All of the options make sense.

NoNewPrivileges, LockPersonality and RestrictSUIDSGID (and maybe others) might introduce regressions if Grafana needs to call a setuid/setgid executable. For example if Grafana needs to call sendmail then it will fail with these options enabled

Grafana uses https://github.com/go-mail/mail/ to send emails, which according to their documentation:

Gomail can only send emails using an SMTP server

So that shouldn't be a problem.

The one area I can see a regression happening is in plugins. It executes plugin processes using os/exec.This is done by github.com/hashicorp/go-plugin. From what I can tell, setuid/setgid is not used, and the plugin processes run as the grafana user.

[kminehart]

Looks like there's some CI trouble. I'll see if I can figure out why that's happening. Is your branch / fork up to date with main?

[erdnaxe]

Looks like there's some CI trouble. I'll see if I can figure out why that's happening. Is your branch / fork up to date with main?

I have just updated my branch with main.

[xlson]

@marefr do you still want me to review this PR, it looks like @kminehart have taken ownership of it?

[kminehart]

@xlson up to you. I'm going to test it locally real quick and it if it works I'll approve it. Feel free to add any feedback if there's anything that sticks out to you. Or if you'd like, I can hold off on my approval until you test it locally too?

Edit: tested locally and it works as expected. I tested the main concern I had which was running a backend datasource.

[marefr]

@xlson I added you because I was added as reviewer and don't have any knowledge of this 😄 As Kevin says, up to you.

[xlson]

@kminehart I'm good, feel free to merge :) Thanks for reviewing it!

[stephenjamieson]

Just FYI, this does break ubuntu 18.04 installs as its sytemd is too old.

[marefr]

@stephenjamieson can you please open a new bug report/issue describing the details? Thanks

[stephenjamieson]

Unfortunately, instead of debugging further I upgraded to Ubuntu 20.04, so I do not have an environment to reproduce again.

[kminehart]

I tested the 8.2.0-beta1 package on ubuntu 18.04 and it seems to be working fine. It complains about some unrecognized options but I haven't had any trouble getting Grafana to start.

[marefr]

Related to #30755 ?

[henriknoerr]

after install of Grafana 8.2 I can no loger start Grafana, getting the error:
t=2021-10-07T13:43:01+0200 lvl=eror msg="Stopped background service *api.HTTPServer" logger=server reason="failed to open listener on address 0.0.0.0:443: listen tcp 0.0.0.0:443: bind: permission denied" t=2021-10-07T13:43:01+0200 lvl=eror msg="Server shutdown" logger=server error="*api.HTTPServer run error: failed to open listener on address 0.0.0.0:443: listen tcp 0.0.0.0:443: bind: permission denied"

I have added binding to low ports with command below which fixes this when upgrading, but not this time.
sudo setcap 'cap_net_bind_service=+ep' /usr/sbin/grafana-server

If I uncomment what was added with this commit I can start Grafana.

/usr/lib/systemd/system/grafana-server.service
#CapabilityBoundingSet= DeviceAllow= #LockPersonality=true #MemoryDenyWriteExecute=true #NoNewPrivileges=true #PrivateDevices=true #PrivateTmp=true #PrivateUsers=true #ProcSubset=pid ProtectClock=true #ProtectControlGroups=true #ProtectHome=true #ProtectHostname=true #ProtectKernelLogs=true #ProtectKernelModules=true #ProtectKernelTunables=true #ProtectProc=invisible #ProtectSystem=full #RemoveIPC=true #RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX #RestrictNamespaces=true #RestrictRealtime=true #RestrictSUIDSGID=true #SystemCallArchitectures=native #SystemCallFilter=@system-service #SystemCallFilter=~@privileged #SystemCallFilter=~@resources #UMask=0027

Red Hat Enterprise Linux release 8.4 (Ootpa)

[erdnaxe]

after install of Grafana 8.2 I can no loger start Grafana, getting the error: t=2021-10-07T13:43:01+0200 lvl=eror msg="Stopped background service *api.HTTPServer" logger=server reason="failed to open listener on address 0.0.0.0:443: listen tcp 0.0.0.0:443: bind: permission denied" t=2021-10-07T13:43:01+0200 lvl=eror msg="Server shutdown" logger=server error="*api.HTTPServer run error: failed to open listener on address 0.0.0.0:443: listen tcp 0.0.0.0:443: bind: permission denied"

I have added binding to low ports with command below which fixes this when upgrading, but not this time. sudo setcap 'cap_net_bind_service=+ep' /usr/sbin/grafana-server

I believe that you need to change the following in your systemd unit (you can make an override):

# Give the CAP_NET_BIND_SERVICE capability
CapabilityBoundingSet=CAP_NET_BIND_SERVICE
AmbientCapabilities=CAP_NET_BIND_SERVICE

# A private user cannot have process capabilities on the host's user
# namespace and thus CAP_NET_BIND_SERVICE has no effect.
PrivateUsers=false

Doing so should give the required capability to use low ports. Can you confirm that this solves your issue?

I am not really clear if this procedure should be documented in Grafana documentation, or if we should rollback PrivateUsers and CapabilityBoundingSet options to allow people to set the capability without overriding systemd unit.

[henriknoerr]

I have overwritten the three sugested settings and it works.

[kminehart]

I think these settings that were added to this PR were intended to prevent less-than-secure setups like this.

@henriknoerr is serving Grafana on port 443 on all interfaces, which is not the default; I wager in most organizations it'd be inadvisable to use a lower port.

I think it makes sense to add this scenario to the documentation. It also may make sense to preserve backwards compatibility to add the CAP_NET_BIND_SERVICE capability to the default systemd settings, though it will allow the less secure scenario of hosting your Grafana instance on a port < 1024.

What do you think, @erdnaxe ?

[erdnaxe]

I think the final choice depends on Grafana team philosophy:

• If they want the update to be as transparent as possible for users that are already setting CAP_NET_BIND_SERVICE, then it is better to revert PrivateUsers and CapabilityBoundingSet options,
• Else the systemd unit override to add CAP_NET_BIND_SERVICE could be documented with a huge warning on top explaining why it might be unsecure.

[kminehart]

I think we will backport some additional documentation in the "What's new" page as well as additional instructions for systemd users installing via rpm and deb.

I did some reading and it seems like it is in our best interest to ensure that users explicitly grant the CAP_NET_BIND_SERVICE setting.

[fauust]

Because of this, grafana fails to start on Ubuntu 18.04.5 LTS (but I guess it is related to systemd version so other distribution may be impacted).

See: #40194

[henriknoerr]

I now see that service failure is not limited to binding on port 443. If image-render plugin is installed, the service will go in a restart loop:

t=2021-10-08T13:46:53+0200 lvl=eror msg="Stopped background service *rendering.RenderingService" logger=server reason="Failed to start renderer plugin: Unrecognized remote plugin message: \n\nThis usually means that the plugin is either invalid or simply\nneeds to be recompiled to support the latest protocol."

t=2021-10-08T13:46:53+0200 lvl=warn msg="plugin failed to exit gracefully" logger=plugins.backend pluginId=grafana-image-renderer

t=2021-10-08T13:46:53+0200 lvl=eror msg="Server shutdown" logger=server error="*rendering.RenderingService run error: Failed to start renderer plugin: Unrecognized remote plugin message: \n\nThis usually means that the plugin is either invalid or simply\nneeds to be recompiled to support the latest protocol."

Again it works with image-render plugin with the new lines in grafana-server.service commented out.

/Henrik

[kminehart]

@fauust yes unfortunately Ubuntu does not have that syscall filter group.

@henriknoerr The line that is causing the image renderer plugin to fail is the MemoryDenyWriteExecute=true line.

For a quick workaround, view this comment: #40187 (comment)

[nsteinmetz]

@kminehart but in the comment, it's false whereas it's true in this PR ?!

MemoryDenyWriteExecute=false

So I don"t get how it will fix #40187

[kminehart]

Yeah, @nsteinmetz setting that to true causes the imagerenderer to fail. false is the default behavior, effectively undoing the change that causes it to fail.

This is the PR that introduced the breaking change, not the PR that will fix the issue.

[nsteinmetz]

ah ok :) thanks for the precision

[erdnaxe]

If some plugins break with MemoryDenyWriteExecute=true, then maybe this option should be reverted. From experience, this option usually breaks Python CFFI and Java.

[kminehart]

Yeah a PR was merged yesterday to revert that option. It looks like it only affected a small amount of plugins, specifically the Image Renderer plugin.

[ulidtko]

This has broken listening on port 80 too.

A systemd override like below fixed it for me.

$ cat /etc/systemd/system/grafana-server.service.d/grant-cap-listen80.conf 
[Service]
CapabilityBoundingSet=CAP_NET_BIND_SERVICE
AmbientCapabilities=CAP_NET_BIND_SERVICE