[Bug]: Cannot start Netdata when binding to a privileged port (80)

[MAH69IK]

Bug description

I am using Netdata on port 80. To allow it to do this - I set the cap_net_bind_service+ep capability. This worked for a long time. But now, after upgrading to 1.45.0 I am getting an error:

Mar 28 14:00:17 netdata systemd[761388]: netdata.service: Failed to execute /usr/sbin/netdata: Operation not allowed
Mar 28 14:00:17 netdata systemd[761388]: netdata.service: Failed at step EXEC spawning /usr/sbin/netdata: Operation not allowed

This happens because since version 1.45.0 the allowed capabilities have been added to the /lib/systemd/system/netdata.service file and among them there is no NET_BIND_SERVICE (by the way this is strange, shouldn't the existing NET_RAW and NET_ADMIN allow to use port 80? But allowing them did not produce the desired result).

Is it possible to add CAP_NET_BIND_SERVICE to the list?

Expected behavior

Bind to privileged port.

Steps to reproduce

1. Install netdata from official .deb
2. Change default port to 80
3. setcap cap_net_bind_service+ep /usr/sbin/netdata
4. systemctl start netdata.servce

Installation method

manual setup of official DEB/RPM packages

System info

Linux netdata 5.10.0-21-amd64 #1 SMP Debian 5.10.162-1 (2023-01-21) x86_64 GNU/Linux
/etc/os-release:PRETTY_NAME="Debian GNU/Linux 11 (bullseye)"
/etc/os-release:NAME="Debian GNU/Linux"
/etc/os-release:VERSION_ID="11"
/etc/os-release:VERSION="11 (bullseye)"
/etc/os-release:VERSION_CODENAME=bullseye
/etc/os-release:ID=debian

Netdata build info

Packaging:
    Netdata Version ____________________________________________ : v1.45.0
    Installation Type __________________________________________ : binpkg-deb
    Package Architecture _______________________________________ : x86_64
    Package Distro _____________________________________________ :  
    Configure Options __________________________________________ : dummy-configure-command
Default Directories:
    User Configurations ________________________________________ : /etc/netdata
    Stock Configurations _______________________________________ : /usr/lib/netdata/conf.d
    Ephemeral Databases (metrics data, metadata) _______________ : /var/cache/netdata
    Permanent Databases ________________________________________ : /var/lib/netdata
    Plugins ____________________________________________________ : /usr/libexec/netdata/plugins.d
    Static Web Files ___________________________________________ : /var/lib/netdata/www
    Log Files __________________________________________________ : /var/log/netdata
    Lock Files _________________________________________________ : /var/lib/netdata/lock
    Home _______________________________________________________ : /var/lib/netdata
Operating System:
    Kernel _____________________________________________________ : Linux
    Kernel Version _____________________________________________ : 5.10.0-21-amd64
    Operating System ___________________________________________ : Debian GNU/Linux
    Operating System ID ________________________________________ : debian
    Operating System ID Like ___________________________________ : unknown
    Operating System Version ___________________________________ : 11 (bullseye)
    Operating System Version ID ________________________________ : none
    Detection __________________________________________________ : /etc/os-release
Hardware:
    CPU Cores __________________________________________________ : 2
    CPU Frequency ______________________________________________ : 1999000000
    RAM Bytes __________________________________________________ : 8343982080
    Disk Capacity ______________________________________________ : 64424509440
    CPU Architecture ___________________________________________ : x86_64
    Virtualization Technology __________________________________ : kvm
    Virtualization Detection ___________________________________ : systemd-detect-virt
Container:
    Container __________________________________________________ : none
    Container Detection ________________________________________ : systemd-detect-virt
    Container Orchestrator _____________________________________ : none
    Container Operating System _________________________________ : none
    Container Operating System ID ______________________________ : none
    Container Operating System ID Like _________________________ : none
    Container Operating System Version _________________________ : none
    Container Operating System Version ID ______________________ : none
    Container Operating System Detection _______________________ : none
Features:
    Built For __________________________________________________ : Linux
    Netdata Cloud ______________________________________________ : YES
    Health (trigger alerts and send notifications) _____________ : YES
    Streaming (stream metrics to parent Netdata servers) _______ : YES
    Back-filling (of higher database tiers) ____________________ : YES
    Replication (fill the gaps of parent Netdata servers) ______ : YES
    Streaming and Replication Compression ______________________ : YES (zstd lz4 gzip)
    Contexts (index all active and archived metrics) ___________ : YES
    Tiering (multiple dbs with different metrics resolution) ___ : YES (5)
    Machine Learning ___________________________________________ : YES
Database Engines:
    dbengine ___________________________________________________ : YES
    alloc ______________________________________________________ : YES
    ram ________________________________________________________ : YES
    none _______________________________________________________ : YES
Connectivity Capabilities:
    ACLK (Agent-Cloud Link: MQTT over WebSockets over TLS) _____ : YES
    static (Netdata internal web server) _______________________ : YES
    h2o (web server) ___________________________________________ : YES
    WebRTC (experimental) ______________________________________ : NO
    Native HTTPS (TLS Support) _________________________________ : YES
    TLS Host Verification ______________________________________ : YES
Libraries:
    LZ4 (extremely fast lossless compression algorithm) ________ : YES
    ZSTD (fast, lossless compression algorithm) ________________ : YES
    zlib (lossless data-compression library) ___________________ : YES
    Brotli (generic-purpose lossless compression algorithm) ____ : NO
    protobuf (platform-neutral data serialization protocol) ____ : YES (system)
    OpenSSL (cryptography) _____________________________________ : YES
    libdatachannel (stand-alone WebRTC data channels) __________ : NO
    JSON-C (lightweight JSON manipulation) _____________________ : YES
    libcap (Linux capabilities system operations) ______________ : NO
    libcrypto (cryptographic functions) ________________________ : YES
    libyaml (library for parsing and emitting YAML) ____________ : YES
Plugins:
    apps (monitor processes) ___________________________________ : YES
    cgroups (monitor containers and VMs) _______________________ : YES
    cgroup-network (associate interfaces to CGROUPS) ___________ : YES
    proc (monitor Linux systems) _______________________________ : YES
    tc (monitor Linux network QoS) _____________________________ : YES
    diskspace (monitor Linux mount points) _____________________ : YES
    freebsd (monitor FreeBSD systems) __________________________ : NO
    macos (monitor MacOS systems) ______________________________ : NO
    statsd (collect custom application metrics) ________________ : YES
    timex (check system clock synchronization) _________________ : YES
    idlejitter (check system latency and jitter) _______________ : YES
    bash (support shell data collection jobs - charts.d) _______ : YES
    debugfs (kernel debugging metrics) _________________________ : YES
    cups (monitor printers and print jobs) _____________________ : YES
    ebpf (monitor system calls) ________________________________ : YES
    freeipmi (monitor enterprise server H/W) ___________________ : YES
    nfacct (gather netfilter accounting) _______________________ : YES
    perf (collect kernel performance events) ___________________ : YES
    slabinfo (monitor kernel object caching) ___________________ : YES
    Xen ________________________________________________________ : YES
    Xen VBD Error Tracking _____________________________________ : NO
    Logs Management ____________________________________________ : YES
Exporters:
    AWS Kinesis ________________________________________________ : NO
    GCP PubSub _________________________________________________ : NO
    MongoDB ____________________________________________________ : YES
    Prometheus (OpenMetrics) Exporter __________________________ : YES
    Prometheus Remote Write ____________________________________ : YES
    Graphite ___________________________________________________ : YES
    Graphite HTTP / HTTPS ______________________________________ : YES
    JSON _______________________________________________________ : YES
    JSON HTTP / HTTPS __________________________________________ : YES
    OpenTSDB ___________________________________________________ : YES
    OpenTSDB HTTP / HTTPS ______________________________________ : YES
    All Metrics API ____________________________________________ : YES
    Shell (use metrics in shell scripts) _______________________ : YES
Debug/Developer Features:
    Trace All Netdata Allocations (with charts) ________________ : NO
    Developer Mode (more runtime checks, slower) _______________ : NO

Additional info

No response

[tkatsoulas]

Hello @MAH69IK, I understand your use case, but this is not something that we should enforce by default (grant CAP_NET_BIND_SERVICE). The fact that it was working before it was an issue/omission from our end which 99% introduced #17159. So I would suggest the following.

1. For your case you can add the CAP_NET_BIND_SERVICE in your netdata service file.
2. From our end we can have an admonition about this use case in our docs.

And just for the record, sorry for any inconvenience. Because we didn't communicate this limitation in the release notes. Most kindly saying :)

[ilyam8]

shouldn't the existing NET_RAW and NET_ADMIN allow to use port 80

@MAH69IK these caps are different according to docs

       CAP_NET_BIND_SERVICE
              Bind a socket to Internet domain privileged ports (port
              numbers less than 1024).

       CAP_NET_RAW
              •  Use RAW and PACKET sockets;
              •  bind to any address for transparent proxying.

       CAP_NET_ADMIN
              Perform various network-related operations:
              •  interface configuration;
              •  administration of IP firewall, masquerading, and
                 accounting;
              •  modify routing tables;
              •  bind to any address for transparent proxying;
              •  set type-of-service (TOS);
              •  clear driver statistics;
              •  set promiscuous mode;
              •  enabling multicasting;
              •  use [setsockopt(2)](https://man7.org/linux/man-pages/man2/setsockopt.2.html) to set the following socket options:
                 SO_DEBUG, SO_MARK, SO_PRIORITY (for a priority outside
                 the range 0 to 6), SO_RCVBUFFORCE, and SO_SNDBUFFORCE.

[ilyam8]

@MAH69IK we discussed your request and decided not to add CAP_NET_BIND_SERVICE to the list by default.

To do it yourself:

• edit netdata unit file

sudo systemctl edit netdata

• add the following:

[Service]
CapabilityBoundingSet=CAP_NET_BIND_SERVICE

• reload systemd daemon and restart netdata service

sudo systemctl daemon-reload
sudo systemctl restart netdata

[MAH69IK]

Okay. Thanks for the quick feedback!

[ilyam8]

@MAH69IK hey, out of curiosity - why do you bind Netdata to 80?

[MAH69IK]

@ilyam8 Two reasons. If I type the address in the browser manually, I don't need to specify the port. And, with a centralized configuration, I can just label the server "web" and get the right nftables rules without having to worry about individually configuring ports for multiple web services, each of which often use their own default ports.