Skip to content

Services restart during first boot, flooding the system log #1831

Description

@Tbaile

Steps to reproduce

  • Boot a machine with a freshly flashed image, or reset an existing one with firstboot && reboot
  • Wait for the boot to complete
  • Inspect the boot messages:
tail -f /var/log/messages | grep -E "nginx|netifyd|vmalert|telegraf|victoria"

Expected behavior

Every service starts once, after the services it needs are already listening. The boot
log contains no connection errors between local services and no duplicate service
starts.

Actual behavior

Several services start before the services they talk to, so they either get restarted
later in the boot sequence or keep retrying until their dependency is up. Both cases
fill the log with avoidable messages at every boot:

  • nginx starts before ns-ui and ns-reverse-proxy have generated its location
    files, so it starts with an incomplete configuration and ns-ui restarts it right
    after — nginx is started twice on every boot
  • netifyd starts before ns-flows and ns-stats, so the traffic data it sends to
    them fails until they are listening
  • rsyslog starts forwarding to victoria-logs long before victoria-logs is up, so
    early boot messages are lost and the forwarding errors are logged
  • vmalert and telegraf start before victoria-metrics, so their queries and metric
    writes fail until it is up

Components

Image version: 8.8.0
netifyd 5.2.6
nginx-ssl 1.26.3
ns-monitoring 1.2.1 (ns-flows, ns-stats)
ns-reverse-proxy 1.0.0
ns-ui 2.23.2
rsyslog 8.2506.0
telegraf 1.39.1
victoria-logs 1.51.0
victoria-metrics 1.146.0

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    Projects

    Status
    ToDo 🕐

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions