[BUG] Reloading via systemctl kills carbonapi process #552

ryangsteele · 2021-01-12T20:10:47Z

Describe the bug
Upon issuing a systemctl reload carbonapi.service, the carbonapi process dies instead of being gracefully restarted.

CarbonAPI Version
0.14.1-1 (latest stable Debian package from packagecloud)

Logs
Running journalctl -u carbonapi.service after a systemctl reload carbonapi.service yields the following errors:

Jan 12 19:37:43 carbonapi.test.host systemd[1]: Reloading CarbonAPI - replacement for graphite-web written in Go.
Jan 12 19:37:43 carbonapi.test.host systemd[1]: Reloaded CarbonAPI - replacement for graphite-web written in Go.
Jan 12 19:37:43 carbonapi.test.host carbonapi[3711]: 2021-01-12T19:37:43.426Z        INFO        main        will parse config as yaml        {"config_file": "/etc/carbonapi/carbonapi.yaml"}
Jan 12 19:37:43 carbonapi.test.host carbonapi[3711]: 2021-01-12T19:37:43.426Z        INFO        main        will parse config as yaml        {"config_file": "/etc/carbonapi/carbonapi.yaml"}
Jan 12 19:37:43 carbonapi.test.host carbonapi[3711]: 2021-01-12T19:37:43.432Z        INFO        main        cache: in-memory cache configured
Jan 12 19:37:43 carbonapi.test.host carbonapi[3711]: 2021-01-12T19:37:43.432Z        INFO        main        backendCache: in-memory cache configured
Jan 12 19:37:43 carbonapi.test.host carbonapi[3711]: 2021-01-12T19:37:43.432Z        INFO        main        cache: in-memory cache configured
Jan 12 19:37:43 carbonapi.test.host carbonapi[3711]: 2021-01-12T19:37:43.433Z        INFO        main        backendCache: in-memory cache configured
Jan 12 19:37:43 carbonapi.test.host carbonapi[3711]: 2021-01-12T19:37:43.432Z        INFO        main        starting carbonapi        {"build_version": "0.14.1", "config": "{\"ExtrapolateExperiment\":false,\"Logger\":[{\"logger\":\"\",\"file\":\"/var/log/carbonapi/carbonapi.log\",\"level\":\"error\",\"encoding\":\"console\",\"encoding-time\":\"iso8601
\",\"encoding-duration\":\"seconds\",\"sample-tick\":\"\",\"sample-initial\":0,\"sample-thereafter\":0}],\"Listen\":\"0.0.0.0:8082\",\"Buckets\":10,\"Concurency\":1000,\"ResponseCacheConfig\":{\"Type\":\"mem\",\"Size\":0,\"MemcachedServers\":[],\"DefaultTimeoutSec\":60},\"BackendCacheConfig\":{\"Type\":\"mem\",\"Size\":0,\"MemcachedServers\":null,\"DefaultTimeoutS
ec\":60},\"Cpus\":0,\"TimezoneString\":\"\",\"UnicodeRangeTables\":null,\"Graphite\":{\"Pattern\":\"{prefix}.{fqdn}\",\"Host\":\"127.0.0.1:2003\",\"Interval\":60000000000,\"Prefix\":\"carbon.api\"},\"IdleConnections\":1000,\"PidFile\":\"/var/run/carbonapi/carbonapi.pid\",\"SendGlobsAsIs\":null,\"AlwaysSendGlobsAsIs\":null,\"MaxBatchSize\":0,\"Zipper\":\"\",\"Upstr
eams\":{\"ConcurrencyLimitPerServer\":0,\"MaxIdleConnsPerHost\":100,\"Backends\":null,\"BackendsV2\":{\"Backends\":[{\"GroupName\":\"go-carbon\",\"Protocol\":\"carbonapi_v3_pb\",\"LBMethod\":\"broadcast\",\"Servers\":[\"http://127.0.0.1:8081\"],\"Timeouts\":{\"Find\":2000000000,\"Render\":10000000000,\"Connect\":200000000},\"ConcurrencyLimit\":0,\"KeepAliveInterva
l\":30000000000,\"MaxIdleConnsPerHost\":100,\"MaxTries\":3,\"MaxBatchSize\":0,\"BackendOptions\":null,\"ForceAttemptHTTP2\":false,\"DoMultipleRequestsIfSplit\":false}],\"MaxIdleConnsPerHost\":0,\"ConcurrencyLimitPerServer\":0,\"Timeouts\":{\"Find\":2000000000,\"Render\":10000000000,\"Connect\":200000000},\"KeepAliveInterval\":0,\"MaxTries\":0,\"MaxBatchSize\":0},\
"MaxBatchSize\":0,\"FallbackMaxBatchSize\":0,\"MaxTries\":0,\"DoMultipleRequestsIfSplit\":false,\"CarbonSearch\":{\"Backend\":\"\",\"Prefix\":\"virt.v1.*\"},\"CarbonSearchV2\":{\"Backends\":null,\"MaxIdleConnsPerHost\":0,\"ConcurrencyLimitPerServer\":0,\"Timeouts\":{\"Find\":0,\"Render\":0,\"Connect\":0},\"KeepAliveInterval\":0,\"MaxTries\":0,\"MaxBatchSize\":0,\"
Prefix\":\"\"},\"ExpireDelaySec\":0,\"TLDCacheDisabled\":true,\"InternalRoutingCache\":600000000000,\"Timeouts\":{\"Find\":2000000000,\"Render\":10000000000,\"Connect\":200000000},\"KeepAliveInterval\":30000000000,\"ScaleToCommonStep\":true},\"ExpireDelaySec\":600,\"GraphiteWeb09Compatibility\":false,\"IgnoreClientTimeout\":false,\"DefaultColors\":null,\"GraphTemp
lates\":\"\",\"FunctionsConfigs\":{},\"HeadersToPass\":[\"X-Dashboard-Id\",\"X-Grafana-Org-Id\",\"X-Panel-Id\"],\"HeadersToLog\":[\"X-Dashboard-Id\",\"X-Grafana-Org-Id\",\"X-Panel-Id\"],\"Define\":null,\"Prefix\":\"\",\"Expvar\":{\"Listen\":\"\",\"Enabled\":true,\"PProfEnabled\":false},\"NotFoundStatusCode\":200,\"HTTPResponseStackTrace\":true}"}
Jan 12 19:37:43 carbonapi.test.host carbonapi[3711]: 2021-01-12T19:37:43.434Z        INFO        main        expvar handler will listen on a separate address/port        {"expvar_listen": "", "pprof_enabled": false}
Jan 12 19:37:43 carbonapi.test.host carbonapi[3711]: 2021-01-12T19:37:43.434Z        FATAL        main        failed to start http server        {"error": "error inheriting socket fd 3: file file+net listener: fcntl: bad file descriptor"}
Jan 12 19:37:43 carbonapi.test.host carbonapi[3711]: 2021-01-12T19:37:43.435Z        INFO        main        starting carbonapi        {"build_version": "0.14.1", "config": "{\"ExtrapolateExperiment\":false,\"Logger\":[{\"logger\":\"\",\"file\":\"/var/log/carbonapi/carbonapi.log\",\"level\":\"error\",\"encoding\":\"console\",\"encoding-time\":\"iso8601
\",\"encoding-duration\":\"seconds\",\"sample-tick\":\"\",\"sample-initial\":0,\"sample-thereafter\":0}],\"Listen\":\"0.0.0.0:8082\",\"Buckets\":10,\"Concurency\":1000,\"ResponseCacheConfig\":{\"Type\":\"mem\",\"Size\":0,\"MemcachedServers\":[],\"DefaultTimeoutSec\":60},\"BackendCacheConfig\":{\"Type\":\"mem\",\"Size\":0,\"MemcachedServers\":null,\"DefaultTimeoutS
ec\":60},\"Cpus\":0,\"TimezoneString\":\"\",\"UnicodeRangeTables\":null,\"Graphite\":{\"Pattern\":\"{prefix}.{fqdn}\",\"Host\":\"127.0.0.1:2003\",\"Interval\":60000000000,\"Prefix\":\"carbon.api\"},\"IdleConnections\":1000,\"PidFile\":\"/var/run/carbonapi/carbonapi.pid\",\"SendGlobsAsIs\":null,\"AlwaysSendGlobsAsIs\":null,\"MaxBatchSize\":0,\"Zipper\":\"\",\"Upstr
eams\":{\"ConcurrencyLimitPerServer\":0,\"MaxIdleConnsPerHost\":100,\"Backends\":null,\"BackendsV2\":{\"Backends\":[{\"GroupName\":\"go-carbon\",\"Protocol\":\"carbonapi_v3_pb\",\"LBMethod\":\"broadcast\",\"Servers\":[\"http://127.0.0.1:8081\"],\"Timeouts\":{\"Find\":2000000000,\"Render\":10000000000,\"Connect\":200000000},\"ConcurrencyLimit\":0,\"KeepAliveInterva
l\":30000000000,\"MaxIdleConnsPerHost\":100,\"MaxTries\":3,\"MaxBatchSize\":0,\"BackendOptions\":null,\"ForceAttemptHTTP2\":false,\"DoMultipleRequestsIfSplit\":false}],\"MaxIdleConnsPerHost\":0,\"ConcurrencyLimitPerServer\":0,\"Timeouts\":{\"Find\":2000000000,\"Render\":10000000000,\"Connect\":200000000},\"KeepAliveInterval\":0,\"MaxTries\":0,\"MaxBatchSize\":0},\
"MaxBatchSize\":0,\"FallbackMaxBatchSize\":0,\"MaxTries\":0,\"DoMultipleRequestsIfSplit\":false,\"CarbonSearch\":{\"Backend\":\"\",\"Prefix\":\"virt.v1.*\"},\"CarbonSearchV2\":{\"Backends\":null,\"MaxIdleConnsPerHost\":0,\"ConcurrencyLimitPerServer\":0,\"Timeouts\":{\"Find\":0,\"Render\":0,\"Connect\":0},\"KeepAliveInterval\":0,\"MaxTries\":0,\"MaxBatchSize\":0,\"
Prefix\":\"\"},\"ExpireDelaySec\":0,\"TLDCacheDisabled\":true,\"InternalRoutingCache\":600000000000,\"Timeouts\":{\"Find\":2000000000,\"Render\":10000000000,\"Connect\":200000000},\"KeepAliveInterval\":30000000000,\"ScaleToCommonStep\":true},\"ExpireDelaySec\":600,\"GraphiteWeb09Compatibility\":false,\"IgnoreClientTimeout\":false,\"DefaultColors\":null,\"GraphTemp
lates\":\"\",\"FunctionsConfigs\":{},\"HeadersToPass\":[\"X-Dashboard-Id\",\"X-Grafana-Org-Id\",\"X-Panel-Id\"],\"HeadersToLog\":[\"X-Dashboard-Id\",\"X-Grafana-Org-Id\",\"X-Panel-Id\"],\"Define\":null,\"Prefix\":\"\",\"Expvar\":{\"Listen\":\"\",\"Enabled\":true,\"PProfEnabled\":false},\"NotFoundStatusCode\":200,\"HTTPResponseStackTrace\":true}"}
Jan 12 19:37:43 carbonapi.test.host carbonapi[3711]: 2021-01-12T19:37:43.436Z        INFO        main        expvar handler will listen on a separate address/port        {"expvar_listen": "", "pprof_enabled": false}
Jan 12 19:37:43 carbonapi.test.host carbonapi[3711]: 2021-01-12T19:37:43.437Z        FATAL        main        failed to start http server        {"error": "error inheriting socket fd 3: file file+net listener: fcntl: bad file descriptor"}

CarbonAPI Configuration:
Note: I have tried with pidFile set to the empty string ("") as well as having it set to the same pidfile defined in /etc/systemd/system/carbonapi.service (as suggested in #426), the latter of which is shown below:

listen: "0.0.0.0:8082"
pidFile: "/var/run/carbonapi/carbonapi.pid"
prefix: ""
expvar:
  enabled: true
  pprofEnabled: false
  listen: ""
graphite:
    host: "127.0.0.1:2003"
    interval: "60s"
    prefix: "carbon.api"
    pattern: "{prefix}.{fqdn}"
headersToPass:
  - "X-Dashboard-Id"
  - "X-Grafana-Org-Id"
  - "X-Panel-Id"
headersToLog:
  - "X-Dashboard-Id"
  - "X-Grafana-Org-Id"
  - "X-Panel-Id"
notFoundStatusCode: 200
concurency: 1000
cache:
   type: "mem"
   size_mb: 0
   defaultTimeoutSec: 60
backendCache:
   type: "mem"
   size_mb: 0
   defaultTimeoutSec: 60
cpus: 0
tz: ""
maxBatchSize: 0
idleConnections: 1000
upstreams:
    tldCacheDisabled: true
    buckets: 10
    timeouts:
        find: "2s"
        render: "10s"
        connect: "200ms"
    concurrencyLimitPerServer: 0
    keepAliveInterval: "30s"
    maxIdleConnsPerHost: 100
    doMultipleRequestsIfSplit: false
    backendsv2:
        backends:
          -
            groupName: "go-carbon"
            protocol: "carbonapi_v3_pb"
            lbMethod: "broadcast"
            maxTries: 3
            maxBatchSize: 0
            keepAliveInterval: "30s"
            concurrencyLimit: 0
            maxIdleConnsPerHost: 100
            forceAttemptHTTP2: false
            doMultipleRequestsIfSplit: false
            timeouts:
                find: "2s"
                render: "10s"
                connect: "200ms"
            servers:
                - "http://127.0.0.1:8081"
    graphite09compat: false
logger:
    - logger: ""
      file: "/var/log/carbonapi/carbonapi.log"
      level: "error"
      encoding: "console"
      encodingTime: "iso8601"
      encodingDuration: "seconds"

Simplified query (if applicable)
N/A

Backend metric retention and aggregation schemas
N/A

Backend response (if possible)
N/A

Additional context
Contents of /etc/systemd/system/carbonapi.service:

[Unit]
Description=CarbonAPI - replacement for graphite-web written in Go
After=syslog.target network.target

[Service]
Type=simple
EnvironmentFile=/etc/default/carbonapi
ExecStart=/usr/bin/carbonapi $CARBONAPI_ARGS
ExecReload=/bin/kill -USR2 $MAINPID
PIDFile=/var/run/carbonapi/carbonapi.pid
LimitNOFILE=200000
RuntimeDirectory=carbonapi
RuntimeDirectoryMode=0755

[Install]
WantedBy=multi-user.target

Contents of /etc/default/carbonapi:

CARBONAPI_ARGS="-config /etc/carbonapi/carbonapi.yaml"

If there is no desire to support reloads going forward, I can remove that capability as part of #551.

The text was updated successfully, but these errors were encountered:

Civil · 2021-01-12T21:02:25Z

Last time I've used it was about January 2018 I think. Back then it was working, however it is possible that it no longer works in recent versions of Go or in some specific combinations of Go and kernel (library that was used to support that functionality is archives by authors, without any reasons specified).

However no one ever complained before but I don't know if that's because no one used it or it's working in most of the cases.

I'll try to have a look and at least reproduce it. (but no promise on a timeline though, sorry).

ryangsteele · 2021-01-13T16:11:12Z

No worries @Civil - for now I can just use restart upon config changes.

I'll try and dig in to this more as time permits too. The kernels I tested on were all in the 4.15.0 branch (4.15.0-112.113 and 4.15.0-130.134). If there's any other information I've left out that would be useful, just let me know.

Thanks for the prompt reply!

* Remove gracehttp as it is no longer supported (repo was archives) and it was reported to be broken (#552) * Force reuseport on listening sockets. That would allow to start multiple carbonapi binaries

Civil · 2021-05-18T00:08:05Z

As reload feature was removed since the bug was open, I'll reclassify it as "enhancement" as I'm open to re-introduce that back, if there will be a reasonable way to support reload (by reasonable - that will still compile on OSs that doesn't have systemd, for example, as I think another way is to use sd-* helpers there and bindings for systemd, but I'm not sure it's easy to make that optional)

ryangsteele added the bug label Jan 12, 2021

Civil added enhancement help wanted and removed bug labels May 18, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BUG] Reloading via systemctl kills carbonapi process #552

[BUG] Reloading via systemctl kills carbonapi process #552

ryangsteele commented Jan 12, 2021 •

edited

Civil commented Jan 12, 2021

ryangsteele commented Jan 13, 2021

Civil commented May 18, 2021

[BUG] Reloading via systemctl kills carbonapi process #552

[BUG] Reloading via systemctl kills carbonapi process #552

Comments

ryangsteele commented Jan 12, 2021 • edited

Civil commented Jan 12, 2021

ryangsteele commented Jan 13, 2021

Civil commented May 18, 2021

ryangsteele commented Jan 12, 2021 •

edited