100% CPU usage #80
Description
TT uses 100% of cpu after a few seconds.
When I implemented tt, no problems, just worked fine. Usage is pretty intensive, 700-1000 metrics every Telegraf 10s flush.
After various days, it started to use 100% of cpu, but still works. Restarting service keep CPU usage under control.
But now, it goes almost directly to 100% cpu.
I compile last version with same results.
I clean the data folder (dec 2024 and first days 2025, almost 1,6Gb ) and appears to works fine, but I want to figure what is happening.
ls -ltr ticktock-bkp2/2024/12
total 56
drwxr--r-- 32 root root 4096 Dec 2 14:06 1733097600.1733184000
drwxr--r-- 32 root root 4096 Dec 3 13:35 1733184000.1733270400
drwxr--r-- 32 root root 4096 Dec 4 16:58 1733270400.1733356800
drwxr--r-- 32 root root 4096 Dec 5 00:00 1733356800.1733443200
drwxr--r-- 32 root root 4096 Dec 9 15:13 1733702400.1733788800
drwxr--r-- 32 root root 4096 Dec 10 11:45 1733788800.1733875200
drwxr--r-- 32 root root 4096 Dec 11 11:49 1733875200.1733961600
drwxr--r-- 32 root root 4096 Dec 12 12:54 1733961600.1734048000
drwxr--r-- 32 root root 4096 Dec 13 13:04 1734048000.1734134400
drwxr--r-- 32 root root 4096 Dec 16 11:53 1734307200.1734393600
drwxr--r-- 32 root root 4096 Dec 17 12:13 1734393600.1734480000
drwxr--r-- 32 root root 4096 Dec 18 12:16 1734480000.1734566400
drwxr--r-- 32 root root 4096 Dec 19 14:02 1734566400.1734652800
drwxr--r-- 2 root root 4096 Jan 7 15:36 rollup
du -sk ticktock-bkp2/2024
852976 ticktock-bkp2/2024
ls -ltr ticktock-bkp2/2025/01
total 32
drwxr--r-- 2 root root 4096 Jan 7 15:36 rollup
drwxr--r-- 32 root root 4096 Jan 7 15:36 1736208000.1736294400
drwxr--r-- 32 root root 4096 Jan 8 00:01 1736294400.1736380800
drwxr--r-- 32 root root 4096 Jan 9 00:00 1736380800.1736467200
drwxr--r-- 32 root root 4096 Jan 10 00:00 1736467200.1736553600
drwxr--r-- 32 root root 4096 Jan 13 11:14 1736726400.1736812800
drwxr--r-- 32 root root 4096 Jan 14 10:28 1736812800.1736899200
drwxr--r-- 32 root root 4096 Jan 15 11:12 1736899200.1736985600
du -sk ticktock-bkp2/2025
793928 ticktock-bkp2/2025
Here's tt.conf:
# Remove the leading semicolon to enable the config.
# TickTock home directory.
# If specified, data will be stored under <ticktock.home>/data;
# logs will be stored under <ticktock.home>/log;
; ticktock.home = /etc/ticktock
tsdb.data.dir = /mnt/dietpi_userdata/ticktock
append.log.dir = /var/log/ticktock
# The HTTP server port number;
; http.server.port = 6182
# The TCP server port number;
# The first one accepts data in OpenTSDB's telnet format;
# The second one accepts data in InfluxDB's line protocol format;
# If any one of these are not used, omit it like this:
# tcp.server.port = ,6180 // only use InfluxDB's format;
; tcp.server.port = 6181,6180
# This size needs to be big enough to hold the largest HTTP request.
; tcp.buffer.size = 512kb
# How often should we flush data to disk. Default is 5 minutes.
# Which means you will lose the last 5 minutes of data if the
# server is terminated abnormally. Increasing this frequency
# will have a negative impact on performance, severely if more
# than once a minute.
; tsdb.flush.frequency = 5min
# Resolution of timestamps on data points;
# Either millisecond or second;
# This config can't be changed on existing databases;
tsdb.timestamp.resolution = millisecond
# Supported log levels: TRACE, DEBUG, TCP, HTTP, INFO, WARN, ERROR, FATAL
log.level = WARN
# How often to flush append logs?
# Note that data that came after last flush may be lost forever.
append.log.flush.frequency = 10s
Logs are empty:
ls -ltr /var/log/ticktock
total 0
Just in case, tt Telegraf's config:
[agent]
interval = "200ms"
round_interval = true
metric_batch_size = 1000
metric_buffer_limit = 10000
collection_jitter = "0s"
flush_interval = "60s"
flush_jitter = "5s"
precision = ""
hostname = ""
omit_hostname = false
[[outputs.opentsdb]]
prefix = "telegraf."
host = "tcp://127.0.0.1"
port = 6181
How can debug this behavior?
Thanks!!