[Beta Support]: Potential performance regression in 0.17.0-beta1 #21490

ivitjuk · 2025-12-31T15:00:34Z

ivitjuk
Dec 31, 2025

Describe the problem you are having

I have noticed that 15min system load has increased by ~0.3 on my Frigate machine after upgrading to 0.17.0-beta1 (chart bellow). While this does not seem to be significant, I wanted to report it. It looks like this is caused by some kind of a maintenance process which constantly takes ~10% of CPU due to high rate (~50/s) of clone3()->accept4()->read()/write() chain of system calls.

While this could be working as expected, it seems absurd to me that a maintenance process takes more compute resources than a ffmpeg capture process (~7% CPU utilization).

See bellow for data points I collected.

15min system load

Offending process

root     1497902  9.8  1.7 2400388 137872 ?      Sl   06:40  53:18      |           \_ /usr/bin/python3 -Wignore:::numpy.core.getlimits -c from multiprocessing.forkserver import main; main(5, 7, ['sqlite3', 'numpy', 'cv2', 'peewee', 'zmq', 'ruamel.yaml', 'frigate.camera.maintainer'], **{'sys_path': ['/opt/frigate', '/usr/lib/python311.zip', '/usr/lib/python3.11', '/usr/lib/python3.11/lib-dynload', '/usr/local/lib/python3.11/dist-packages', '/usr/lib/python3/dist-packages']})

Excessive cloning seems to be the culprit

❯ sudo timeout  10 strace -f -p 1497902 -e clone3|& grep clone3|wc                     
    562    6740  175732                                                                
❯ bc -ql                                                                               
562/10                                                                                 
56.20000000000000000000

Beta Version

0.17.0-beta1

Issue Category

Performance / Stability

Frigate config file

version: 0.17-0

auth:
  enabled: true
  trusted_proxies:
    - 172.0.0.0/8

tls:
  enabled: false

mqtt:
  enabled: true
  host: mosquitto

timestamp_style:
  position: br

record:
  enabled: true
  alerts:
    retain:
      days: 10
      mode: all
  detections:
    retain:
      days: 10
      mode: all

audio:
  min_volume: 6500

snapshots:
  enabled: true
  clean_copy: true
  crop: false
  timestamp: true
  retain:
    default: 10

motion:
  mqtt_off_delay: 10

face_recognition:
  enabled: false

detect:
  enabled: true
  fps: 5
  annotation_offset: -600
  width: 1280
  height: 720

detectors:
  ov_1:
    type: openvino
    device: GPU
  ov_2:
    type: openvino
    device: GPU

model:
  model_type: yolonas
  width: 320
  height: 320
  input_tensor: nchw
  input_pixel_format: bgr
  path: /config/yolo_nas_s.onnx
  labelmap_path: /labelmap/coco-80.txt

go2rtc:
  streams:
    street:
      - rtsp://admin:...@192.168.1.64:554
      - ffmpeg:street#audio=opus
    entrance:
      - rtsp://admin:...@192.168.1.62:554
      - ffmpeg:entrance#audio=opus
    terace:
      - rtsp://admin:...@192.168.1.61:554
      - ffmpeg:terace#audio=opus
    # spare:
    #   - rtsp://admin:...@192.168.1.63:554

ffmpeg:
  hwaccel_args: preset-vaapi
  input_args: -rtsp_transport tcp -timeout 5000000 -fflags nobuffer -flags low_delay -use_wallclock_as_timestamps 1
  output_args:
    record: preset-record-generic-audio-copy

cameras:
  street:
    ui:
      order: 0
    detect:
      width: 2304
      height: 1296
      stationary:
        interval: 50
        threshold: 15
    mqtt:
      crop: false
      height: 640
      quality: 85
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/street
          roles:
            - detect
            - record
            - audio
    audio:
      enabled: true
      listen:
        - alarm
        - fire_alarm
        - car_alarm
    motion:
      improve_contrast: false 
      lightning_threshold: 0.3 
      mask: 
        0.039,1,0.775,1,0.55,0.336,0.543,0.263,0.522,0.255,0.499,0.192,0.507,0.114,0.625,0.221,0.781,0.398,1,0.703,1,0,0,0,0,0.196,0.097,0.168,0.115,0.227,0.199,0.221,0.194,0.144,0.471,0.072,0.453,0.288,0.283,0.398,0.241,0.537,0.415,0.546,0.456,0.923,0.312,0.927,0.308,0.553,0.266,0.553,0.272,0.924,0.039,0.931
    objects:
      track:
        - person
        - car
      filters:
        person:
          min_score: 0.2  
          threshold: 0.5  
    zones:
      frontyard:
        coordinates: 0.057,0.417,0.419,0.454,0.474,1,0,1,0,0.683
        objects: person
      stairs:
        coordinates: 0.051,0.323,0.504,0.145,0.488,0.281,0.058,0.433
        objects: person
        filters:
          person:
            min_area: 2000
            max_area: 20000 

      parking_north:
        coordinates: 0.778,1,1,1,1,0.773,0.833,0.499,0.614,0.511
        objects:
          - person
          - car
        filters:
          person:
            min_area: 30000 
        loitering_time: 0
        inertia: 3
      parking_south:
        coordinates: 0.617,0.528,0.843,0.512,0.627,0.222,0.497,0.182
        objects: person
        filters:
          person:
            min_area: 30000 
        loitering_time: 0
        inertia: 3
    review:
      alerts:
        required_zones:
          - frontyard
          - stairs
          - parking_north
          - parking_south
  entrance:
    ui:
      order: 1
    mqtt:
      crop: false
      height: 640
      quality: 85
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/entrance
          roles:
            - detect
            - record
    objects:
      track:
        - person
        - dog
        - cat
        - bird
    motion:
      improve_contrast: false 
      contour_area: 200 
      mask: 0,0.934,0.332,0.93,0.332,1,0,1
  terace:
    ui:
      order: 2
    mqtt:
      crop: false
      height: 640
      quality: 85
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/terace
          roles:
            - detect
            - record
    objects:
      track:
        - person
      filters:
        person:
          max_ratio: 0.65 
    motion:
      improve_contrast: false 
      contour_area: 200 
      mask:
        - 1,0.392,1,0,0.737,0,0.722,0.273
        - 0.338,0.937,0,0.936,0,1,0.339,1

Relevant Frigate log output

N/A

Relevant go2rtc log output (if applicable)

No response

Install method

Docker Compose

docker-compose file or Docker CLI command

frigate:
    container_name: frigate
    image: ghcr.io/blakeblackshear/frigate:0.17.0-beta1
    privileged: true
    restart: always
    depends_on:
      mosquitto:
        condition: service_healthy
    entrypoint: ["/bin/bash", "/frigate-entrypoint.sh"]
    shm_size: 300mb
    deploy:
      resources:
        limits:
          memory: 3G
    labels:
      - "rsyslog_enabled"
    devices:
      - /dev/bus/usb:/dev/bus/usb
      - /dev/apex_0:/dev/apex_0
      - /dev/dri/renderD128
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - ./config/frigate/config.yml:/config/config.yml
      - ./config/frigate/yolo_nas_s.onnx:/config/yolo_nas_s.onnx
      - ./config/frigate/frigate-entrypoint.sh:/frigate-entrypoint.sh
      - ./volumes/frigate/media:/media/frigate/
      - ./volumes/frigate/config:/config/
      - type: tmpfs
        target: /tmp/cache
        tmpfs:
          size: 1000000000
    environment:
      TZ: Europe/Belgrade
      FRIGATE_RTSP_PASSWORD: ...
      LIBVA_DRIVER_NAME: i965
    secrets:
      - source: frigate_api
        target: /run/secrets/PLUS_API_KEY

Operating system

Debian

CPU / GPU / Hardware

No response

Screenshots

No response

Steps to reproduce

No response

Any other information that may be helpful

No response

NickM-27 · 2025-12-31T15:03:51Z

NickM-27
Dec 31, 2025
Collaborator Sponsor

It's not clear which process is actually being pointed at here. All we can see is the start command which pulls in from forkserver and a few pre-loaded imports (including camera maintainer) which greatly improves startup time.

2 replies

ivitjuk Dec 31, 2025
Author

Following is the full Frigate process tree. I believe the offending process is 1497902. I am not too familiar with debugging Python code, but I will try to investigate more in a hope to narrow down the related piece of code that is generating this load.

root     1497409  0.1  0.1 1238360 14100 ?       Sl   06:40   0:35 /usr/bin/containerd-shim-runc-v2 -namespace moby -id 90a89f0ed4f49043368c144776bef005d8e82ec0385bd21179f5fa8aa1b475e8 -address /run/containerd/containerd.sock
root     1497432  0.0  0.0    440   104 ?        Ss   06:40   0:00  \_ /package/admin/s6/command/s6-svscan -d4 -- /run/service
root     1497473  0.0  0.0    220    72 ?        S    06:40   0:00      \_ s6-supervise s6-linux-init-shutdownd
root     1497474  0.0  0.0    208     4 ?        Ss   06:40   0:00      |   \_ /package/admin/s6-linux-init/command/s6-linux-init-shutdownd -d3 -c /run/s6/basedir -g 3000 -C -B
root     1497483  0.0  0.0    220    80 ?        S    06:40   0:00      \_ s6-supervise frigate-log
nobody   1497551  0.0  0.0    292     4 ?        Ss   06:40   0:00      |   \_ s6-log -b -- T 1 n0 s10000000 T /dev/shm/logs/frigate
root     1497484  0.0  0.0    220    76 ?        S    06:40   0:00      \_ s6-supervise frigate
root     1497594  7.9  5.5 3710420 446300 ?      Ssl  06:40  45:20      |   \_ python3 -u -m frigate
root     1497894  0.0  0.1  16876 12396 ?        S    06:40   0:00      |       \_ /usr/bin/python3 -Wignore:::numpy.core.getlimits -c from multiprocessing.resource_tracker import main;main(5)
root     1497895  0.0  2.6 818992 209784 ?       S    06:40   0:04      |       \_ /usr/bin/python3 -Wignore:::numpy.core.getlimits -c from multiprocessing.forkserver import main; main(5, 7, ['sqlite3', 'numpy', 'cv2', 'peewee', 'zmq', 'ruamel.yaml', 'frigate.camera.maintainer'], **{'sys_path': ['/opt/frigate', '/usr/lib/python311.zip', '/usr/lib/python3.11', '/usr/lib/python3.11/lib-dynload', '/usr/local/lib/python3.11/dist-packages', '/usr/lib/python3/dist-packages']})
root     1497902  9.8  1.7 2400404 137860 ?      Sl   06:40  55:56      |           \_ /usr/bin/python3 -Wignore:::numpy.core.getlimits -c from multiprocessing.forkserver import main; main(5, 7, ['sqlite3', 'numpy', 'cv2', 'peewee', 'zmq', 'ruamel.yaml', 'frigate.camera.maintainer'], **{'sys_path': ['/opt/frigate', '/usr/lib/python311.zip', '/usr/lib/python3.11', '/usr/lib/python3.11/lib-dynload', '/usr/local/lib/python3.11/dist-packages', '/usr/lib/python3/dist-packages']})
root     1498081  2.7  2.1 1365976 174348 ?      Sl   06:40  15:27      |           \_ frigate.recording_manager
root     1498094  1.0  2.0 1413932 161600 ?      SNl  06:40   5:51      |           \_ frigate.review_segment_manager
root     1498104  1.7  2.3 1877924 185680 ?      Sl   06:40   9:43      |           \_ frigate.embeddings_manager
root     1498135  0.7  5.4 1976864 434512 ?      Sl   06:40   4:04      |           \_ frigate.detector:ov_1
root     1498146  0.7  5.4 1976864 434984 ?      Sl   06:40   4:04      |           \_ frigate.detector:ov_2
root     1498158  2.3  2.3 2269904 185144 ?      SNl  06:40  13:22      |           \_ frigate.output
root     1498191  0.0  0.2 157584 19416 ?        SNs  06:40   0:00      |           |   \_ /usr/lib/ffmpeg/7.0/bin/ffmpeg -threads 1 -f rawvideo -pix_fmt yuv420p -video_size 2304x1296 -i pipe: -threads 1 -f mpegts -s 1280x720 -codec:v mpeg1video -q 8 -bf 0 pipe:
root     1498205  0.0  0.2 157584 17036 ?        SNs  06:40   0:00      |           |   \_ /usr/lib/ffmpeg/7.0/bin/ffmpeg -threads 1 -f rawvideo -pix_fmt yuv420p -video_size 1280x720 -i pipe: -threads 1 -f mpegts -s 1280x720 -codec:v mpeg1video -q 8 -bf 0 pipe:
root     1498211  0.0  0.2 157584 16948 ?        SNs  06:40   0:00      |           |   \_ /usr/lib/ffmpeg/7.0/bin/ffmpeg -threads 1 -f rawvideo -pix_fmt yuv420p -video_size 1280x720 -i pipe: -threads 1 -f mpegts -s 1280x720 -codec:v mpeg1video -q 8 -bf 0 pipe:
root     1498218  0.0  0.5 528380 41644 ?        SNsl 06:40   0:08      |           |   \_ /usr/lib/ffmpeg/7.0/bin/ffmpeg -threads 1 -f rawvideo -pix_fmt yuv420p -video_size 1280x720 -i pipe: -threads 1 -f mpegts -s 1280x720 -codec:v mpeg1video -q 8 -bf 0 pipe:
root     1498183  3.9  2.0 1202092 160564 ?      Sl   06:40  22:27      |           \_ frigate.audio_manager
root     1498236  1.3  0.4 528960 33472 ?        Ssl  06:40   7:42      |           |   \_ /usr/lib/ffmpeg/7.0/bin/ffmpeg -vn -threads 1 -hide_banner -loglevel warning -threads 2 -rtsp_transport tcp -timeout 5000000 -fflags nobuffer -flags low_delay -use_wallclock_as_timestamps 1 -i rtsp://127.0.0.1:8554/street -threads 1 -f s16le -ar 16000 -ac 1 -y pipe:
root     1498210  4.2  2.8 1558148 226740 ?      Sl   06:40  24:18      |           \_ frigate.process:street
root     1498289  7.3  2.0 1386364 164772 ?      Sl   06:40  41:42      |           \_ frigate.capture:street
root     1498325 15.3  1.8 800824 151284 ?       Ssl  06:40  87:05      |           |   \_ /usr/lib/ffmpeg/7.0/bin/ffmpeg -hide_banner -loglevel warning -threads 2 -hwaccel_flags allow_profile_mismatch -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -hwaccel_output_format vaapi -rtsp_transport tcp -timeout 5000000 -fflags nobuffer -flags low_delay -use_wallclock_as_timestamps 1 -i rtsp://127.0.0.1:8554/street -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copy /tmp/cache/street@%Y%m%d%H%M%S%z.mp4 -r 5 -vf fps=5,scale_vaapi=w=2304:h=1296,hwdownload,format=nv12,eq=gamma=1.4:gamma_weight=0.5 -threads 2 -f rawvideo -pix_fmt yuv420p pipe:
root     1498353  1.0  1.8 1435168 148860 ?      Sl   06:40   5:54      |           \_ frigate.process:entrance
root     1498411  2.8  1.8 1351128 148988 ?      Sl   06:40  16:08      |           \_ frigate.capture:entrance
root     1498455  6.7  0.9 792060 79348 ?        Ssl  06:40  37:59      |           |   \_ /usr/lib/ffmpeg/7.0/bin/ffmpeg -hide_banner -loglevel warning -threads 2 -hwaccel_flags allow_profile_mismatch -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -hwaccel_output_format vaapi -rtsp_transport tcp -timeout 5000000 -fflags nobuffer -flags low_delay -use_wallclock_as_timestamps 1 -i rtsp://127.0.0.1:8554/entrance -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copy /tmp/cache/entrance@%Y%m%d%H%M%S%z.mp4 -r 5 -vf fps=5,scale_vaapi=w=1280:h=720,hwdownload,format=nv12,eq=gamma=1.4:gamma_weight=0.5 -threads 2 -f rawvideo -pix_fmt yuv420p pipe:
root     1498491  1.0  1.8 1435876 147228 ?      Sl   06:40   5:56      |           \_ frigate.process:terace
root     1498569  2.8  1.8 1351128 150760 ?      Sl   06:40  16:15      |           \_ frigate.capture:terace
root     1498615  6.7  1.0 792024 80420 ?        Ssl  06:40  38:03      |               \_ /usr/lib/ffmpeg/7.0/bin/ffmpeg -hide_banner -loglevel warning -threads 2 -hwaccel_flags allow_profile_mismatch -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -hwaccel_output_format vaapi -rtsp_transport tcp -timeout 5000000 -fflags nobuffer -flags low_delay -use_wallclock_as_timestamps 1 -i rtsp://127.0.0.1:8554/terace -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copy /tmp/cache/terace@%Y%m%d%H%M%S%z.mp4 -r 5 -vf fps=5,scale_vaapi=w=1280:h=720,hwdownload,format=nv12,eq=gamma=1.4:gamma_weight=0.5 -threads 2 -f rawvideo -pix_fmt yuv420p pipe:
root     1497485  0.0  0.0    220    80 ?        S    06:40   0:00      \_ s6-supervise s6rc-oneshot-runner
root     1497506  0.0  0.0    208    64 ?        Ss   06:40   0:00      |   \_ /package/admin/s6/command/s6-ipcserverd -1 -- /package/admin/s6/command/s6-ipcserver-access -v0 -E -l0 -i data/rules -- /package/admin/s6/command/s6-sudod -t 30000 -- /package/admin/s6-rc/command/s6-rc-oneshot-run -l ../.. --
root     1497486  0.0  0.0    220    76 ?        S    06:40   0:00      \_ s6-supervise nginx-log
nobody   1497553  0.0  0.0    296     4 ?        Ss   06:40   0:00      |   \_ s6-log -b -- T 1 n0 s10000000 T /dev/shm/logs/nginx
root     1497487  0.0  0.0    220    80 ?        S    06:40   0:00      \_ s6-supervise go2rtc
root     1497584  7.1  0.4 1242844 34188 ?       Ssl  06:40  40:31      |   \_ /usr/local/go2rtc/bin/go2rtc -config=/config/go2rtc_homekit.yml -config=/dev/shm/go2rtc.yaml
root     1497488  0.0  0.0    220    76 ?        S    06:40   0:00      \_ s6-supervise certsync-log
nobody   1497554  0.0  0.0    292     4 ?        Ss   06:40   0:00      |   \_ s6-log -b -- T 1 n0 s10000000 T /dev/shm/logs/certsync
root     1497489  0.0  0.0    220    76 ?        S    06:40   0:00      \_ s6-supervise nginx
root     1497601  0.0  0.1 589932 15024 ?        Ss   06:40   0:00      |   \_ nginx: master process nginx
root     1497644  0.0  0.0 853232  7260 ?        Sl   06:40   0:06      |       \_ nginx: worker process
root     1497645  0.0  0.0 853232  7308 ?        Sl   06:40   0:04      |       \_ nginx: worker process
root     1497646  0.0  0.0 853232  7272 ?        Sl   06:40   0:02      |       \_ nginx: worker process
root     1497648  0.0  0.0 853232  7304 ?        Sl   06:40   0:03      |       \_ nginx: worker process
root     1497660  0.0  0.0 590136  5080 ?        S    06:40   0:00      |       \_ nginx: cache manager process
root     1497490  0.0  0.0    220    76 ?        S    06:40   0:00      \_ s6-supervise certsync
root     1497796  0.0  0.0   3932  2856 ?        Ss   06:40   0:00      |   \_ bash ./run.user certsync
root     3304251  0.0  0.0   2492   876 ?        S    15:00   0:00      |       \_ sleep 9999
root     1497491  0.0  0.0    220    76 ?        S    06:40   0:00      \_ s6-supervise go2rtc-healthcheck
root     1497592  0.0  0.0   3932  2944 ?        Ss   06:40   0:01      |   \_ bash ./run.user go2rtc-healthcheck
root     3547567  0.0  0.0   2492   892 ?        S    16:06   0:00      |       \_ sleep 30s
root     1497492  0.0  0.0    220    80 ?        S    06:40   0:00      \_ s6-supervise go2rtc-log
nobody   1497558  0.0  0.0    292     4 ?        Ss   06:40   0:00      |   \_ s6-log -b -- T 1 n0 s10000000 T /dev/shm/logs/go2rtc
root     1497493  0.0  0.0    220    80 ?        S    06:40   0:00      \_ s6-supervise s6rc-fdholder
root     1497503  0.0  0.0    304   172 ?        Ss   06:40   0:00          \_ /package/admin/s6-2.13.2.0/command/s6-fdholderd -1 -n 256 -i data/rules

NickM-27 Dec 31, 2025
Collaborator Sponsor

So it appears to be the forkserver process itself. This is generally expected change for 0.17 as we changed the start method. It is unclear why yours specifically is having issues, unless some process is continually being stopped / restarting which keeps the fork process active

ivitjuk · 2025-12-31T15:34:04Z

ivitjuk
Dec 31, 2025
Author

Full strace activity between two clone3() syscalls, 3605966 is cloned, exchanges a few messages, and then dies.

[pid 1497903] clone3({flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID, child_tid=0x7f32c17fa990, parent_tid=0x7f32c17fa990, exit_signal=0, stack=0x7f32c0ffa000, stack_size=0x7fff80, tls=0x7f32c17fa6c0}strace: Process 3605966 attached
 <unfinished ...>
[pid 3605966] rseq(0x7f32c17fafe0, 0x20, 0, 0x53053053 <unfinished ...>
[pid 1497903] <... clone3 resumed> => {parent_tid=[1949816]}, 88) = 1949816
[pid 3605966] <... rseq resumed>)       = 0
[pid 1497903] rt_sigprocmask(SIG_SETMASK, [],  <unfinished ...>
[pid 3605966] set_robust_list(0x7f32c17fa9a0, 24 <unfinished ...>
[pid 1497903] <... rt_sigprocmask resumed>NULL, 8) = 0
[pid 3605966] <... set_robust_list resumed>) = 0
[pid 3605966] rt_sigprocmask(SIG_SETMASK, [],  <unfinished ...>
[pid 1497903] futex(0xa5b8f4, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 3605966] <... rt_sigprocmask resumed>NULL, 8) = 0
[pid 1500073] <... futex resumed>)      = 0
[pid 1497903] <... futex resumed>)      = 1
[pid 3605966] gettid( <unfinished ...>
[pid 1500073] futex(0xa5b8f8, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...>
[pid 3605966] <... gettid resumed>)     = 1949816
[pid 1497903] futex(0xa5b8f8, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 3605966] mmap(NULL, 16384, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0 <unfinished ...>
[pid 1500073] <... futex resumed>)      = -1 EAGAIN (Resource temporarily unavailable)
[pid 1497903] <... futex resumed>)      = 0
[pid 3605966] <... mmap resumed>)       = 0x7f32f038f000
[pid 1497903] futex(0x7f32ec0096f0, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, FUTEX_BITSET_MATCH_ANY <unfinished ...>
[pid 1500073] futex(0xa5b8f8, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 3605966] gettid()                  = 1949816
[pid 1500073] futex(0xa5b8f0, FUTEX_WAIT_BITSET_PRIVATE, 0, {tv_sec=3478879, tv_nsec=883338995}, FUTEX_BITSET_MATCH_ANY <unfinished ...>
[pid 3605966] futex(0x7f32ec0096f0, FUTEX_WAKE_PRIVATE, 1) = 1
[pid 1497903] <... futex resumed>)      = 0
[pid 1497903] futex(0xa5b8f0, FUTEX_WAIT_BITSET_PRIVATE, 0, {tv_sec=3478879, tv_nsec=883571387}, FUTEX_BITSET_MATCH_ANY <unfinished ...>
[pid 3605966] futex(0xa5b8f0, FUTEX_WAKE_PRIVATE, 1) = 1
[pid 1500073] <... futex resumed>)      = 0
[pid 1497903] <... futex resumed>)      = -1 EAGAIN (Resource temporarily unavailable)
[pid 3605966] getrandom( <unfinished ...>
[pid 1500073] futex(0xa5b8f8, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 3605966] <... getrandom resumed>"\xeb\xee\xde\xab\x81\xcc\xf6\x83\xe4\x36\xc3\x5a\x62\xd3\xec\xd2\x41\x0d\x6f\xf1", 20, 0) = 20
[pid 1497903] futex(0xa5b8f0, FUTEX_WAIT_BITSET_PRIVATE, 0, {tv_sec=3478879, tv_nsec=883571387}, FUTEX_BITSET_MATCH_ANY <unfinished ...>
[pid 3605966] futex(0xa5b8f4, FUTEX_WAIT_BITSET_PRIVATE, 0, {tv_sec=3478879, tv_nsec=883824057}, FUTEX_BITSET_MATCH_ANY <unfinished ...>
[pid 1500073] <... futex resumed>)      = 0
[pid 1500073] futex(0xa5b8f0, FUTEX_WAKE_PRIVATE, 1) = 1
[pid 1497903] <... futex resumed>)      = 0
[pid 1500073] write(24, "\0\0\0!\200\4\225\26\0\0\0\0\0\0\0\214\7#RETURN\224G@\24\0\0\0\0"..., 37 <unfinished ...>
[pid 1497903] futex(0xa5b8f8, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 1500073] <... write resumed>)      = 37
[pid 1497903] <... futex resumed>)      = 0
[pid 1500073] futex(0xa5b8f4, FUTEX_WAIT_BITSET_PRIVATE, 0, {tv_sec=3478879, tv_nsec=884400893}, FUTEX_BITSET_MATCH_ANY) = -1 EAGAIN (Resource temporarily unavailable)
[pid 1497903] futex(0xa5b8f4, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 3605966] <... futex resumed>)      = 0
[pid 1500073] futex(0xa5b8f8, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...>
[pid 1497903] <... futex resumed>)      = 1
[pid 3605966] futex(0xa5b8f4, FUTEX_WAIT_BITSET_PRIVATE, 0, {tv_sec=3478879, tv_nsec=883824057}, FUTEX_BITSET_MATCH_ANY <unfinished ...>
[pid 1500073] <... futex resumed>)      = -1 EAGAIN (Resource temporarily unavailable)
[pid 1497903] futex(0xa5b8f8, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 1500073] futex(0xa5b8f8, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 1497903] <... futex resumed>)      = 0
[pid 1500073] <... futex resumed>)      = 0
[pid 1497903] accept4(6,  <unfinished ...>
[pid 1500073] futex(0xa5b8f4, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 3605966] <... futex resumed>)      = 0
[pid 1500073] <... futex resumed>)      = 1
[pid 3605966] futex(0xa5b8f8, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...>
[pid 1500073] futex(0xa5b8f8, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 3605966] <... futex resumed>)      = -1 EAGAIN (Resource temporarily unavailable)
[pid 1500073] <... futex resumed>)      = 0
[pid 3605966] futex(0xa5b8f8, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 1500073] read(24,  <unfinished ...>
[pid 3605966] <... futex resumed>)      = 0
[pid 1500073] <... read resumed>"\0\0\0'", 4) = 4
[pid 1500073] futex(0xa5b8f0, FUTEX_WAIT_BITSET_PRIVATE, 0, {tv_sec=3478879, tv_nsec=885329450}, FUTEX_BITSET_MATCH_ANY <unfinished ...>
[pid 3605966] futex(0xa5b8f0, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 1500073] <... futex resumed>)      = -1 EAGAIN (Resource temporarily unavailable)
[pid 3605966] <... futex resumed>)      = 0
[pid 1500073] futex(0xa5b8f8, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...>
[pid 3605966] futex(0xa5b8f8, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 1500073] <... futex resumed>)      = -1 EAGAIN (Resource temporarily unavailable)
[pid 3605966] <... futex resumed>)      = 0
[pid 1500073] futex(0xa5b8f8, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 3605966] write(21, "\0\0\0\37#CHALLENGE#\353\356\336\253\201\314\366\203\3446\303Zb\323\354\322A"..., 35 <unfinished ...>
[pid 1500073] <... futex resumed>)      = 0
[pid 3605966] <... write resumed>)      = 35
[pid 3605966] futex(0xa5b8f4, FUTEX_WAIT_BITSET_PRIVATE, 0, {tv_sec=3478879, tv_nsec=885603872}, FUTEX_BITSET_MATCH_ANY <unfinished ...>
[pid 1500073] futex(0xa5b8f4, FUTEX_WAKE_PRIVATE, 1) = 1
[pid 3605966] <... futex resumed>)      = 0
[pid 1500073] read(24,  <unfinished ...>
[pid 3605966] futex(0xa5b8f8, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 1500073] <... read resumed>"\200\4\225\34\0\0\0\0\0\0\0(\214\f7f32f4733410\224\214\3get"..., 39) = 39
[pid 3605966] <... futex resumed>)      = 0
[pid 1500073] futex(0xa5b8f0, FUTEX_WAIT_BITSET_PRIVATE, 0, {tv_sec=3478879, tv_nsec=885922841}, FUTEX_BITSET_MATCH_ANY <unfinished ...>
[pid 3605966] futex(0xa5b8f0, FUTEX_WAKE_PRIVATE, 1) = 1
[pid 1500073] <... futex resumed>)      = 0
[pid 3605966] read(21, "\0\0\0\20", 4)  = 4
[pid 1500073] futex(0xa5b8f8, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 3605966] futex(0xa5b8f4, FUTEX_WAIT_BITSET_PRIVATE, 0, {tv_sec=3478879, tv_nsec=886148942}, FUTEX_BITSET_MATCH_ANY <unfinished ...>
[pid 1500073] <... futex resumed>)      = 0
[pid 1500073] futex(0xa5b8f4, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 3605966] <... futex resumed>)      = 0
[pid 1500073] <... futex resumed>)      = 1
[pid 3605966] futex(0xa5b8f8, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...>
[pid 1500073] futex(0xa5b8f8, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 3605966] <... futex resumed>)      = -1 EAGAIN (Resource temporarily unavailable)
[pid 1500073] <... futex resumed>)      = 0
[pid 3605966] futex(0xa5b8f8, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 1500073] write(24, "\0\0\0!\200\4\225\26\0\0\0\0\0\0\0\214\7#RETURN\224G\0\0\0\0\0\0"..., 37 <unfinished ...>
[pid 3605966] <... futex resumed>)      = 0
[pid 3605966] read(21,  <unfinished ...>
[pid 1500073] <... write resumed>)      = 37
[pid 3605966] <... read resumed>"\363\317\232\225e:\335\316,q\177\2743R\251\364", 16) = 16
[pid 3605966] futex(0xa5b8f0, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 1500073] futex(0xa5b8f0, FUTEX_WAIT_BITSET_PRIVATE, 0, {tv_sec=3478879, tv_nsec=886782458}, FUTEX_BITSET_MATCH_ANY <unfinished ...>
[pid 3605966] <... futex resumed>)      = 0
[pid 1500073] <... futex resumed>)      = -1 EAGAIN (Resource temporarily unavailable)
[pid 3605966] write(21, "\0\0\0\t#WELCOME#", 13 <unfinished ...>
[pid 1500073] futex(0xa5b8f8, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 3605966] <... write resumed>)      = 13
[pid 1500073] <... futex resumed>)      = 0
[pid 3605966] futex(0xa5b8f4, FUTEX_WAIT_BITSET_PRIVATE, 0, {tv_sec=3478879, tv_nsec=887115482}, FUTEX_BITSET_MATCH_ANY <unfinished ...>
[pid 1500073] futex(0xa5b8f4, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 3605966] <... futex resumed>)      = 0
[pid 1500073] <... futex resumed>)      = 1
[pid 3605966] futex(0xa5b8f8, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...>
[pid 1500073] futex(0xa5b8f8, FUTEX_WAKE_PRIVATE, 1) = 1
[pid 3605966] <... futex resumed>)      = 0
[pid 3605966] futex(0xa5b8f8, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 3605966] read(21,  <unfinished ...>
[pid 1500073] read(24,  <unfinished ...>
[pid 3605966] <... read resumed>"\0\0\0\37", 4) = 4
[pid 1500073] <... read resumed>"\0\0\0'", 4) = 4
[pid 3605966] read(21, "#CHALLENGE#\263\4\327'lH\"I\4k\200(\307\300\255\334uroL", 31) = 31
[pid 3605966] futex(0xa5b8f0, FUTEX_WAIT_BITSET_PRIVATE, 0, {tv_sec=3478879, tv_nsec=887773574}, FUTEX_BITSET_MATCH_ANY <unfinished ...>
[pid 1500073] futex(0xa5b8f0, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 3605966] <... futex resumed>)      = -1 EAGAIN (Resource temporarily unavailable)
[pid 1500073] <... futex resumed>)      = 0
[pid 3605966] futex(0xa5b8f8, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...>
[pid 1500073] futex(0xa5b8f8, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 3605966] <... futex resumed>)      = 0
[pid 1500073] <... futex resumed>)      = 1
[pid 3605966] futex(0xa5b8f8, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 1500073] read(24, "\200\4\225\34\0\0\0\0\0\0\0(\214\f7f32f4731fd0\224\214\3get"..., 39) = 39
[pid 3605966] futex(0xa5b8f4, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 1500073] futex(0xa5b8f4, FUTEX_WAIT_BITSET_PRIVATE, 0, {tv_sec=3478879, tv_nsec=888124249}, FUTEX_BITSET_MATCH_ANY <unfinished ...>
[pid 3605966] <... futex resumed>)      = 0
[pid 3605966] write(21, "\0\0\0\20\210\16#-\2628/\233\245\244/\347G\2\365\271", 20 <unfinished ...>
[pid 1500073] <... futex resumed>)      = -1 EAGAIN (Resource temporarily unavailable)
[pid 3605966] <... write resumed>)      = 20
[pid 1500073] futex(0xa5b8f8, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 3605966] futex(0xa5b8f0, FUTEX_WAIT_BITSET_PRIVATE, 0, {tv_sec=3478879, tv_nsec=888427375}, FUTEX_BITSET_MATCH_ANY <unfinished ...>
[pid 1500073] <... futex resumed>)      = 0
[pid 1500073] futex(0xa5b8f0, FUTEX_WAKE_PRIVATE, 1) = 1
[pid 3605966] <... futex resumed>)      = 0
[pid 3605966] futex(0xa5b8f8, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...>
[pid 1500073] futex(0xa5b8f8, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 3605966] <... futex resumed>)      = 0
[pid 1500073] <... futex resumed>)      = 1
[pid 3605966] futex(0xa5b8f8, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 3605966] read(21, "\0\0\0\t", 4)   = 4
[pid 1500073] write(24, "\0\0\0!\200\4\225\26\0\0\0\0\0\0\0\214\7#RETURN\224G\0\0\0\0\0\0"..., 37 <unfinished ...>
[pid 3605966] read(21, "#WELCOME#", 9)  = 9
[pid 3605966] read(21, "\0\0\0,", 4)    = 4
[pid 1500073] <... write resumed>)      = 37
[pid 3605966] read(21, "\200\4\225!\0\0\0\0\0\0\0(N\214\6decref\224\214\f7f32f471"..., 44) = 44
[pid 1500073] read(24, "\0\0\0'", 4)    = 4
[pid 1500073] futex(0xa5b8f4, FUTEX_WAIT_BITSET_PRIVATE, 0, {tv_sec=3478879, tv_nsec=889591751}, FUTEX_BITSET_MATCH_ANY <unfinished ...>
[pid 3605966] futex(0xa5b8f4, FUTEX_WAKE_PRIVATE, 1) = 1
[pid 1500073] <... futex resumed>)      = 0
[pid 3605966] futex(0xa5b8f8, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 1500073] futex(0xa5b8f8, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...>
[pid 3605966] <... futex resumed>)      = 0
[pid 1500073] <... futex resumed>)      = -1 EAGAIN (Resource temporarily unavailable)
[pid 3605966] write(21, "\0\0\0\31\200\4\225\16\0\0\0\0\0\0\0\214\7#RETURN\224N\206\224.", 29 <unfinished ...>
[pid 1500073] futex(0xa5b8f8, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 3605966] <... write resumed>)      = 29
[pid 1500073] <... futex resumed>)      = 0
[pid 3605966] futex(0xa5b8f0, FUTEX_WAIT_BITSET_PRIVATE, 0, {tv_sec=3478879, tv_nsec=890355644}, FUTEX_BITSET_MATCH_ANY <unfinished ...>
[pid 1500073] futex(0xa5b8f0, FUTEX_WAKE_PRIVATE, 1) = 1
[pid 3605966] <... futex resumed>)      = 0
[pid 3605966] futex(0xa5b8f8, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 1500073] read(24,  <unfinished ...>
[pid 3605966] <... futex resumed>)      = 0
[pid 3605966] close(21 <unfinished ...>
[pid 1500073] <... read resumed>"\200\4\225\34\0\0\0\0\0\0\0(\214\f7f32f2be3890\224\214\3get"..., 39) = 39
[pid 3605966] <... close resumed>)      = 0
[pid 1500073] write(24, "\0\0\0\32\200\4\225\17\0\0\0\0\0\0\0\214\7#RETURN\224K\0\206\224.", 30 <unfinished ...>
[pid 3605966] munmap(0x7f32f038f000, 16384 <unfinished ...>
[pid 1500073] <... write resumed>)      = 30
[pid 3605966] <... munmap resumed>)     = 0
[pid 3605966] rt_sigprocmask(SIG_BLOCK, ~[RT_1], NULL, 8) = 0
[pid 3605966] madvise(0x7f32c0ffa000, 8368128, MADV_DONTNEED) = 0
[pid 3605966] exit(0)                   = ?
[pid 3605966] +++ exited with 0 +++
[pid 1500073] read(24, "\0\0\0'", 4)    = 4
[pid 1500073] read(24, "\200\4\225\34\0\0\0\0\0\0\0(\214\f7f32f4730750\224\214\3get"..., 39) = 39
[pid 1500073] write(24, "\0\0\0\32\200\4\225\17\0\0\0\0\0\0\0\214\7#RETURN\224K\0\206\224.", 30) = 30
[pid 1500073] read(24, "\0\0\0'", 4)    = 4
[pid 1500073] read(24, "\200\4\225\34\0\0\0\0\0\0\0(\214\f7f32f472b110\224\214\3get"..., 39) = 39
[pid 1500073] write(24, "\0\0\0!\200\4\225\26\0\0\0\0\0\0\0\214\7#RETURN\224G@\24ffff"..., 37) = 37
[pid 1500073] read(24, "\0\0\0'", 4)    = 4
[pid 1500073] read(24, "\200\4\225\34\0\0\0\0\0\0\0(\214\f7f32f4740990\224\214\3get"..., 39) = 39
[pid 1500073] write(24, "\0\0\0!\200\4\225\26\0\0\0\0\0\0\0\214\7#RETURN\224G@\24\0\0\0\0"..., 37) = 37
[pid 1500073] read(24, "\0\0\0'", 4)    = 4
[pid 1500073] read(24, "\200\4\225\34\0\0\0\0\0\0\0(\214\f7f32f4728990\224\214\3get"..., 39) = 39
[pid 1500073] write(24, "\0\0\0!\200\4\225\26\0\0\0\0\0\0\0\214\7#RETURN\224G\0\0\0\0\0\0"..., 37) = 37
[pid 1500073] read(24, "\0\0\0'", 4)    = 4
[pid 1500073] read(24, "\200\4\225\34\0\0\0\0\0\0\0(\214\f7f32f2be2290\224\214\3get"..., 39) = 39
[pid 1500073] write(24, "\0\0\0!\200\4\225\26\0\0\0\0\0\0\0\214\7#RETURN\224G\0\0\0\0\0\0"..., 37) = 37
[pid 1500073] read(24, "\0\0\0'", 4)    = 4
[pid 1500073] read(24, "\200\4\225\34\0\0\0\0\0\0\0(\214\f7f32f47518d0\224\214\3get"..., 39) = 39
[pid 1500073] write(24, "\0\0\0\33\200\4\225\20\0\0\0\0\0\0\0\214\7#RETURN\224M\314\3\206\224.", 31) = 31
[pid 1500073] read(24, "\0\0\0'", 4)    = 4
[pid 1500073] read(24, "\200\4\225\34\0\0\0\0\0\0\0(\214\f7f32f47518d0\224\214\3get"..., 39) = 39
[pid 1500073] write(24, "\0\0\0\33\200\4\225\20\0\0\0\0\0\0\0\214\7#RETURN\224M\314\3\206\224.", 31) = 31
[pid 1500073] read(24, "\0\0\0'", 4)    = 4
[pid 1500073] read(24, "\200\4\225\34\0\0\0\0\0\0\0(\214\f7f32f4726710\224\214\3get"..., 39) = 39
[pid 1500073] write(24, "\0\0\0\33\200\4\225\20\0\0\0\0\0\0\0\214\7#RETURN\224M6\4\206\224.", 31) = 31
[pid 1500073] read(24, "\0\0\0'", 4)    = 4
[pid 1500073] read(24, "\200\4\225\34\0\0\0\0\0\0\0(\214\f7f32f4751850\224\214\3get"..., 39) = 39
[pid 1500073] write(24, "\0\0\0\33\200\4\225\20\0\0\0\0\0\0\0\214\7#RETURN\224M\n\4\206\224.", 31) = 31
[pid 1500073] read(24, "\0\0\0'", 4)    = 4
[pid 1500073] read(24, "\200\4\225\34\0\0\0\0\0\0\0(\214\f7f32f4751850\224\214\3get"..., 39) = 39
[pid 1500073] write(24, "\0\0\0\33\200\4\225\20\0\0\0\0\0\0\0\214\7#RETURN\224M\n\4\206\224.", 31) = 31
[pid 1500073] read(24, "\0\0\0'", 4)    = 4
[pid 1500073] read(24, "\200\4\225\34\0\0\0\0\0\0\0(\214\f7f32f472b110\224\214\3get"..., 39) = 39
[pid 1500073] write(24, "\0\0\0!\200\4\225\26\0\0\0\0\0\0\0\214\7#RETURN\224G@\24ffff"..., 37) = 37
[pid 1500073] read(24, "\0\0\0'", 4)    = 4
[pid 1500073] read(24, "\200\4\225\34\0\0\0\0\0\0\0(\214\f7f32f4740990\224\214\3get"..., 39) = 39
[pid 1500073] write(24, "\0\0\0!\200\4\225\26\0\0\0\0\0\0\0\214\7#RETURN\224G@\24\0\0\0\0"..., 37) = 37
[pid 1500073] read(24, "\0\0\0'", 4)    = 4
[pid 1500073] read(24, "\200\4\225\34\0\0\0\0\0\0\0(\214\f7f32f4728990\224\214\3get"..., 39) = 39
[pid 1500073] write(24, "\0\0\0!\200\4\225\26\0\0\0\0\0\0\0\214\7#RETURN\224G\0\0\0\0\0\0"..., 37) = 37
[pid 1500073] read(24, "\0\0\0'", 4)    = 4
[pid 1500073] read(24, "\200\4\225\34\0\0\0\0\0\0\0(\214\f7f32f2be2290\224\214\3get"..., 39) = 39
[pid 1500073] write(24, "\0\0\0!\200\4\225\26\0\0\0\0\0\0\0\214\7#RETURN\224G\0\0\0\0\0\0"..., 37) = 37
[pid 1500073] read(24, "\0\0\0'", 4)    = 4
[pid 1500073] read(24, "\200\4\225\34\0\0\0\0\0\0\0(\214\f7f32f4742250\224\214\3get"..., 39) = 39
[pid 1500073] write(24, "\0\0\0\32\200\4\225\17\0\0\0\0\0\0\0\214\7#RETURN\224K\0\206\224.", 30) = 30
[pid 1500073] read(24, "\0\0\0'", 4)    = 4
[pid 1500073] read(24, "\200\4\225\34\0\0\0\0\0\0\0(\214\f7f32f4750ed0\224\214\3get"..., 39) = 39
[pid 1500073] write(24, "\0\0\0\32\200\4\225\17\0\0\0\0\0\0\0\214\7#RETURN\224K\0\206\224.", 30) = 30
[pid 1497903] <... accept4 resumed>{sa_family=AF_UNIX}, [110->2], SOCK_CLOEXEC) = 23
[pid 1500073] read(24,  <unfinished ...>
[pid 1497903] getsockname(23, {sa_family=AF_UNIX, sun_path="/tmp/pymp-c64cbkl5/listener-iwbbszkv"}, [128->39]) = 0
[pid 1497903] ioctl(23, FIONBIO, [0])   = 0
[pid 1497903] gettid()                  = 401
[pid 1497903] rt_sigprocmask(SIG_BLOCK, ~[], [], 8) = 0

6 replies

ivitjuk Dec 31, 2025
Author

Here are five successive 10s measurements for the clone3() syscall, each of them counted ~500 processes being cloned, or ~50/s. Hopefully this counts as an evidence of excessive child creation:

❯ sudo timeout  10 strace -f -p 1497902 -e clone3|& grep clone3|wc           
    494    5927  153230                                                      
                                                                            
❯ sudo timeout  10 strace -f -p 1497902 -e clone3|& grep clone3|wc           
    512    6146  160647                                                      
                                                                             
❯ sudo timeout  10 strace -f -p 1497902 -e clone3|& grep clone3|wc           
    592    7092  184046                                                      
                                                                             
rigate dev 10s                                              
❯ sudo timeout  10 strace -f -p 1497902 -e clone3|& grep clone3|wc           
    561    6729  175662                                                      
                                                                             
❯ sudo timeout  10 strace -f -p 1497902 -e clone3|& grep clone3|wc           
    505    6055  158618

Process 1497902 is constantly at 9.8% CPU utilization:

❯ ps u 1497902
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root     1497902  9.8  1.7 2400404 137848 ?      Sl   06:40  59:59 /usr/bin/python3 -Wignore:::numpy.core.getlimits -c from multiprocessi

hawkeye217 Dec 31, 2025
Collaborator

This does look odd, but my own installation doesn't look like this, and I can't readily reproduce it.

You could attach strace to capture the caller, not just clone count.

ivitjuk Dec 31, 2025
Author

I captured the full strace between two clone3() syscalls in #21490 (comment)

You can see there that a child is created, it exchanges a few messages and than dies.

Does that help, or do you need something else?

hawkeye217 Dec 31, 2025
Collaborator

So the forkserver is just doing exactly what it’s being asked to do.

Next step to pinpoint it would be to log who is requesting the worker creation (eg, where the multiprocessing Process or Pool is invoked), or temporarily add logging around worker startup to confirm the call site. You'll have to do this inside the Frigate container, of course.

ivitjuk Dec 31, 2025
Author

Ok, thanks. I'll pause for now. Will update the issue once I collect new data.

ivitjuk · 2026-01-06T07:00:01Z

ivitjuk
Jan 6, 2026
Author

This seems to be related to the audio detection, which was enabled in my setup. After disabling it completely clone3() syscalls dropped by almost 10x to ~7/s, and CPU utilization dropped ~3x to ~3% for the forkserver process. I didn't look into what exactly causes audio decoder to induce this behavior.

❯ ps aux  | grep forkserver
root      999510  0.1  2.7 819048 218948 ?       S    06:44   0:04 /usr/bin/python3 -Wignore:::numpy.core.getlimits -c from multiprocessing.forkserver import main; main(5, 7, ['sqlite3', 'numpy', 'cv2', 'peewee', 'zmq', 'ruamel.yaml', 'frigate.camera.maintainer'], **{'sys_path': ['/opt/frigate', '/usr/lib/python311.zip', '/usr/lib/python3.11', '/usr/lib/python3.11/lib-dynload', '/usr/local/lib/python3.11/dist-packages', '/usr/lib/python3/dist-packages']})
root      999517  3.7  1.7 2113680 139012 ?      Sl   06:44   2:40 /usr/bin/python3 -Wignore:::numpy.core.getlimits -c from multiprocessing.forkserver import main; main(5, 7, ['sqlite3', 'numpy', 'cv2', 'peewee', 'zmq', 'ruamel.yaml', 'frigate.camera.maintainer'], **{'sys_path': ['/opt/frigate', '/usr/lib/python311.zip', '/usr/lib/python3.11', '/usr/lib/python3.11/lib-dynload', '/usr/local/lib/python3.11/dist-packages', '/usr/lib/python3/dist-packages']})
virtus   1047552  0.0  0.0   6372   644 pts/0    S+   07:55   0:00 grep --color forkserver

❯ sudo timeout  10 strace -f -p 999517 -e clone3|& grep clone3|wc                                                                                                                                                                                                        
[sudo] password for virtus: 
     73     876   22828

6 replies

hawkeye217 Jan 6, 2026
Collaborator

A simple patch you can apply in __main__.py before the main() call:

   try:
        import logging
        import os
        import traceback

        if os.getenv("FRIGATE_DEBUG_MP_START") == "1":
            _mp_logger = logging.getLogger("frigate.mpdebug")
            _orig_mp_start = mp.Process.start

            def _debug_mp_start(self, *args, **kwargs):
                try:
                    _mp_logger.info(
                        f"multiprocessing.Process.start() called for process name={getattr(self, 'name', None)}"
                    )
                    stack = "".join(traceback.format_stack(limit=20)[:-1])
                    for line in stack.splitlines():
                        _mp_logger.debug(line)
                except Exception:
                    _mp_logger.exception("Failed to log mp start stack")
                return _orig_mp_start(self, *args, **kwargs)

            mp.Process.start = _debug_mp_start
    except Exception:
        pass

This uses an env var, FRIGATE_DEBUG_MP_START=1 to add the wrapper and create the additional logging.

ivitjuk Jan 11, 2026
Author

Unfortunately I wasn't able to capture anything.

I added code segment per your instructions, with removed condition and log level set to error, just to make sure I get the logs. Then I enabled audio processing and confirmed that clone3() is in the "high frequency" mode.

What I saw is just a few processes being started during Frigate restart, and nothing else afterwards.

ivitjuk Jan 11, 2026
Author

List of processes being created at the startup:

frigate              | 2026-01-11 11:38:50.856319892  [2026-01-11 11:38:50] frigate.mpdebug                ERROR   : multiprocessing.Process.start() called for process name=frigate.recording_manager
frigate              | 2026-01-11 11:38:50.931699155  [2026-01-11 11:38:50] frigate.mpdebug                ERROR   : multiprocessing.Process.start() called for process name=frigate.review_segment_manager
frigate              | 2026-01-11 11:38:51.024197155  [2026-01-11 11:38:51] frigate.mpdebug                ERROR   : multiprocessing.Process.start() called for process name=frigate.embeddings_manager
frigate              | 2026-01-11 11:38:51.867658535  [2026-01-11 11:38:51] frigate.mpdebug                ERROR   : multiprocessing.Process.start() called for process name=frigate.detector:ov_1
frigate              | 2026-01-11 11:38:51.935088867  [2026-01-11 11:38:51] frigate.mpdebug                ERROR   : multiprocessing.Process.start() called for process name=frigate.detector:ov_2
frigate              | 2026-01-11 11:38:52.029682110  [2026-01-11 11:38:52] frigate.mpdebug                ERROR   : multiprocessing.Process.start() called for process name=frigate.output
frigate              | 2026-01-11 11:38:52.160646332  [2026-01-11 11:38:52] frigate.mpdebug                ERROR   : multiprocessing.Process.start() called for process name=frigate.audio_manager
frigate              | 2026-01-11 11:38:52.384186272  [2026-01-11 11:38:52] frigate.mpdebug                ERROR   : multiprocessing.Process.start() called for process name=frigate.process:street
frigate              | 2026-01-11 11:38:52.597748481  [2026-01-11 11:38:52] frigate.mpdebug                ERROR   : multiprocessing.Process.start() called for process name=frigate.capture:street
frigate              | 2026-01-11 11:38:52.881073514  [2026-01-11 11:38:52] frigate.mpdebug                ERROR   : multiprocessing.Process.start() called for process name=frigate.process:entrance
frigate              | 2026-01-11 11:38:53.526075136  [2026-01-11 11:38:53] frigate.mpdebug                ERROR   : multiprocessing.Process.start() called for process name=frigate.capture:entrance
frigate              | 2026-01-11 11:38:53.822695580  [2026-01-11 11:38:53] frigate.mpdebug                ERROR   : multiprocessing.Process.start() called for process name=frigate.process:terace
frigate              | 2026-01-11 11:38:54.007550666  [2026-01-11 11:38:54] frigate.mpdebug                ERROR   : multiprocessing.Process.start() called for process name=frigate.capture:terace

hawkeye217 Jan 11, 2026
Collaborator

Thanks for testing that, this is actually very helpful.

The fact that wrapping multiprocessing.Process.start() shows no process creation after startup definitively rules out Frigate restarting workers or spawning processes in a loop. So this isn’t coming from our process manager or the audio detector directly.

At this point the only explanation I can think of is that the forkserver activity is coming from Python multiprocessing internals, not explicit Process usage in Frigate. Audio detection significantly increases IPC traffic by reporting dbFS and RMS values, which in your case seems to be causing short-lived helper processes to be created. That would explain:

why audio amplifies the behavior
why disabling audio reduces it ~10x
why Process.start() is never called
why we can’t reproduce this elsewhere

This appears to be environment-specific multiprocessing behavior rather than a Frigate bug. We don’t currently have a reproducible case or a clear change we could make without redesigning how metrics and IPC are handled.

If you discover a minimal Python repro outside of Frigate, that would be the next actionable step. Otherwise, this is best treated as a platform-specific performance characteristic rather than something we can fix in Frigate.

ivitjuk Jan 11, 2026
Author

Thanks @hawkeye217 , that makes sense. Time permitting, I will continue investigating this and will update you if I get any meaningful insights. In the meantime, I think we can close this discussion.

Uh oh!

[Beta Support]: Potential performance regression in 0.17.0-beta1 #21490

Uh oh!

Uh oh!

ivitjuk Dec 31, 2025

Describe the problem you are having

15min system load

Offending process

Excessive cloning seems to be the culprit

Beta Version

Issue Category

Frigate config file

Relevant Frigate log output

Relevant go2rtc log output (if applicable)

Install method

docker-compose file or Docker CLI command

Operating system

CPU / GPU / Hardware

Screenshots

Steps to reproduce

Any other information that may be helpful

Replies: 3 comments · 14 replies

Uh oh!

NickM-27 Dec 31, 2025 Collaborator Sponsor

Uh oh!

ivitjuk Dec 31, 2025 Author

Uh oh!

NickM-27 Dec 31, 2025 Collaborator Sponsor

Uh oh!

Uh oh!

ivitjuk Dec 31, 2025 Author

Uh oh!

ivitjuk Dec 31, 2025 Author

Uh oh!

hawkeye217 Dec 31, 2025 Collaborator

Uh oh!

ivitjuk Dec 31, 2025 Author

Uh oh!

hawkeye217 Dec 31, 2025 Collaborator

Uh oh!

ivitjuk Dec 31, 2025 Author

Uh oh!

ivitjuk Jan 6, 2026 Author

Uh oh!

hawkeye217 Jan 6, 2026 Collaborator

Uh oh!

ivitjuk Jan 11, 2026 Author

Uh oh!

ivitjuk Jan 11, 2026 Author

Uh oh!

hawkeye217 Jan 11, 2026 Collaborator

Uh oh!

ivitjuk Jan 11, 2026 Author

ivitjuk
Dec 31, 2025

Replies: 3 comments 14 replies

NickM-27
Dec 31, 2025
Collaborator Sponsor

ivitjuk Dec 31, 2025
Author

NickM-27 Dec 31, 2025
Collaborator Sponsor

ivitjuk
Dec 31, 2025
Author

ivitjuk Dec 31, 2025
Author

hawkeye217 Dec 31, 2025
Collaborator

ivitjuk Dec 31, 2025
Author

hawkeye217 Dec 31, 2025
Collaborator

ivitjuk Dec 31, 2025
Author

ivitjuk
Jan 6, 2026
Author

hawkeye217 Jan 6, 2026
Collaborator

ivitjuk Jan 11, 2026
Author

ivitjuk Jan 11, 2026
Author

hawkeye217 Jan 11, 2026
Collaborator

ivitjuk Jan 11, 2026
Author