Many « Something went wrong while fetching latest comments. » error messages (429 HTTP status code)

[stephane-klein]

Hi,

I run GitLab 14.10.2 on this server:

• Processor 1× AMD EPYC 7281 CPU - 16C/32T - 2.1 GHz
• 96 GB DDR4 ECC

I set this parameter in docker-compose.yml:

  postgres:
    image: index.docker.io/postgres:12.4-alpine
    command: >
      -c shared_buffers=2000MB
      -c maintenance_work_mem=150MB
      -c work_mem=50MB
      -c seq_page_cost=0.8
      -c random_page_cost=1.1
      -c effective_io_concurrency=200
    ...

  gitlab:
    ...
    environment:

      ...
      # 32*16
      DB_POOL: 512

      ...

      PUMA_WORKERS: 32
      PUMA_THREADS_MIN: 1
      # https://docs.gitlab.com/ee/install/requirements.html#puma-threads
      PUMA_THREADS_MAX: 4
      PUMA_PER_WORKER_MAX_MEMORY_MB: 4048
      PUMA_MASTER_MAX_MEMORY_MB: 20048

and I have many « Something went wrong while fetching latest comments. » error message.

More detail, example of failed requests:

• 429 status code on GET https://..../issues/3154/realtime_changes
• 429 status code on GET https://..../issues/3154/notes x 4

I don't know what direction to look in to fix this issue 🤔

Best regards,
Stéphane

[stephane-klein]

I see that these requests are managed by gitlab-workhorse.

[stephane-klein]

root@57a5e31db5a9:/home/git/gitlab# tail -f /var/log/gitlab/gitlab/production.log
Started GET "/foo/bar/-/issues/6401/realtime_changes" for xxx.xxx.xx.xxx at 2022-05-20 19:00:00 +0200
Completed 429 Too Many Requests in 0ms (Allocations: 5)

[stephane-klein]

I found the solution in this thread: https://gitlab.com/gitlab-org/gitlab/-/issues/11458

How to fix this issue:

• go to « Admin Area > Settings > Preferences > Pooling interval multiplier »
• remove Polling interval multiplier 0 value and replace it by 1 or 2.

Never set this setting to 0!