Broken queue causes various problems, need to report in self-check

[joshrnoll]

Description

Sometime while running 1.23.4, the push mirrors to my GitHub repositories stopped synchronizing.

The last known good sync was on Feb 28, 2025, 8:29:30 AM (this was on a mirror configured to sync every 8hrs).

I am using a GitHub personal access token (fine-grained) with Read access to metadata and Read/Write access to contents. The GitHub repo is a public repo. I've attempted to remove the mirror config from the repository and reconfigure it. When configuring, I am using my GitHub username in the username field and my PAT in the password field. I am also selecting the 'Sync when commits are pushed' checkbox.

I've even tried deleting the remote GitHub repo completely and re-creating it. The sync still won't seem to go through. When clicking on 'synchronize now', a banner displays showing me that it is trying to sync (see screenshot), but even after waiting more than 24 hours, the repo never syncs.

I attempted updating to 1.23.5 but still get the same results

I see the following logs in the Docker container:

$ docker logs gitea | grep sync
2025/03/14 05:59:41 ...ices/mirror/queue.go:66:func1() [E] Unable to push sync request for to the queue for pull mirror repo[10]. Error: already in queue
2025/03/14 06:09:00 ...ices/mirror/queue.go:66:func1() [E] Unable to push sync request for to the queue for pull mirror repo[10]. Error: already in queue

After noting these logs, I attempted flushing the queues with the gitea manager command, however this produces a timeout error:

$ docker exec -it gitea sh
/ # su git
gitea:/$ gitea manager flush-queues
context deadline exceeded
internal API error response, status=408

I get the same timeout error when adding the --timeout flag (I've gone up to 10m).

Gitea Version

1.23.5

Can you reproduce the bug on the Gitea demo site?

No

Log Gist

No response

Screenshots

Git Version

1.23.5

Operating System

Ubuntu 24.04.1 LTS

How are you running Gitea?

Docker container (non-rootless), deployed with custom Ansible playbook. Gitea is behind a reverse proxy (Traefik) and configured for OIDC with Authentik. Playbook below for reference.

---
- name: Deploy {{ deploy_container_service_name }}
  hosts: "{{ deploy_container_host_group_name }}"
  vars:
    deploy_container_service_name: gitea # Service/container name
    deploy_container_host_group_name: gitea # Host group for container to be deployed to
    deploy_container_service_image: gitea/gitea # Container image -- ex. louislam/uptime-kuma
    deploy_container_service_tag: 1.23.5 # Image tag -- ex. latest
    deploy_container_service_port: "3000" # Must be quoted -- ex. "80" not 80
    deploy_container_service_scheme: http # http or https
    deploy_container_forward_auth: false # true or false
    deploy_container_tailnet_name: "{{ tailnet_name }}"
    deploy_container_oauth_client_secret: "{{ tailscale_containers_oauth_client['secret'] }}"
    deploy_container_domain_name: "{{ domain_name }}"
    deploy_container_cloudflare_email: "{{ cloudflare_email }}"
    deploy_container_cloudflare_api_token: "{{ cloudflare_api_token }}"
    deploy_container_volumes: # Customize or omit
      - /home/{{ ansible_user }}/gitea/data:/data
      - /etc/timezone:/etc/timezone:ro
      - /etc/localtime:/etc/localtime:ro
    deploy_container_env_vars: # Customize or omit
      USER_UID: "1000"
      USER_GID: "999"
      GITEA__DEFAULT__APP_NAME: "NollHomeLab"
      GITEA__mailer__ENABLED: "true"
      GITEA__mailer__PROTOCOL: smtp
      GITEA__mailer__SMTP_PORT: "25"
      GITA__mailer__SMTP_ADDR: ntfy.{{ tailnet_name }}
      GITEA__mailer__FROM: noreply@gitea.{{ tailnet_name }}
      GITEA__mailer__ENVELOPE_FROM: noreply@gitea.{{ tailnet_name }}
      GITEA__server__ROOT_URL: https://gitea.{{ domain_name }}
      GITEA__openid__ENABLE_OPENID_SIGNIN: "true"
      GITEA__openid__ENABLE_OPENID_SIGNUP: "true"
      GITEA__service__ALLOW_ONLY_EXTERNAL_REGISTRATION: "true"
      GITEA__service__DISABLE_REGISTRATION: "false"
      GITEA__mirror__ENABLED: "true"
      GITEA__log__LEVEL: "Debug"
    deploy_container_labels:
      - key: "traefik.http.middlewares.gitea-login-redirect.redirectregex.regex"
        value: "^https://gitea.{{ domain_name }}/user/login*"
      - key: "traefik.http.middlewares.gitea-login-redirect.redirectregex.replacement"
        value: "https://gitea.{{ domain_name }}/user/oauth2/Authentik"
      - key: "traefik.http.routers.gitea.middlewares"
        value: "gitea-login-redirect"      

  tasks:
    - name: Include vars
      ansible.builtin.include_vars:
        dir: "{{ root_playbook_dir }}/vars"

    - name: Deploy container
      ansible.builtin.include_role:
        name: "{{ root_playbook_dir }}/roles/deploy_container"
...

Database

SQLite

[wxiaoguang]

Maybe this could help your case: #33862 (comment)

[joshrnoll]

This fixed it!

For those who stumble on this in the future:

Go to Site Administration --> Monitoring --> Queues

Click on Review / Add Workers next to the mirror queue, then click Remove all (I had 9 in the queue before clearing)

After doing this I initially received a permission denied error, but this was because I had deleted and recreated my remote repo without recreating the PAT:

Once I recreated my PAT, I got a successful sync again.

@wxiaoguang thanks for the help! Is this a bug? Should I leave this issue open or go ahead and close?

[wxiaoguang]

It is a bug (explained here #33862 (comment))

But it's difficult to fix.

The best approach we could do at the moment is to add the "queue check" to "self check" page to remind users .....

So I think we can leave this open until we have a good "self check" prompt

[wxiaoguang]

By the way, the corruption only occurs in "levelqueue". So you could switch to "redis" queue which won't get stuck 😆

[TheFox0x7]

Would it be possible to replace levelqueue with something that's less prone to failing?

I've been eyeing nats but it's a bit harder to integrate it into queues (with persistence that is) than I originally thought, however it can be embedded into the binary which is why I find worth considering.

[wxiaoguang]

Would it be possible to replace levelqueue with something that's less prone to failing?

No idea at the moment. Gitea's queue has some special behaviors like "unique queue". IMO making levelqueue could self-heal is a quick approach.