Git push fails with "Permission denied" on tmp_objdir in rootless Gitea on Kubernetes with old Longhorn CSI driver

[PatrickHuetter]

Description

Environment

• Gitea Version: 1.24.2-rootless
• Deployment Method: Official Helm Chart v12.1.1
• Kubernetes: K3s
• Storage: Longhorn volumes
• Container Runtime: Running as uid=1000(git) gid=1000(git)

Problem Description

When pushing repositories with more than ~30 files via HTTPS, git push consistently fails with:

remote: fatal: could not open '/data/git/gitea-repositories/myorg/myrepo.git/./objects/tmp_objdir-incoming-[random]/pack/tmp_pack_[random]' for reading: Permission denied
error: unpack failed: index-pack abnormal exit
To https://git.example.com/myorg/myrepo.git
 ! [remote rejected] main -> main (unpacker error)
error: failed to push some refs to 'https://git.example.com/myorg/myrepo.git'

Small commits (1-5 files) usually work without issues. The error occurs consistently with larger pushes.

What I've Tried

1. Security Context Configuration

Added comprehensive security contexts to ensure proper permissions:

podSecurityContext:
  fsGroup: 1000
  fsGroupChangePolicy: "Always"  # Also tried "OnRootMismatch"
  runAsUser: 1000
  runAsGroup: 1000
  runAsNonRoot: true

containerSecurityContext:
  allowPrivilegeEscalation: false
  capabilities:
    drop:
      - ALL
  privileged: false
  readOnlyRootFilesystem: false
  runAsGroup: 1000
  runAsNonRoot: true
  runAsUser: 1000

2. Init Container for Permission Fixes

Attempted to fix permissions via initPreScript:

initPreScript: |
  # Create required directories with proper permissions
  mkdir -p /data/gitea/tmp || true
  chmod 1777 /data/gitea/tmp || true
  mkdir -p /tmp/gitea || true
  chmod 1777 /tmp/gitea || true
  # Fix repository permissions if needed
  find /data/git/gitea-repositories -type d -name objects -exec chmod 775 {} \; 2>/dev/null || true

3. Resource Limits

Increased resources to rule out memory/CPU constraints:

resources:
  limits:
    cpu: 2000m
    memory: 4Gi
  requests:
    cpu: 500m
    memory: 1Gi

4. Repository Configuration

Tried setting custom temp path:

gitea:
  config:
    repository:
      TEMP_PATH: /data/gitea/tmp

Observations

1. The pod is running with correct user/group (1000:1000)
2. Directory permissions look correct:

   drwxrwsr-x    5 git      git           4096 Jul 10 12:11 /data/git
   drwxrwsr-x   43 git      git           4096 Apr 27 16:34 /data/git/gitea-repositories
   drwxr-sr-x    7 git      git           4096 Jul  9 09:46 /data/git/gitea-repositories/myorg/myrepo.git
   drwxr-sr-x    4 git      git           4096 Jul 10 12:14 /data/git/gitea-repositories/myorg/myrepo.git/objects
   

3. The error occurs when git-receive-pack creates temporary directories during object unpacking
4. The temporary directories (tmp_objdir-incoming-*) appear to be created with incorrect permissions

Questions

1. Is this a known issue with the rootless image when handling large git pushes?
2. Are there specific volume mount options or security contexts required for rootless Gitea on Kubernetes?
3. Would switching to the non-rootless image be the recommended solution?
4. Are there any git configuration options that could help with the temporary directory permissions?

Additional Information

• The volume is a Longhorn PVC mounted at /data
• The issue is reproducible across pod restarts
• SSH git operations have not been tested yet
• No relevant error messages in Gitea logs besides the git-receive-pack output

Any help or guidance would be greatly appreciated!

Gitea Version

1.24.2

Can you reproduce the bug on the Gitea demo site?

No

Log Gist

No response

Screenshots

No response

Git Version

No response

Operating System

NixOS with k3s (Kubernetes) with Longhorn and gitea as container

How are you running Gitea?

NixOS with k3s (Kubernetes) with Longhorn and gitea as container

Database

PostgreSQL

[wxiaoguang]

IIRC some other users also reported similar problems, but there is still no conclusion or solution ....

Is this a known issue with the rootless image when handling large git pushes?

I think yes but I don't have more idea (I don't use rootless)

Would switching to the non-rootless image be the recommended solution?

"rootful" and "rootless" use different directory layout and config, I don't think it can be switched easily, unless you backup the database and files, and setup a clean "rootful" instance, and put the files back to correct locations.

Are there any git configuration options that could help with the temporary directory permissions?

No idea, but I feel that the "tmp_objdir" is controlled by git command, so no Gitea config option would affect it

[PatrickHuetter]

I tested and checked so many things. I can't figure out what the problem is. It's massively blocking currently since we can't really push many files at once.

[wxiaoguang]

Some more questions:

1. Is the problem reproducible in only one or two repositories, or it is reproducible in all repositories?
2. If you deploy a clean local rootless instance, is it able to reproduce?

[PatrickHuetter]

Some more questions:

1. Is the problem reproducible in only one or two repositories, or it is reproducible in all repositories?
2. If you deploy a clean local rootless instance, is it able to reproduce?

1. It's reproducible in up to 6 repositories i tested. Also today in a clean new repository. For example: I created a new repository and initalized an angular app with some components and files (some pictures). Pushing the first commit isn't possible. Pushing the same local git repository to another origin (for example gitlab) works. Also mirroring the gitlab repository to gitea afterwards, doesn't work.

2. Didn't test this yet, since i only work on kubernetes currently. I'll look further the next days to test this or just create a new namespace with new gitea instance.

[wxiaoguang]

After googling about the "Permission denied tmp_objdir" error, I found that some users said that it is "filesystem/storage" related.

[PatrickHuetter]

Solution Found: Longhorn Storage Driver Bug

I've found the root cause and solution for this issue! 🎉

The Problem

This is not a Gitea bug, but rather a Longhorn storage driver issue that affects git operations creating temporary files during push/checkout operations.

The Solution

After extensive debugging and reading through release notes, I discovered that Longhorn had a known bug affecting git operations in containers. The issue is documented in [longhorn/longhorn#10975](longhorn/longhorn#10975) and was fixed in recent Longhorn releases.

The problem occurs because:

1. Git creates temporary directories (tmp_objdir-incoming-*) during large push operations
2. Longhorn's CSI driver had permission issues when handling these temporary file operations
3. This particularly affects rootless containers where permission handling is more strict

What Fixed It

Simply updating Longhorn to the latest version resolved the issue completely. The push operations that were failing with 30+ files now work perfectly without any changes to the Gitea configuration.

For Others Experiencing This Issue

If you're running Gitea on Kubernetes with Longhorn storage and experiencing similar permission errors:

1. Check your Longhorn version - The bug was present in older versions
2. Update Longhorn to the latest stable release
3. No Gitea configuration changes are necessary

Why This Matters

This issue can be misleading because:

• The error appears to be a Gitea permission problem
• It only occurs with larger git operations (30+ files)
• All directory permissions appear correct when inspected
• The rootless setup makes it seem like a container permission issue

But it's actually the underlying storage driver failing to properly handle git's temporary file operations during the receive-pack process.

I hope this helps others who might encounter the same issue. The Gitea rootless image works perfectly fine - just make sure your storage driver is up to date!

Environment After Fix

• Gitea: 1.24.2-rootless (unchanged)
• Longhorn: Updated to latest version
• No changes to security contexts, init containers, or Gitea configuration required

Feel free to close this issue as it's not a Gitea bug. Thanks for the great work on Gitea! 🚀