Local path destination pre-check fails on Unraid despite target being writable

[tmeuze]

Component

Agent

Description

Unraid bind mounts directories to containers using overlay2 in a special manner (from what I gather), whereby RW and RO access to root (/) and RW to /mnt is disallowed. Therefore, one must use a full virtual path to a "user share", such as "Arkeep" at /mnt/user/Arkeep.

Arkeep does not seem to appreciate this. After job execution but prior to its run, the prerun check (using mkdir) fails and therefore so does the backup job altogether.

Despite failures in job runs, it is known that the directory/destination is writable per:

# Direct write test — succeeds
docker exec arkeep-agent touch /hostfs/mnt/user/Arkeep/test-write
# → success

# mkdir test — succeeds  
docker exec arkeep-agent mkdir -p /hostfs/mnt/user/Arkeep/test-dir
# → success

Steps to Reproduce

1. Spin up AIO (Server + Agent) container (Debian 13 VPS) - works fine
2. Spin up Agent-only container (Unraid):

services:
  agent:
    image: ghcr.io/arkeep-io/arkeep-agent:latest
    container_name: arkeep-agent
    restart: unless-stopped
    network_mode: bridge
    environment:
      ARKEEP_SERVER_ADDR: "${ARKEEP_SERVER_ADDR:?ARKEEP_SERVER_ADDR is required — gRPC address, e.g. arkeep.example.com:9090}"
      ARKEEP_AGENT_SECRET: "${ARKEEP_AGENT_SECRET:?ARKEEP_AGENT_SECRET is required — must match the server}"
      ARKEEP_SERVER_HTTP_ADDR: "${ARKEEP_SERVER_HTTP_ADDR:-}"
      ARKEEP_STATE_DIR: "/var/lib/arkeep-agent"
      ARKEEP_LOG_LEVEL: "${ARKEEP_LOG_LEVEL:-info}"
      TZ: "${TZ:-UTC}"
      ARKEEP_DOCKER_HOST_ROOT: "/hostfs"
    volumes:
      - /mnt/cache/appdata/arkeep/data/agent:/var/lib/arkeep-agent
      # - /:/hostfs:ro
      - /mnt/user:/hostfs/mnt/user:rw
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - /var/lib/docker/volumes:/var/lib/docker/volumes:ro

Additional testing

• Uncommented /:/hostfs:rw bind
• Altered to /:/hostfs:ro bind
• Commenting ARKEEP_DOCKER_HOST_ROOT variable

3. Enroll agents - works fine
4. Set destination:

• Path: /mnt/user/Arkeep

Additional testing

• Changing path to: /hostfs/mnt/user/Arkeep
• Deleting destination; recreating anew (tried both paths above)

5. Ran job - immediate failure

Additional testing

• Re-ran same job (via policy) after modifying and rebuilding the docker container - always an immediate failure
  • Side note: I noticed that you cannot remove a destination selected for a policy once that policy has been committed. Any subsequent changes will revert (can make a new issue for this).
• Created new policies and job runs to test changes above, with the assumption that there was a hard pairing between policy, destination, job IDs - and attempts.

Expected Behavior

Expected

Agent translates /mnt/user/Arkeep → /hostfs/mnt/user/Arkeep via ARKEEP_DOCKER_HOST_ROOT, then checks writability against the translated path, which is a real writable bind-mount.

Actual

The pre-check fails before the job runs with:
local path "/mnt/user/Arkeep" is not writable: mkdir /mnt: read-only file system

The error shows that the pre-check is attempting mkdir /mnt — walking up from the container root rather than through the hostfs translation — and hitting the read-only overlay at /mnt before ever reaching the writable bind-mount.

Arkeep Version

v0.3.0 (Docker)

Deployment Method

Docker Compose

Relevant Logs

### Job Detail

5:16:24 PM
INFO
backup started
5:16:24 PM
INFO
resolved 1 source(s)
5:16:24 PM
INFO
backing up to destination 019dcb9f-cb48-779f-8cf5-58dffd31ea73 (type: local)
5:16:24 PM
ERROR
backup to destination 019dcb9f-cb48-779f-8cf5-58dffd31ea73 failed: local path "/mnt/user/Arkeep" is not writable: mkdir /mnt: read-only file system — if running inside Docker, set ARKEEP_DOCKER_HOST_ROOT and mount the host filesystem (e.g. /:/hostfs:rw on Linux, C:/:/hostfs/c:rw on Windows), or set PUID/PGID to match the directory owner
5:16:24 PM
ERROR
one or more destinations failed


### Job Detail

5:31:02 PM
INFO
backup started
5:31:02 PM
INFO
resolved 1 source(s)
5:31:02 PM
INFO
backing up to destination 019dcbb1-998a-73a4-8d9c-653192797d66 (type: local)
5:31:02 PM
ERROR
backup to destination 019dcbb1-998a-73a4-8d9c-653192797d66 failed: local path "/hostfs/mnt/user/Arkeep" is not writable: mkdir /hostfs: read-only file system — if running inside Docker, set ARKEEP_DOCKER_HOST_ROOT and mount the host filesystem (e.g. /:/hostfs:rw on Linux, C:/:/hostfs/c:rw on Windows), or set PUID/PGID to match the directory owner
5:31:02 PM
ERROR
one or more destinations failed

Additional Context

Environment:

Arkeep agent version: 0.3.0
Deployment: Docker (Compose Manager on Unraid 6.12+)
Host OS: Unraid (root filesystem is a union/overlay — /mnt is a virtual mount point, not a real writable directory)
Overlay Network: Headscale (Tailscale variant) - working fine

Configuration:

environment:
  ARKEEP_DOCKER_HOST_ROOT: "/hostfs"

volumes:
  - /mnt/user:/hostfs/mnt/user:rw
  - /var/run/docker.sock:/var/run/docker.sock:ro
  - /var/lib/docker/volumes:/var/lib/docker/volumes:ro

Working hypothesis:

The writability pre-check appears to apply mkdir -p style logic against the untranslated path on the container filesystem, rather than against the hostfs-translated path. On a standard Linux host /mnt exists and is writable so this goes unnoticed, but on Unraid /mnt in the container root (via /:/hostfs:ro) is a read-only overlay mount point, exposing the issue.

Thank you in advance! Please let me know if additional information would help.

[filippocrotti12]

Thanks, this is very likely an agent-side bug in v0.3.0 rather than a compose issue.

Your Unraid setup looks correct: with /mnt/user mounted as /hostfs/mnt/user and ARKEEP_DOCKER_HOST_ROOT=/hostfs, the destination /mnt/user/Arkeep should be translated internally to /hostfs/mnt/user/Arkeep.

The error shows the agent is trying to create /mnt/... directly inside the container, so the Docker host-path translation is not being applied before the local destination pre-check.

I’ve implemented a fix locally that:

• improves Docker/container detection, including Unraid/containerd-style environments;
• applies local path translation before checking/creating the destination directory;
• returns a clearer error if the host path is not mounted or writable from the agent container.

This will be included in the next release. Once released, please keep using /mnt/user/Arkeep in the UI rather than /hostfs/mnt/user/Arkeep, because /hostfs is only the internal container mount prefix used by the agent.

[tmeuze]

Thank you my friend!