Issues with Internxt CLI v1.5.3 WebDAV: 401 Unauthorized, 409 Conflict, and Long Hangs During rclone Sync

[Simonz82]

Description
I'm experiencing multiple issues while using Internxt CLI v1.5.3 with rclone to sync large photo/video directories to Internxt via WebDAV. The setup involves syncing directories organized by year (e.g., 2007–2025) from a Synology NAS to Internxt. Despite optimizing the rclone configuration, I encounter persistent errors (401 Unauthorized, 409 Conflict), long hangs without error messages, and slow performance with large files or dense folders. Below, I detail the issues, environment, logs, and attempted solutions, hoping for assistance or guidance.
Environment
Internxt CLI Version: v1.5.3 (internxt --version)

rclone Version: 1.69.1

OS: Synology DSM [Specify DSM version 7.2]

Node.js Version: 20.12.2

Source Directory: /volume2/photo/2009 (example, there are many years)

Destination: internxt:SynologyBackup/photo/2009

Directory Details:
Each year contains subfolders (e.g., /volume2/photo/2009/20091004 Pista Lodi Video).

Subfolders may have hundreds to thousands of files (photos/videos, 1 MB to >5 GB).

Total size per year: ~50–200 GB, with ~7 files missing in 2009.

Network: Stable broadband connection (~20 Mbps upload).

rclone Configuration
ini

[internxt]
type = webdav
url = http://localhost:3005
vendor = other
user = [redacted]
pass = [redacted]

Running internxt webdav enable starts the WebDAV server on port 3005 (confirmed with netstat -tuln | grep :3005).

Using --no-check-certificate due to SSL issues.

Script Details
I'm using a Bash script to sync directories year-by-year with rclone sync. Key options:
--transfers 1 to avoid overloading the server.

--size-only (Internxt doesn't support hashes).

--timeout 300s, --contimeout 300s, --retries 10.

--tpslimit 10, --tpslimit-burst 10.

--bwlimit 1M to reduce server load.

--retries-sleep 5s for delays between retries.

--progress and --verbose for detailed logging.

Exclusions: --exclude "@eaDir/**", --exclude "Thumbs.db", --exclude ".DS_Store".

The script checks for missing files with rclone check, syncs only missing files using --files-from, and verifies with rclone check post-sync. It also retries files with 409 Conflict errors.
Issues Encountered
401 Unauthorized Errors
Description: Previously, rclone commands (mkdir, sync, check) failed with CRITICAL: Failed to create file system for "internxt:": read metadata failed: 401 Unauthorized.

Context: Occurred when creating folders or syncing (e.g., internxt:SynologyBackup/photo/2009). The script incorrectly reported folders as created despite the error.

Log Example:

2025-05-19 14:26:18 CRITICAL: Failed to create file system for "internxt:SynologyBackup/photo": read metadata failed: 401 Unauthorized
2025-05-19 14:26:18 - Cartella SynologyBackup/photo creata con successo

Attempted Fixes:
Re-logged in with internxt login to refresh the token.

Reconfigured the internxt remote in rclone with updated credentials.

Ran internxt webdav status and internxt webdav enable.

Tested with rclone lsd internxt: (sometimes successful, sometimes failed).

Status: Resolved after reconfiguring credentials, but the issue suggests unstable authentication.

409 Conflict Errors
Description: Some files failed to sync with 409 Conflict errors, possibly due to concurrent operations or server-side issues.

Context: Occurred with specific files (e.g., those with special characters like & in folder names). The script retries these files, but some persist.

Log Example:

[Previous logs showed 409 Conflict for files like "matrimonio davide&federica21-05-11"]

Attempted Fixes:
Reduced --transfers to 1 and added --tpslimit 10.

Renamed problematic folders (e.g., replaced & with _).

Used --ignore-existing to retry failed files.

Status: Reduced frequency, but occasional errors persist, requiring manual retries.

Long Hangs Without Errors
Description: The script hangs for hours on certain folders (e.g., /volume2/photo/2009/20091004 Pista Lodi Video) during rclone sync, with no error messages or progress updates despite --progress and --verbose.

Context: The 2009 directory is mostly synced (only 7 files missing). Other subfolders (e.g., Reno 27-09-09) sync quickly (e.g., 66 files in 0.6s). The hanging folder likely contains videos (>5 GB) or many files.

Log Example:

2025-05-19 16:28:34 - Sincronizzo cartella: /volume2/photo/2009/20091004 Pista Lodi Video in internxt:SynologyBackup/photo/2009/20091004 Pista Lodi Video
[No further output for over 1 hour]

Attempted Fixes:
Added --bwlimit 1M to reduce server load.

Increased --timeout and --contimeout to 300s.

Added --retries-sleep 5s for delays.

Restarted the WebDAV server with internxt webdav restart.

Tested manually with rclone sync --dry-run on the folder (still hangs).

Status: Unresolved. Likely caused by large files, dense folders, or server-side issues.

Slow Performance for Already Synced Folders
Description: Even for folders already synced (e.g., 2009 with only 7 missing files), rclone sync and rclone check can be slow, especially with subfolders containing many files or large videos.

Context: Expected quick checks (e.g., 0.5–0.6s for 6–66 files), but some folders take minutes or hang.

Log Example:

2025-05-19 16:28:33 - 6 file saltati (già aggiornati) in /volume2/photo/2009/20090927 Pista Cassano Spinola e Reno/no
[But hangs on other folders like 20091004 Pista Lodi Video]

Attempted Fixes:
Used --size-only to avoid hash checks.

Skipped empty folders with find.

Pre-checked missing files with rclone check and synced only those with --files-from.

Status: Improved with pre-checks, but hangs persist for certain folders.

Steps to Reproduce
Configure rclone with Internxt WebDAV (url=http://localhost:3005).

Run internxt webdav enable to start the WebDAV server.

Use a script or command to sync a large directory (e.g., /volume2/photo/2009) with subfolders containing photos/videos:
bash

rclone sync "/volume2/photo/2009" "internxt:SynologyBackup/photo/2009"
--progress --verbose --transfers 1 --size-only --timeout 300s
--tpslimit 10 --bwlimit 1M --retries 10 --retries-sleep 5s

Observe 401 Unauthorized (if credentials are stale), 409 Conflict (for some files), or hangs (on folders like 20091004 Pista Lodi Video).

Check logs for errors or lack of progress.

Expected Behavior
Reliable authentication without 401 Unauthorized errors.

No 409 Conflict errors for files with standard names.

Quick sync/check for already synced folders (e.g., <1s for dozens of files).

No hangs on large files or dense folders, with clear error messages if issues occur.

Consistent progress updates with --progress and --verbose.

Actual Behavior
Intermittent 401 Unauthorized errors (resolved with re-authentication).

Occasional 409 Conflict errors requiring retries or renaming.

Long hangs (>1 hour) on certain folders without errors or progress.

Slow performance for folders with many files or large videos, even if mostly synced.

Logs
401 Unauthorized Example:

2025-05-19 14:26:18 CRITICAL: Failed to create file system for "internxt:SynologyBackup/photo": read metadata failed: 401 Unauthorized

Hang Example:

2025-05-19 16:28:34 - Sincronizzo cartella: /volume2/photo/2009/20091004 Pista Lodi Video in internxt:SynologyBackup/photo/2009/20091004 Pista Lodi Video
[No output for over 1 hour]

Successful Quick Sync:

2025-05-19 16:28:33 - 6 file saltati (già aggiornati) in /volume2/photo/2009/20090927 Pista Cassano Spinola e Reno/no
Transferred: 0 B / 0 B, -, 0 B/s, ETA -
Checks: 6 / 6, 100%
Elapsed time: 0.5s

Full log available at /var/services/NetBackup/internxt/rclone_upload.log (can share if needed).

Additional Context
The WebDAV server is responsive (port 3005 open), but performance degrades with large files or many files.

Manual tests with rclone lsd internxt: and rclone copy (single file) work, but bulk syncs hang.

Folder 20091004 Pista Lodi Video likely contains videos (>5 GB) or many files, causing hangs.

Using v1.5.3 for improved authentication, but hangs and 409 Conflict issues persist.

Request for Assistance
Are there known issues with WebDAV hangs on large files or dense folders in CLI v1.5.3?

Any recommended rclone flags or Internxt CLI settings to prevent hangs and improve performance?

Could 409 Conflict errors be related to server-side concurrency limits or file name issues?

Is there a way to debug hangs (e.g., enable more verbose WebDAV logs)?

Are there plans to optimize WebDAV for large-scale syncs (e.g., >100 GB, thousands of files)?

Workarounds Tried
Updated CLI to v1.5.3 and refreshed token with internxt login.

Reconfigured rclone remote and tested credentials.

Reduced --transfers, added --bwlimit, --retries-sleep, and --tpslimit.

Pre-checked missing files with rclone check and synced only those.

Renamed folders with special characters (e.g., & to _).

Restarted WebDAV server with internxt webdav restart.

This issue is blocking critical backups of large photo/video archives. A prompt response would be greatly appreciated.

you have the ability to do syncs with RCLONE.
how is it possible that it does not paste a simple rclone string to launch a quick sync with creation of folders and subfolders etc.?
i had to write a 300 line *.sh file and still i always have problems and slowness.
Am I doing something wrong or are there problems ?
because otherwise i will uninstall this service if it is so cumbersome ,slow and problematic

[Simonz82]

I can connect and upload.

But with the rclon sync command, it takes hours and hours to do a sync, just to chack for folders with images already uploaded.

[b3ni]

Same issue

[GiacomoFicarola]

Same issue, I create a small container to upload file via WebDAV on TrueNAS, but I obtain much 409 conflict and various errors.

[thies2005]

Same issue, any update on how to fix

[MrEAlderson]

It occurs since the Internxt WebDav server does not permit uploading files whose parent directories do not exist (ref). However, rclone only creates the folder for the parent folder using its absolute path, and not each and every folder within it (ref). Rclone does not provide anything to overcome that. And according to the WebDav specification, both ways of handling folder creation are acceptable (ref).

The server MUST NOT create those intermediate collections automatically.

Either rclone would have to add a flag to create the directories for all the subdirectories or internxt's client would have to be less restrictive. Until then, you may use this hacky solution, which creates the directories for you:

#!/bin/bash
SRC="/media/xyz"
DST="internxt:"

# Check if the directory exists
if [ ! -d "$SRC" ]; then
  echo "Error: '$SRC' is not a valid directory."
  exit 1
fi

find "$SRC" -type d | while read -r dir; do
    # Skip leaf folders (no subdirectories)
    if ! find "$dir" -mindepth 1 -type d -print -quit | grep -q .; then
        continue
    fi

    rel_path="${dir#$SRC/}"
    remote_path="$DST/$rel_path"

    echo "Creating: $remote_path"
    rclone mkdir "$remote_path" --no-check-certificate --tpslimit 10
done

[thies2005]

Thank you so much MrEAlderson, it worked immedeatly

[AlexisMora]

Hi! on the latest release v1.5.8 We just fixed the 409 conflict bug by automatically creating the paths if the are not found, please try it out. if you encounter some errors pleas reach out to us at hello@internxt.com

Thanks for helping us improve!