Upload via WebDAV on macOS fails silently, creates zero byte files

[philwo]

Steps to reproduce

1. Upgrade Nextcloud AIO from v7.12.1 to v8.2.0.
2. Mount my Nextcloud files via macOS 14.4.1 Finder's WebDAV.
3. Drag and drop a file into the mounted WebDAV share.

Expected behavior

The file gets stored on Nextcloud correctly.

Actual behavior

The upload completes successfully from macOS's perspective, but only an empty, zero byte file is stored in Nextcloud.

Host OS

Ubuntu 22.04.4 LTS

Nextcloud AIO version

v8.2.0

Current channel

latest

Other valuable info

Happens with Nextcloud v28.0.4 as well as the just released Nextcloud v29.0.0.

I checked the diff between v7.12.1 and v8.2.0 and believe the culprit might be this change: #4323 - the idea behind it and the upstream change seems good, but for some reason it's not working (maybe due to something AIO-specific, considering that no one has complained on the upstream PR?).

This is causing data loss for me at the moment, especially because it doesn't result in a user visible error - the Finder and other apps saving files to that mounted WebDAV share believe that they did so successfully, so the user might not notice that their files have never been saved.

I will try to revert this change locally in my configuration as a workaround for now, will report back whether that fixed it.

[szaimen]

cc @artonge I thought this was fixed in nextcloud/server#43129?

Apparently some other people are reporting as well that it doesnt work: nextcloud/server#43047 (comment)

[artonge]

AIO uses Nginx, right? The PR only fix the issue for Apache. I guess we need a Nginx equivalent.
Edit: I even state that limitation in the PR :)

[szaimen]

No, AIO uses Apache internally. That is why it should work in theory. However apparently it doesnt work...

@philwo do you have a reverse proxy in front of AIO?

[philwo]

Thanks for the quick reply!

@szaimen No, I'm using pretty much a default setup of Nextcloud AIO on this VM, without a reverse proxy. The VM also does nothing else and has direct internet access via IPv4 and IPv6.

For reference, here's some hopefully helpful output to confirm that:

root@nc:/var/lib/docker# ss -ntpl
State  Recv-Q Send-Q               Local Address:Port  Peer Address:PortProcess
LISTEN 0      4096                       0.0.0.0:80         0.0.0.0:*    users:(("docker-proxy",pid=1257,fd=4))
LISTEN 0      4096                       0.0.0.0:8443       0.0.0.0:*    users:(("docker-proxy",pid=1125,fd=4))
LISTEN 0      4096                       0.0.0.0:443        0.0.0.0:*    users:(("docker-proxy",pid=552533,fd=4))
LISTEN 0      128                        0.0.0.0:22585      0.0.0.0:*    users:(("sshd",pid=801,fd=3))
LISTEN 0      4096                 127.0.0.53%lo:53         0.0.0.0:*    users:(("systemd-resolve",pid=685,fd=14))
LISTEN 0      4096                       0.0.0.0:8080       0.0.0.0:*    users:(("docker-proxy",pid=1219,fd=4))
LISTEN 0      4096                          [::]:80            [::]:*    users:(("docker-proxy",pid=1273,fd=4))
LISTEN 0      4096                          [::]:8443          [::]:*    users:(("docker-proxy",pid=1132,fd=4))
LISTEN 0      4096                          [::]:443           [::]:*    users:(("docker-proxy",pid=552540,fd=4))
LISTEN 0      128                           [::]:22585         [::]:*    users:(("sshd",pid=801,fd=4))
LISTEN 0      4096                          [::]:8080          [::]:*    users:(("docker-proxy",pid=1226,fd=4))

root@nc:/var/lib/docker# docker ps
CONTAINER ID   IMAGE                              COMMAND                  CREATED          STATUS                    PORTS                                                                                                                               NAMES
ed2804daee27   nextcloud/aio-apache:latest        "/start.sh /usr/bin/…"   46 minutes ago   Up 46 minutes (healthy)   80/tcp, 0.0.0.0:443->443/tcp, 0.0.0.0:443->443/udp, :::443->443/tcp, :::443->443/udp                                                nextcloud-aio-apache
7f3b5019c943   nextcloud/aio-notify-push:latest   "/start.sh"              46 minutes ago   Up 46 minutes (healthy)                                                                                                                                       nextcloud-aio-notify-push
7cb6cf78e761   nextcloud/aio-nextcloud:latest     "/start.sh /usr/bin/…"   46 minutes ago   Up 46 minutes (healthy)   9000/tcp                                                                                                                            nextcloud-aio-nextcloud
4bc09bec451a   nextcloud/aio-redis:latest         "/start.sh"              46 minutes ago   Up 46 minutes (healthy)   6379/tcp                                                                                                                            nextcloud-aio-redis
ba508a412c73   nextcloud/aio-postgresql:latest    "/start.sh"              46 minutes ago   Up 46 minutes (healthy)   5432/tcp                                                                                                                            nextcloud-aio-database
183bbec96c56   nextcloud/all-in-one:latest        "/start.sh"              19 hours ago     Up 19 hours (healthy)     0.0.0.0:80->80/tcp, :::80->80/tcp, 0.0.0.0:8080->8080/tcp, :::8080->8080/tcp, 0.0.0.0:8443->8443/tcp, :::8443->8443/tcp, 9000/tcp   nextcloud-aio-mastercontainer

I was also curious whether maybe something prevents the changes in the .htaccess that nextcloud/server#43129 made from taking effect, but can't see anything.

The conf/nextcloud.conf in the Apache container sets AllowOverride All:

ed2804daee27:/usr/local/apache2$ grep -C 5 AllowOverride conf/nextcloud.conf
    # Nextcloud dir
    DocumentRoot /var/www/html/
    <Directory /var/www/html/>
        Options Indexes FollowSymLinks
        Require all granted
        AllowOverride All
        Options FollowSymLinks MultiViews
        Satisfy Any
        <IfModule mod_dav.c>
            Dav off
        </IfModule>

And the .htaccess also contains the changes from that PR on my system:

ed2804daee27:/usr/local/apache2$ grep -C 5 sendcl /var/www/html/.htaccess
# Here are more information about the issue:
#  - https://docs.cyberduck.io/mountainduck/issues/fastcgi/
#  - https://docs.nextcloud.com/server/latest/admin_manual/issues/general_troubleshooting.html#troubleshooting-webdav
<IfModule setenvif.c>
  <Location "/remote.php">
    SetEnvIf Transfer-Encoding "chunked" proxy-sendcl=1
  </Location>
</IfModule>

AddDefaultCharset utf-8
Options -Indexes

[philwo]

Mhm.. it looks like nextcloud-aio-apache is running Caddy as a reverse proxy for https in front of Apache? Does that cause this issue, maybe? 🤔

root@nc:~# ps x | grep docker-proxy | egrep ' 443$'
 552533 ?        Sl     0:00 /usr/bin/docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 443 -container-ip 172.18.0.7 -container-port 443
 552540 ?        Sl     0:00 /usr/bin/docker-proxy -proto tcp -host-ip :: -host-port 443 -container-ip 172.18.0.7 -container-port 443
 552555 ?        Sl     0:00 /usr/bin/docker-proxy -proto udp -host-ip 0.0.0.0 -host-port 443 -container-ip 172.18.0.7 -container-port 443
 552562 ?        Sl     0:00 /usr/bin/docker-proxy -proto udp -host-ip :: -host-port 443 -container-ip 172.18.0.7 -container-port 443

root@nc:~# docker inspect $(docker ps -q ) --format='{{ printf "%-50s" .Name}} {{range .NetworkSettings.Networks}}{{.IPAddress}} {{end}}' | fgrep 172.18.0.7
/nextcloud-aio-apache                              172.18.0.7

root@nc:~# docker inspect -f '{{.State.Pid}}' nextcloud-aio-apache
552600

root@nc:~# nsenter -t 552600 -n ss -ntpl
State    Recv-Q   Send-Q   Local Address:Port   Peer Address:Port  Process
LISTEN   0        4096     127.0.0.11:42275     0.0.0.0:*          users:(("dockerd",pid=799,fd=73))
LISTEN   0        4096     127.0.0.1:2019       0.0.0.0:*          users:(("caddy",pid=553024,fd=3))
LISTEN   0        4096     *:443                *:*                users:(("caddy",pid=553024,fd=8))
LISTEN   0        511      *:8000               *:*                users:(("httpd",pid=553133,fd=4),("httpd",pid=553052,fd=4),("httpd",pid=553041,fd=4),("httpd",pid=553039,fd=4),("httpd",pid=553026,fd=4))

[szaimen]

Mhm.. it looks like nextcloud-aio-apache is running Caddy as a reverse proxy for https in front of Apache? Does that cause this issue, maybe? 🤔

I cannot tell but shouldn't if I understand this correctly...

[szaimen]

Hi @philwo I just tested with Cyberduck on my test instance and cannot reproduce this issue. So at least for cyberduck it seems to be fixed. Can you please try to revert this PR by settings this value to false:

all-in-one/php/containers.json

Line 61 in bab11ed

"read_only": true,

and then restarting the containers via the AIO interface? Afterwards you should be able to edit the nextcloud.conf inside the container and after a restart it should be taken into account.

[philwo]

Before trying this, I captured some packet traces inside the container with tcpdump in the hope that they might prove useful. These traces show the HTTP traffic between Caddy and Apache, so they also prove that Caddy doesn't somehow interfere with the headers. I did one extra run with Charles Proxy on my client and compared the headers - they're the same, nothing gets lost or modified by Caddy.

I can't reproduce it with Cyberduck, and I believe that's because Cyberduck doesn't use chunked transfer. Only Mountainduck does, their virtual filesystem that can mount network shares as drives. However I can reproduce it with curl (you have to upload a big enough file to trigger the bug, though - 4 MB of random data worked for me).

I captured the packets on the server with tcpdump to see what's going on:

macOS Sonoma 14.4.1:

Hypertext Transfer Protocol, has 513 chunks (including last chunk)
    PUT /remote.php/dav/files/admin/***/**************/random.4MB.finder HTTP/1.1\r\n
    Host: nc.*************.de\r\n
    User-Agent: WebDAVFS/3.0.0 (03008000) Darwin/23.4.0 (arm64)\r\n
    Transfer-Encoding: chunked\r\n
    Accept: */*\r\n
    Authorization: Basic ******=\r\n
    Cookie: ******
    If: (<opaquelocktoken:56c6246edd65237f80ff24280975a1d0>)\r\n
    X-Expected-Entity-Length: 4194304\r\n
    X-Forwarded-For: 123.123.123.12\r\n
    X-Forwarded-Host: nc.*************.de\r\n
    X-Forwarded-Proto: https\r\n
    Accept-Encoding: gzip\r\n
    \r\n
    [Full request URI: http://nc.*************.de/remote.php/dav/files/admin/***/**************/random.4MB.finder]
    [HTTP request 34/42]
    [Prev request in frame: 3550]
    [Response in frame: 6137]
    [Next request in frame: 6156]
    HTTP chunked response
    File Data: 4194304 bytes
    Data (4194304 bytes)

curl 8.4.0 (x86_64-apple-darwin23.0):

Hypertext Transfer Protocol, has 167 chunks (including last chunk)
    PUT /remote.php/dav/files/admin/***/**************/random.4MB.curl.v2 HTTP/1.1\r\n
    Host: nc.*************.de\r\n
    User-Agent: curl/8.4.0\r\n
    Transfer-Encoding: chunked\r\n
    Accept: */*\r\n
    Authorization: Basic ******=\r\n
    Content-Type: application/x-www-form-urlencoded\r\n
    X-Forwarded-For: 123.123.123.12\r\n
    X-Forwarded-Host: nc.*************.de\r\n
    X-Forwarded-Proto: https\r\n
    Accept-Encoding: gzip\r\n
    \r\n
    [Full request URI: http://nc.*************.de/remote.php/dav/files/admin/***/**************/random.4MB.curl.v2]
    [HTTP request 1/1]
    [Response in frame: 952]
    HTTP chunked response
    File Data: 4194304 bytes

Cyberduck 8.8.2 (41344):

Hypertext Transfer Protocol
    PUT /remote.php/dav/files/admin/***/**************/random.4MB.cyberduck HTTP/1.1\r\n
    Host: nc.*************.de\r\n
    User-Agent: Cyberduck/8.8.2.41344 (Mac OS X/14.4.1) (aarch64)\r\n
    Content-Length: 4194304\r\n
    Accept-Encoding: gzip,deflate\r\n
    Authorization: Basic ******=\r\n
    Content-Type: application/octet-stream\r\n
    Cookie: ******
    Expect: 100-continue\r\n
    X-Forwarded-For: 123.123.123.12\r\n
    X-Forwarded-Host: nc.*************.de\r\n
    X-Forwarded-Proto: https\r\n
    \r\n
    [Full request URI: http://nc.*************.de/remote.php/dav/files/admin/***/**************/random.4MB.cyberduck]
    [HTTP request 14/15]
    [Response in frame: 2819]
    [Next request in frame: 2822]
    File Data: 4194304 bytes

This is the result on the server:

[philwo]

I can confirm that reverting https://github.com/nextcloud/all-in-one/pull/4323/files makes both curl and macOS Finder work fine immediately :)

[philwo]

By the way, the trick with setting "read_only": false, is really nice, thank you for sharing that! I was looking for exactly this earlier today.

Unfortunately (although it could also be seen as a safety feature?), when I stop and start the containers via the AIO interface, any changed files get overwritten again. I think this is because

all-in-one/php/src/Controller/DockerController.php

Lines 60 to 67 in a1037ba

	$this->dockerActionManager->DeleteContainer($container);
	$this->dockerActionManager->CreateVolumes($container);
	if ($pullImage) {
	$this->dockerActionManager->PullImage($container);
	}
	$this->dockerActionManager->CreateContainer($container);
	$this->dockerActionManager->StartContainer($container);
	$this->dockerActionManager->ConnectContainerToNetwork($container);

actually deletes and then recreates all containers then.

However, for this test it was no problem, as I could just run apachectl -k restart after editing the config and didn't have to restart the whole container.

[philwo]

Ah, I forgot to mention, this is the full command-line how to reproduce this issue with curl (I also confirmed it with the latest curl version built from source to ensure it's not just an issue with the Apple version):

dd if=/dev/urandom of=random.4MB bs=1M count=4
curl \
  -v \
  --user "admin:hunter2" \
  -H "Transfer-Encoding: chunked" \
  --data-binary "@random.4MB" \
  -X PUT \
  "https://nc.example.com/remote.php/dav/files/admin/random.4MB.curl"

[szaimen]

Thanks a lot for your help on this @philwo! :)

Due to you I found a fix in nextcloud/server#45035

[szaimen]

This is now fixed with v8.2.1 Beta. Testing and feedback is welcome! See https://github.com/nextcloud/all-in-one#how-to-switch-the-channel