problems sending large data on macOS over unix domain sockets

[bagder]

(I received this bug report by a user who shall remain anonymous)

curl 7.68.0 seems to have problems sending large data on macOS over unix domain sockets. You can reproduce by using netcat as a server like this:

rm -rf /tmp/foofoo.sock && nc -U -l /tmp/foofoo.sock

And then invoke curl:

curl -v --unix-sock /tmp/foofoo.sock -H "$(python -c 'print "x"*90000'): OK" http://localhost:8888/

(This has curl send a very long single header that ends with OK)

If you look at the output from curl, you’ll notice that it never gets around to sending the OK part of the string. This seems to be specific to recent curls and to macOS

Taking a look at dtrace it seems like curl stops asking select whether the socket is writable

$ sudo dtrace -n 'syscall::sendto:entry / execname == "curl" && arg0 == 5 / { printf("%s(%d): %d, %s, %d", execname, pid, arg0, copyinstr(arg1), arg2); } syscall::sendto:return / execname == "curl" / { printf("return %d %d", arg1, errno); } syscall::select:entry / execname == "curl" / { printf("select(%d, %p, %p, %p)\n", arg0, arg1, arg2, arg3); printf("READ:\n"); tracemem(copyin(arg1, 128), 16); printf("WRITE:\n"); tracemem(copyin(arg2, 128), 128);   } syscall::select:return / execname=="curl" / { printf("%s return: %d %d", probefunc, arg1, errno); }'
Password:
dtrace: description 'syscall::sendto:entry ' matched 4 probes
CPU     ID                    FUNCTION:NAME
  2    347                     select:entry select(4, 7ffeebbdf230, 7ffeebbdf2b0, 7ffeebbdf330)
READ:
             0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f  0123456789abcdef
         0: 08 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
WRITE:
             0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f  0123456789abcdef
         0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
        10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
        20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
        30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
        40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
        50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
        60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
        70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
  2    348                    select:return select return: 0 0
  2    347                     select:entry select(6, 7ffeebbdf170, 7ffeebbdf0f0, 7ffeebbdf1f0)
READ:
             0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f  0123456789abcdef
         0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
WRITE:
             0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f  0123456789abcdef
         0: 20 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ...............
        10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
        20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
        30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
        40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
        50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
        60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
        70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
  2    348                    select:return select return: 1 0
  2    427                     sendto:entry curl(4116): 5, GET / HTTP/1.1
Host: localhost:8888
User-Agent: curl/7.68.0
Accept: */*
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx, 90084
  2    428                    sendto:return return 8192 0
  6    347                     select:entry select(6, 7ffeebbdf230, 7ffeebbdf1b0, 7ffeebbdf2b0)
READ:
             0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f  0123456789abcdef
         0: 20 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ...............
WRITE:
             0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f  0123456789abcdef
         0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
        10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
        20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
        30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
        40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
        50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
        60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
        70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
  6    348                    select:return select return: 1 0

(This is a printout of the send calls and then the fdsets passed to select, the relevant FD is 5 so 0x20)

[bagder]

What puzzles me the most about this bug is that very little internals even know what kind of socket it is working with and most of the code is just agnostic and work with "a socket" independent of what kind of connection it is. It might imply that unix domain sockets somehow behaves differently than SOCK_STREAM ones do - on macos. Research is required.

The bug triggers if the custom header is larger than 8100 bytes something. Then only the first piece of the request is sent and nothing more is ever sent. The socket just doesn't seem to signal itself as writable anymore...

[bagder]

The bug triggers on macOS if the custom header is larger than 8100 bytes something. Then only the first piece of the request is sent and nothing more is ever sent. Clearly the logic for handling a "split" request send is broken, and I believe it is broken on all platforms.

This bug also reproduces on Linux but then at around ~220K so the custom header needs to be made much larger there.

[bagder]

I arrived at this little patch that lets me write a simple test case for this problem using the regular test suite, which makes it easier to work on a fix:

diff --git a/lib/http.c b/lib/http.c
index 324dd7252..814d69fd2 100644
--- a/lib/http.c
+++ b/lib/http.c
@@ -1227,12 +1227,25 @@ CURLcode Curl_add_buffer_send(Curl_send_buffer **inp,
       return result;
     }
     memcpy(data->state.ulbuf, ptr, sendsize);
     ptr = data->state.ulbuf;
   }
-  else
+  else {
+#ifdef CURLDEBUG
+    /* Allow debug builds override this logic to force short initial sends */
+    char *p = getenv("CURL_SMALLREQSEND");
+    if(p) {
+      size_t altsize = (size_t)strtoul(p, NULL, 10);
+      if(altsize)
+        sendsize = CURLMIN(size, altsize);
+      else
+        sendsize = size;
+    }
+    else
+#endif
     sendsize = size;
+  }
 
   result = Curl_write(conn, sockfd, ptr, sendsize, &amount);
 
   if(!result) {
     /*