Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

problems sending large data on macOS over unix domain sockets #4919

Closed
bagder opened this issue Feb 12, 2020 · 3 comments
Closed

problems sending large data on macOS over unix domain sockets #4919

bagder opened this issue Feb 12, 2020 · 3 comments
Assignees

Comments

@bagder
Copy link
Member

@bagder bagder commented Feb 12, 2020

(I received this bug report by a user who shall remain anonymous)

curl 7.68.0 seems to have problems sending large data on macOS over unix domain sockets. You can reproduce by using netcat as a server like this:

rm -rf /tmp/foofoo.sock && nc -U -l /tmp/foofoo.sock

And then invoke curl:

curl -v --unix-sock /tmp/foofoo.sock -H "$(python -c 'print "x"*90000'): OK" http://localhost:8888/

(This has curl send a very long single header that ends with OK)

If you look at the output from curl, you’ll notice that it never gets around to sending the OK part of the string. This seems to be specific to recent curls and to macOS

Taking a look at dtrace it seems like curl stops asking select whether the socket is writable

$ sudo dtrace -n 'syscall::sendto:entry / execname == "curl" && arg0 == 5 / { printf("%s(%d): %d, %s, %d", execname, pid, arg0, copyinstr(arg1), arg2); } syscall::sendto:return / execname == "curl" / { printf("return %d %d", arg1, errno); } syscall::select:entry / execname == "curl" / { printf("select(%d, %p, %p, %p)\n", arg0, arg1, arg2, arg3); printf("READ:\n"); tracemem(copyin(arg1, 128), 16); printf("WRITE:\n"); tracemem(copyin(arg2, 128), 128);   } syscall::select:return / execname=="curl" / { printf("%s return: %d %d", probefunc, arg1, errno); }'
Password:
dtrace: description 'syscall::sendto:entry ' matched 4 probes
CPU     ID                    FUNCTION:NAME
  2    347                     select:entry select(4, 7ffeebbdf230, 7ffeebbdf2b0, 7ffeebbdf330)
READ:
             0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f  0123456789abcdef
         0: 08 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
WRITE:
             0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f  0123456789abcdef
         0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
        10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
        20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
        30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
        40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
        50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
        60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
        70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
  2    348                    select:return select return: 0 0
  2    347                     select:entry select(6, 7ffeebbdf170, 7ffeebbdf0f0, 7ffeebbdf1f0)
READ:
             0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f  0123456789abcdef
         0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
WRITE:
             0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f  0123456789abcdef
         0: 20 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ...............
        10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
        20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
        30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
        40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
        50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
        60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
        70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
  2    348                    select:return select return: 1 0
  2    427                     sendto:entry curl(4116): 5, GET / HTTP/1.1
Host: localhost:8888
User-Agent: curl/7.68.0
Accept: */*
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx, 90084
  2    428                    sendto:return return 8192 0
  6    347                     select:entry select(6, 7ffeebbdf230, 7ffeebbdf1b0, 7ffeebbdf2b0)
READ:
             0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f  0123456789abcdef
         0: 20 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ...............
WRITE:
             0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f  0123456789abcdef
         0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
        10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
        20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
        30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
        40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
        50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
        60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
        70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
  6    348                    select:return select return: 1 0

(This is a printout of the send calls and then the fdsets passed to select, the relevant FD is 5 so 0x20)

@bagder
Copy link
Member Author

@bagder bagder commented Apr 5, 2020

What puzzles me the most about this bug is that very little internals even know what kind of socket it is working with and most of the code is just agnostic and work with "a socket" independent of what kind of connection it is. It might imply that unix domain sockets somehow behaves differently than SOCK_STREAM ones do - on macos. Research is required.

The bug triggers if the custom header is larger than 8100 bytes something. Then only the first piece of the request is sent and nothing more is ever sent. The socket just doesn't seem to signal itself as writable anymore...

@bagder
Copy link
Member Author

@bagder bagder commented Apr 7, 2020

The bug triggers on macOS if the custom header is larger than 8100 bytes something. Then only the first piece of the request is sent and nothing more is ever sent. Clearly the logic for handling a "split" request send is broken, and I believe it is broken on all platforms.

This bug also reproduces on Linux but then at around ~220K so the custom header needs to be made much larger there.

@bagder
Copy link
Member Author

@bagder bagder commented Apr 7, 2020

I arrived at this little patch that lets me write a simple test case for this problem using the regular test suite, which makes it easier to work on a fix:

diff --git a/lib/http.c b/lib/http.c
index 324dd7252..814d69fd2 100644
--- a/lib/http.c
+++ b/lib/http.c
@@ -1227,12 +1227,25 @@ CURLcode Curl_add_buffer_send(Curl_send_buffer **inp,
       return result;
     }
     memcpy(data->state.ulbuf, ptr, sendsize);
     ptr = data->state.ulbuf;
   }
-  else
+  else {
+#ifdef CURLDEBUG
+    /* Allow debug builds override this logic to force short initial sends */
+    char *p = getenv("CURL_SMALLREQSEND");
+    if(p) {
+      size_t altsize = (size_t)strtoul(p, NULL, 10);
+      if(altsize)
+        sendsize = CURLMIN(size, altsize);
+      else
+        sendsize = size;
+    }
+    else
+#endif
     sendsize = size;
+  }
 
   result = Curl_write(conn, sockfd, ptr, sendsize, &amount);
 
   if(!result) {
     /*
@bagder bagder self-assigned this Apr 7, 2020
bagder added a commit that referenced this issue Apr 7, 2020
… off

Fixes #4919
@bagder bagder linked a pull request that will close this issue Apr 7, 2020
@bagder bagder closed this in 0ef54ab Apr 8, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

1 participant
You can’t perform that action at this time.