Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cannot send message larger than 8192 bytes with websocket using unix socket on libcurl 8.10.0 and later, which previously worked #15865

Closed
na-trium-144 opened this issue Dec 30, 2024 · 11 comments
Assignees

Comments

@na-trium-144
Copy link

na-trium-144 commented Dec 30, 2024

I did this

Trying to send large message using websocket via unix socket.

Minimal reproducible example:

#include <curl/curl.h>
#include <iostream>

int main(){
    CURL *handle = curl_easy_init();
    curl_easy_setopt(handle, CURLOPT_VERBOSE, 1L);
    curl_easy_setopt(handle, CURLOPT_URL, "ws://localhost");
    curl_easy_setopt(handle, CURLOPT_UNIX_SOCKET_PATH, "/tmp/webcface/7530.sock");
    curl_easy_setopt(handle, CURLOPT_CONNECT_ONLY, 2L);
    auto ret = curl_easy_perform(handle);
    std::cout << "connect ret=" << ret << std::endl;

    std::string msg(20000, 'a');
    std::size_t sent;
    ret = curl_ws_send(handle, msg.c_str(), msg.size(), &sent, 0, CURLWS_BINARY);
    std::cout << "send ret=" << ret << ", sent=" << sent << "/" << msg.size() << std::endl;

    curl_easy_cleanup(handle);
}

When I run this with libcurl 8.11.1, the trace output says it sent only 8192 bytes, while return value indicates that it sent 19999 bytes (seems to be always message length - 1) and error code is CURLE_AGAIN.
(Meanwhile the server side seems to be receiving nothing.)

* Received 101, switch to WebSocket; mask 325ab3d7
* 0 bytes websocket payload
* Connection #0 to host localhost left intact
connect ret=0
* WS-ENC: sending [BIN payload=0/20000]
* WS-ENC: buffered [BIN payload=20000/20000]
* WS: flushed 8192 bytes
send ret=81, sent=19999/20000

It seems message of 8192 bytes or less (8184 bytes or less in actual message) can be sent without problem.

I expected the following

With libcurl 8.8.0 it had always successfully sent 20000 bytes of message over unix socket.

* Received 101, switch to WebSocket; mask d8c06d35
* 0 bytes websocket payload
* Connection #0 to host localhost left intact
connect ret=0
* WS-ENC: sending [BIN payload=0/20000]
* WS-ENC: buffered [BIN payload=20000/20000]
* WS: flushed 8192 bytes
* WS: flushed 4104 bytes
* WS: flushed 4096 bytes
* WS: flushed 3616 bytes
send ret=0, sent=20000/20000

Additionally, if I use normal tcp connection instead of unix socket (by just removing CURLOPT_UNIX_SOCKET_PATH line), I could send 20000 bytes of message with websocket even with libcurl 8.11.1.

* Received 101, switch to WebSocket; mask 0bbe3615
* 0 bytes websocket payload
* Connection #0 to host localhost left intact
connect ret=0
* WS-ENC: sending [BIN payload=0/20000]
* WS-ENC: buffered [BIN payload=20000/20000]
* WS: flushed 20008 bytes
send ret=0, sent=20000/20000

Is there any breaking change (or bug) about unix socket between 8.8.0 and 8.11.1?

curl/libcurl version

the not working version

curl 8.11.1 (aarch64-apple-darwin23.6.0) libcurl/8.11.1 OpenSSL/3.4.0 (SecureTransport) zlib/1.2.12 brotli/1.1.0 zstd/1.5.6 AppleIDN libssh2/1.11.1 nghttp2/1.64.0 librtmp/2.3
Release-Date: 2024-12-11
Protocols: dict file ftp ftps gopher gophers http https imap imaps ipfs ipns ldap ldaps mqtt pop3 pop3s rtmp rtsp scp sftp smb smbs smtp smtps telnet tftp ws wss
Features: alt-svc AsynchDNS brotli GSS-API HSTS HTTP2 HTTPS-proxy IDN IPv6 Kerberos Largefile libz MultiSSL NTLM SPNEGO SSL threadsafe TLS-SRP UnixSockets zstd

the working version

curl 8.8.0 (aarch64-apple-darwin23.6.0) libcurl/8.8.0 (SecureTransport) OpenSSL/3.4.0 zlib/1.2.12 brotli/1.1.0 zstd/1.5.6 libssh2/1.11.1 nghttp2/1.64.0 librtmp/2.3
Release-Date: 2024-05-22
Protocols: dict file ftp ftps gopher gophers http https imap imaps ipfs ipns ldap ldaps mqtt pop3 pop3s rtmp rtsp scp sftp smb smbs smtp smtps telnet tftp ws wss
Features: alt-svc AsynchDNS brotli GSS-API HSTS HTTP2 HTTPS-proxy IPv6 Kerberos Largefile libz MultiSSL NTLM SPNEGO SSL threadsafe TLS-SRP UnixSockets zstd

operating system

MacOS Sonoma

uname -a:

Darwin kou-MAir.local 23.6.0 Darwin Kernel Version 23.6.0: Mon Jul 29 21:14:21 PDT 2024; root:xnu-10063.141.2~1/RELEASE_ARM64_T8103 arm64
@na-trium-144 na-trium-144 changed the title Cannot send message larger than 8192 bytes with websocket using unix socket on libcurl 8.11.1, which had worked Cannot send message larger than 8192 bytes with websocket using unix socket on libcurl 8.11.1, which previously worked Dec 30, 2024
@na-trium-144
Copy link
Author

I did git bisect and figured out this behavior introduced at 3e64569 (#14458).

@na-trium-144 na-trium-144 changed the title Cannot send message larger than 8192 bytes with websocket using unix socket on libcurl 8.11.1, which previously worked Cannot send message larger than 8192 bytes with websocket using unix socket on libcurl 8.10.0 and later, which previously worked Dec 31, 2024
@na-trium-144
Copy link
Author

na-trium-144 commented Dec 31, 2024

So I think the actual issues are the following:

  • Unlike tcp socket, when using unix socket it only flush 8192 bytes at once. I don't know why but this may be not a bug because libcurl 8.8.0 had the same behavior.
  • Previously curl_ws_send had automatically retried after flushing 8192 bytes. But after websocket, introduce blocking sends #14458 it does not.
  • When the flush is partial, the return value for the number of bytes sent is wrong.
  • How to retry the curl_ws_send then? (I couldn't figure out from the documentation...)

@icing Do you have any idea?

@na-trium-144
Copy link
Author

Additional note: I tested this with Linux and Windows, and it worked without problem there. So only macOS has this issue?

@icing
Copy link
Contributor

icing commented Jan 2, 2025

@na-trium-144 the difference on macOS is probably due to different internal buffers for unix domain sockets, meaning other OS would most likely block on larger sizes.

As to the behaviour you see: curl_ws_send() will return CURLE_AGAIN, when it could not send all data. If will return in the sent parameter how many bytes were written. This will be less than the buflen you passed.

You have then to call curl_ws_send() again with updated buffer and buflen. If you do that in a busy loop or extract the socket and poll that, is up to you.

Does that help?

@na-trium-144
Copy link
Author

@icing Thanks for comment!
The value returned as sent looks wrong for me... because it returns 19999 while the trace says it flushed 8192 bytes. (maybe 'flushed' and 'written' are different things here?)
but anyway I tried calling curl_ws_send again like this:

std::string msg(20000, 'a');
const char *begin = msg.c_str();
const char *current = begin;
const char *end = begin + msg.size();
int i = 0;
do {
    std::size_t sent;
    std::cout << std::endl;
    std::cout << "calling curl_ws_send() #" << ++i
        << ": msg from " << current - begin << " to " << end - begin << std::endl;
    ret = curl_ws_send(handle, current, end - current, &sent, 0, CURLWS_BINARY);
    std::cout << "ret=" << ret << ", sent=" << sent << std::endl;
    current += sent;
    std::this_thread::sleep_for(std::chrono::seconds(1));
} while (ret == CURLE_AGAIN);
calling curl_ws_send() #1: msg from 0 to 20000
* [WS] curl_ws_send(len=20000, fragsize=0, flags=2), raw=0
* [WS] curl_ws_send(len=20000), sendbuf=0 space_left=131070
* WS-ENC: sending [BIN payload=0/20000]
* WS-ENC: buffered [BIN payload=20000/20000]
* WS: flushed 8192 bytes
* [WS] flush EAGAIN, 11816 bytes remain in buffer
* [WS] curl_ws_send(len=20000, fragsize=0, flags=2, raw=0) -> 81, 19999
ret=81, sent=19999

calling curl_ws_send() #2: msg from 19999 to 20000
* [WS] curl_ws_send(len=1, fragsize=0, flags=2), raw=0
* WS: flushed 8192 bytes
* [WS] flush EAGAIN, 3623 bytes remain in buffer
* [WS] curl_ws_send(len=1, fragsize=0, flags=2, raw=0) -> 81, 0
ret=81, sent=0

calling curl_ws_send() #3: msg from 19999 to 20000
* [WS] curl_ws_send(len=1, fragsize=0, flags=2), raw=0
* WS: flushed 3623 bytes
* [WS] curl_ws_send(len=1), sendbuf=0 space_left=131070
* WS-ENC: sending [BIN payload=0/1]
* WS-ENC: buffered [BIN payload=1/1]
* WS: flushed 7 bytes
* [WS] curl_ws_send(len=0, fragsize=0, flags=2, raw=0) -> 0, 1
ret=0, sent=1

It looks working,
But then I found another problem: this code should send 20000 'a's, but the server receives 19999 'a's plus random 1 byte (changes every time I run this).
When the message is less than 8192 bytes (so that it can send it at once,) the message is correct.

icing added a commit to icing/curl that referenced this issue Jan 3, 2025
- add DEBUG env var CURL_WS_CHUNK_EAGAIN to simulate blocking
  aftert a chunk of an encoded websocket frame has been sent.
- update test_20_08 to simulate conditions in curl#15865
@icing
Copy link
Contributor

icing commented Jan 3, 2025

Oh, thanks for all the details. I simulated this in curl's test suite and indeed, the last byte was corrupted. Oh my. I fixed this in #15901 and hope this works for you as well.

If you can build curl from that PR and verify the fix, that would be appreciated!

@na-trium-144
Copy link
Author

I tried with that PR and it's working fine. Thanks!

@jay
Copy link
Member

jay commented Jan 3, 2025

As to the behaviour you see: curl_ws_send() will return CURLE_AGAIN, when it could not send all data. If will return in the sent parameter how many bytes were written. This will be less than the buflen you passed.

You have then to call curl_ws_send() again with updated buffer and buflen. If you do that in a busy loop or extract the socket and poll that, is up to you.

Are you sure about this? It is not documented and is out of character with how send/recv calls work. Why should the caller have to adjust buffer and buflen in the subsequent call? If a recv/send returns an EAGAIN then it would make sense to send the same buffer and length again. That is just how those calls work. As you know. I don't get it

Ref: https://curl.se/libcurl/c/curl_ws_send.html

@icing
Copy link
Contributor

icing commented Jan 3, 2025

I agree that this is not how send/recv ususally works. I think someone needs to fix this.

Update: I implemented sane behaviour for curl_ws_send() in #15901.

icing added a commit to icing/curl that referenced this issue Jan 4, 2025
- add DEBUG env var CURL_WS_CHUNK_EAGAIN to simulate blocking
  aftert a chunk of an encoded websocket frame has been sent.
- update test_20_08 to simulate conditions in curl#15865
@icing
Copy link
Contributor

icing commented Jan 6, 2025

@na-trium-144 In #15901 I now made curl_ws_send() behave sanely. Meaning when only 8192 bytes could be sent, it will return CURLE_OK and send=8192. It will only CURLE_AGAIN when nothing of your payload could be sent (and send will then always be 0).

This should not invalidate the changes you made, but you might want to test this again.

@jay
Copy link
Member

jay commented Jan 16, 2025

Thanks

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Development

No branches or pull requests

4 participants