Hangs when HTTP/2 is used

[jcharaoui]

When HTTP/2 is used between libcurl and the server, HTTPDirFS will hang when requesting a subdirectory. The initial request for the root directory works fine, it seems to hang when trying to list a directory or requesting a file after the initial setup. For example, with CURLOPT_VERBOSE:

path_download(/foo, 0-4096);
* Expire in 0 ms for 6 (transfer 0x7f0788005500)
* Expire in 15000 ms for 2 (transfer 0x7f0788005500)
* 24 bytes stray data read before trying h2 connection
* 24 bytes stray data read before trying h2 connection
* 24 bytes stray data read before trying h2 connection
* 24 bytes stray data read before trying h2 connection
* 24 bytes stray data read before trying h2 connection
* 24 bytes stray data read before trying h2 connection
* 24 bytes stray data read before trying h2 connection
* Found bundle for host lib3.net: 0x561def466ad0 [can multiplex]
* Connection 0 seems to be dead!

The problems doesn't occur when using HTTP/1.1.

[fangfufu]

Have you got a server that I can test with?

[jcharaoui]

@fangfufu Here: https://lib3.net/httpdirfs

Enabling CURLOPT_FORBID_REUSE prevents the problem from occuring.

[fangfufu]

I am not a big fan of CURLOPT_FORBID_REUSE, as disallowing connection reuse causes performance issues. I now set CURLOPT_HTTP_VERSION to CURL_HTTP_VERSION_1_1.

Fixed in 4491377.

[jcharaoui]

Hrm. A forced protocol downgrade seems like a harsh workaround...

Isn't there some way we could just properly close all open connections after each filesystem operation?

[fangfufu]

Right now network connections are not coupled to filesystem operations. I think it is better performance to allow libcurl to reuse existing connections for the next filesystem operation, rather than opening and closing a connection after each operation.

I have absolutely no idea why I can't use HTTP2. I don't know if it is related to this:
https://curl.haxx.se/mail/lib-2015-11/0107.html

[fangfufu]

I think this is definitely a problem with libcurl, or maybe a misconfiguration of libcurl.

I created a dev branch, with HTTP2 and HTTP2 multiplexing enabled. It gets stuck at the following line:
https://github.com/fangfufu/httpdirfs/blob/dev/src/network.c#L159

I have no idea why it gets stuck in curl_multi_perform() and how to diagnose it.

[jcharaoui]

Let's push the protocol downgrade as a workaround for the next version, but keep the HTTP/2 branch for debugging. It might be worth it to dig deeper into the documentation or check with libcurl developpers.

[fangfufu]

I tried setting connection timeout to 1 sec, it doesn't help. I think this might needs some sort of packet sniffing, or at least check externally to see if the connection is still open.

path_download(/foo, 0-4096);
* Expire in 0 ms for 6 (transfer 0x7fd8b4002c90)
pthread_mutex_lock(&transfer_lock)
curl_multi_perform start
* Expire in 1000 ms for 2 (transfer 0x7fd8b4002c90)
* 24 bytes stray data read before trying h2 connection
* 24 bytes stray data read before trying h2 connection
* Found bundle for host lib3.net: 0x560791c06ae0 [can multiplex]
* Connection 0 seems to be dead!

Update: This seems to occasionally help.

[jcharaoui]

I tried building with newer libcurl, 7.65.1 but there was no improvement.

[fangfufu]

Setting CURLOPT_CONNECTUTIMEOUT to 1 seems to help sporadically.

[fangfufu]

By the way, this has got nothing to do with multithreading - running HTTPDirFS in single thread mode does not help with this problem.

[fangfufu]

Okay, I figured out why now. I use curl's share interface to share as many things as possible between curl handles. It looks like the flag CURL_LOCK_DATA_CONNECT makes it crash. This flag "put the connection cache in the share object and make all easy handles using this share object share the connection cache." [1]

Since we are using curl multi interface, it is not necessary to configure CURL_LOCK_DATA_CONNECT to share connections, furthermore, according to "libcurl does not support doing HTTP/2 streams in different threads using a shared connection". [1]

Finally I cannot get [2] to work at all, even after setting CURL_HTTP_VERSION_1_1 and use HTTP-only link. So I think the best solution is not setting CURL_LOCK_DATA_CONNECT. This allows HTTP2 to be used.

Fixed in ed37aa5