Libcurl does not seem to send user agent reliably #6312
Comments
Please tell us how to reproduce this. And if any of the details in that forum thread is important for the issue, I would appreciate if you would include them in the issue here. |
I am unfortunately unable to specify exact steps to reproduce, as the tracker specifics are not known to me. As a result,I can only assume that this is a common issue with any tracker, possibly with HTTPS enabled. I am aware that this makes debugging and possibly fixing the issue a lot harder than it should be. In general:
|
That seems like a very far-going assumption that I can't see any evidence for. We need a recipe for this issue in order to conclude that this is in fact a libcurl problem and not just a side effect of something in the application code. libcurl is after all used by a large number of applications and we've had no others report an issue like this during the half year 7.71.0 has been out. The fact that no user-agent appears when you run the application is not a sure sign of a libcurl issue... |
You are right, it is not guaranteed, but the fact that a Curl upgrade specifically causes this issue makes it likely that it was indeed a change in libcurl in 7.71.0. Transmission 2.94 was released on the 1st of May, 2018. This release and the subsequent 3.0 version (released on the 22nd of May 2020) work with Arch Linux's core/curl 7.70.0 and do not have issues when announcing. Supposing that there were some API changes for lib curl in 7.71.0 and above, and Transmission is not using the lib curl API as expected, so the user agent is not set properly and it is indeed an application side effect as suggested, I believe that this is something that should be handled as it might break backward compatibility. |
The sending of and setting user-agent has not changed in many years (maybe 15?), which is partly why I have a hard time to suspect thhat this is a libcurl regression. But I've been wrong before. |
I can't think of any other report like this. You could bisect. |
I don't think we can do anything about this issue without either a way to reproduce ourselves, or a very detailed description of exactly what the app does (libcurl-wise) when it fails. |
I have raised an issue for the Transmission client to request more information, as I believe they can provide details on how libcurl is used in the background. |
Can confirm that I'm experiencing the same issue on Arch Linux ARM and Transmission. Issue seems to be present in:
Downgrading to curl-7.67.0-3-armv7h.pkg.tar.xz (SHA-1: cc5d4f89e05a945a74bfce36f84786eb9e68e64d) seems to be a workaround. I was unable to test other versions as those were the only ones present in the pacman cache in my machine. |
+1 to confirm this is an issue. I'm getting intermittent "banned client" errors due to this. Edit: Using rTorrent:0.9.6/0.13.6 |
Adding more "me toos" to this issue doesn't help. We need more details. For example, show us the full HTTP requests done with a working libcurl and one with the problematic request? Perhaps even going so far to bisect to the exact libcurl change that introduced the problem. No other users report these problems and I libcurl is used quite widely. |
As seen in the Transmission issue thread, this problem is also not easily reproducible for all Transmission users or authors so apparently it's not just a libcurl upgrade that's needed to trigger this. |
I've also been experiencing this issue since 7.71. I used mitmproxy to look at the requests transmission makes. I've only tried with trackers using TLS. I started transmission after editing the service by adding:
After which I downloaded some ubuntu images from: https://torrent.ubuntu.com/tracker_index After that I changed the tracker to: https://tracker.nanoha.org:443/announce I've tried this with the ubuntu tracker, which all send a user-agent even if they announce at the same time. Now I'm not sure if this is intended/supposed to happen with HTTP/2.0. Is curl supposed to send a user-agent with each request, or is the server (using HTTP/2.0) supposed to know the user-agent is the same for all requests? I don't know enough about the specifications. I could try a git bisect sometime later, but a quick search in the commits between curl-7_70_0 and curl-7_71_0 did show this issue which might be related? #5567 |
Ah yes, I had forgot about that change that indeed was introduced in 7.71.0. That change fixed a user-agent issue that otherwise could make curl use the same user-agent for multiple requests on the same connection as the agent would (wrongly) be associated with the connection and not with the particular individual transfers that could be using the same connection (when multiplexed like with HTTP/2). This could thus make a mistake in the client-code using libcurl to change behavior. This could also explain why Transmission is still the only application that shows this problem half a year after that libcurl version was released.
libcurl sends no user-agent at all by default and HTTP doesn't mandate that one is used. It is an optional header. If the user-agent is set for a transfer in libcurl with the |
Thanks for the quick response :-) Looking at this comment it appears to also be affecting rTorrent. I've tried the code from this comment, Which is supposedly how transmission uses curl. I adjusted it by setting: curl_easy_setopt(e, CURLOPT_URL, "https://tracker.nanoha.org:443/announce");
curl_easy_setopt(e, CURLOPT_CAINFO, "/home/arjan/.mitmproxy/mitmproxy-ca-cert.pem");
curl_easy_setopt(e, CURLOPT_PROXY, "127.0.0.1:8080"); and set The first 11 requests had a user-agent, the rest didn't. However, I'm not familiar enough with this to know if it's an implementation issue or not. I can imagine it's not up to you to debug the code of others. If I should continue this in the transmission issue let me know. |
If you have a way to reproduce that sounds like an excellent step no matter what!
|
|
I tried the code from that comment (adjusted to init CURL_GLOBAL_ALL) and let it run for 15 minutes but nothing happened. The python server is HTTP/1.0. I then changed to https://http2.golang.org/ECHO and it showed up pretty quickly in the verbose output:
|
Woah! What code is that exactly? Can you make zip for me to try? |
Here is a reproducer: package main
import (
"log"
"net/http"
)
func main() {
srv := &http.Server{Addr: ":8000", Handler: http.HandlerFunc(handle)}
log.Printf("Serving on https://0.0.0.0:8000")
log.Fatal(srv.ListenAndServeTLS("cert.pem", "key.pem"))
}
func handle(w http.ResponseWriter, r *http.Request) {
log.Printf("UserAgent: %s", r.UserAgent())
if (r.UserAgent() != "foobar/1.0") {
log.Fatalf("Unexpected UserAgent")
}
w.Write([]byte("dummy"))
}
|
Thanks, that's neat, but I was curious about the code for the client side... |
I've used the same client code test-https.c from my comment transmission/transmission#1563 (comment). |
Ah, I missed that part. Thanks, will run ahead and try this now! |
It reproduces for me too. |
Thanks for looking into it! |
It is a libcurl flaw and I know why it happens. Working on a fix. |
... and not in the connection setup, as for multiplexed transfers the connection setup might be skipped and then the transfer would end up without the set user-agent! Reported-by: Flameborn on github Assisted-by: Andrey Gursky Assisted-by: Jay Satiro Fixes #6312
Please try out #6417 if you can build a local libcurl. It is a fairly small patch. It fixes the problem in my tests. |
I've just built current git master, which fails the test. After applying the patch I've let the test running for 7 minutes without facing the bug. Looks good, thanks! |
Glad @jay and @andreygursky helped with making it reproduce-able. It would probably take me a bit longer. I hope my info helped speed things up. I've build curl with the fix and can confirm the issue is now gone (even with transmission). Thanks @bagder |
I can confirm. The issue is gone here as well when using the patched build.
Thank you very much for investigating this and for the quick fix.
… On 2021. Jan 5., at 16:21, Arjan ***@***.***> wrote:
Glad @jay <https://github.com/jay> and @andreygursky <https://github.com/andreygursky> helped with making it reproduce-able. It would probably take me a bit longer. I hope my info helped speed things up.
I've build curl with the fix and can confirm the issue is now gone (even with transmission). Thanks @bagder <https://github.com/bagder>
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub <#6312 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAHLD4GSEYARUAKVGRHSMY3SYMVBFANCNFSM4UYIGM7Q>.
|
Thank you everyone! The fix has now landed in master. |
Transmission does not always send a user agent string when announcing to a torrent tracker.
Even when specified in its config file, the sent user agent string is empty.
This is after upgrading to Curl (libcurl) 7.71.0 and above, using Transmission 2.94 and 3.0. Older Curl versions work as expected.
See a forum thread here: https://forum.transmissionbt.com/viewtopic.php?f=2&t=20449&sid=5ab5f6c305ead5a64dbd72b0608cf729
I expected the following
User agent sent when announcing.
curl/libcurl version
Curl 7.71.0 and above.
operating system
Arch Linux ARM (Linux idris 5.4.81-1-ARCH #1 SMP PREEMPT Mon Dec 7 20:38:46 UTC 2020 armv7l GNU/Linux)
The text was updated successfully, but these errors were encountered: