Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add portable way to unblock/alert blocking curl_multi_wait from another thread #4418

Closed
pps83 opened this issue Sep 25, 2019 · 6 comments

Comments

@pps83
Copy link
Contributor

@pps83 pps83 commented Sep 25, 2019

I'm very happy with curl :) but the way I implemented http code in our project is by using 1ms busy wait worker thread that doesn't even use curl_multi_wait. It simply sleeps(1ms) and then does curl_multi_perform. One of the reasons is because I wasn't able to (or rather didn't want to) implement portable code that would be able to "unblock" curl_multi_wait call if I needed to. On non-windows systems generally it's a simple task: an anonymous pipe can be used to alert curl_multi_wait. On windows however, this won't work as select on windows requires socket to be passed. It would be helpful if libcurl instead provided through some api a way to unblock curl_multi_wait call from another thread (perhaps internally it already has something like that?).
On non-windows systems it may implement it as anonymous pipe. On windows it may use self-connected udp socket (this way nobody on localhost would be able to send data to it).

@bagder

This comment has been minimized.

Copy link
Member

@bagder bagder commented Sep 25, 2019

Yes it would be useful and I'm sure appreciated by other users too. No, libcurl doesn't have anything like that internally already.

@pps83

This comment has been minimized.

Copy link
Contributor Author

@pps83 pps83 commented Sep 25, 2019

Somewhat related quick question: I use CURLOPT_CONNECT_ONLY (for websockets) and when I curl_easy_cleanup the handle (after it was remove from multi etc) tcp connection doesn't seem to get closed (according to wireshark). In my case app runs for longer time and I enabled TCP keepalives and a minute after I performed all closing I see keepalive messages in wireshark (1122 is websocket server in nodejs that replies with websocket-close and closes connection):

image

Is it curl bug, or I need to dome something else to close connection?

@pps83

This comment has been minimized.

Copy link
Contributor Author

@pps83 pps83 commented Sep 25, 2019

I tried to set CURLOPT_FORBID_REUSE and CURLOPT_CONNECT_ONLY, but in this case nothing works: after connect succeeds I cannot make any sends, curl_easy_send weirdly returns CURLE_UNSUPPORTED_PROTOCOL. I use 7.64.1

(when we tried to switch to 7.65.3 we had catastrophic TLS 5x-10x overload on our load balancers and it was not related to connection max age and wasn't related to that bug that divided max age time twice. We had to roll back as an emergency for now)

@bagder

This comment has been minimized.

Copy link
Member

@bagder bagder commented Sep 25, 2019

That sounds like a totally different issue than "unblocking" curl_multi_wait from another thread. If this is a bug, please file this as a separate issue with details to allow us to reproduce.

@bagder

This comment has been minimized.

Copy link
Member

@bagder bagder commented Oct 10, 2019

I've jotted down some ideas for curl_multi_unblock in the wiki. I'm interested in feedback and comments!

@jay

This comment has been minimized.

Copy link
Member

@jay jay commented Oct 10, 2019

I think an interesting issue is whether a socket always receiving data with received data like in curl_multi_perform would cause an unblock, like can curl_multi_perform be always out (I think I asked about this before my read of the code was unclear on it) and if there is data pending should it return. So say you want to add some URLs to the multi, in your loop you've got a lock free list or something that is checked each time multi perform returns, well you want those transfers added right away, so you in the other thread you add them then you can call unblock?

@bagder bagder closed this in cba52e2 Nov 8, 2019
ngg added a commit to tresorit/curl that referenced this issue Nov 17, 2019
This commit adds curl_multi_unblock() which was previously in the TODO
list. To enable the functionality a new setopt option
CURLMOPT_ENABLE_UNBLOCK has been also added.

On some platforms and with some configurations this feature might not be
available or can fail, in these cases a new error code
(CURLM_UNBLOCK_FAILURE) is returned from curl_multi_setopt() or from
curl_multi_unblock().

Fixes curl#4418
ngg added a commit to tresorit/curl that referenced this issue Nov 17, 2019
This commit adds curl_multi_unblock() which was previously in the TODO
list. To enable the functionality a new setopt option
CURLMOPT_ENABLE_UNBLOCK has been also added.

On some platforms and with some configurations this feature might not be
available or can fail, in these cases a new error code
(CURLM_UNBLOCK_FAILURE) is returned from curl_multi_setopt() or from
curl_multi_unblock().

Fixes curl#4418
ngg added a commit to tresorit/curl that referenced this issue Nov 18, 2019
This commit adds curl_multi_unblock() which was previously in the TODO
list. To enable the functionality a new setopt option
CURLMOPT_ENABLE_UNBLOCK has been also added.

On some platforms and with some configurations this feature might not be
available or can fail, in these cases a new error code
(CURLM_UNBLOCK_FAILURE) is returned from curl_multi_setopt() or from
curl_multi_unblock().

Fixes curl#4418
ngg added a commit to tresorit/curl that referenced this issue Nov 18, 2019
This commit adds curl_multi_unblock() which was previously in the TODO
list. To enable the functionality a new setopt option
CURLMOPT_ENABLE_UNBLOCK has been also added.

On some platforms and with some configurations this feature might not be
available or can fail, in these cases a new error code
(CURLM_UNBLOCK_FAILURE) is returned from curl_multi_setopt() or from
curl_multi_unblock().

Fixes curl#4418
ngg added a commit to tresorit/curl that referenced this issue Nov 18, 2019
This commit adds curl_multi_unblock() which was previously in the TODO
list. To enable the functionality a new setopt option
CURLMOPT_ENABLE_UNBLOCK has been also added.

On some platforms and with some configurations this feature might not be
available or can fail, in these cases a new error code
(CURLM_UNBLOCK_FAILURE) is returned from curl_multi_setopt() or from
curl_multi_unblock().

Fixes curl#4418
ngg added a commit to tresorit/curl that referenced this issue Nov 18, 2019
This commit adds curl_multi_unblock() which was previously in the TODO
list.

On some platforms and with some configurations this feature might not be
available or can fail, in these cases a new error code
(CURLM_UNBLOCK_FAILURE) is returned from curl_multi_unblock().

Fixes curl#4418
ngg added a commit to tresorit/curl that referenced this issue Nov 19, 2019
This commit adds curl_multi_unblock() which was previously in the TODO
list.

On some platforms and with some configurations this feature might not be
available or can fail, in these cases a new error code
(CURLM_UNBLOCK_FAILURE) is returned from curl_multi_unblock().

Fixes curl#4418
ngg added a commit to tresorit/curl that referenced this issue Nov 20, 2019
This commit adds curl_multi_unblock() which was previously in the TODO
list.

On some platforms and with some configurations this feature might not be
available or can fail, in these cases a new error code
(CURLM_UNBLOCK_FAILURE) is returned from curl_multi_unblock().

Fixes curl#4418
ngg added a commit to tresorit/curl that referenced this issue Nov 20, 2019
This commit adds curl_multi_unblock() which was previously in the TODO
list.

On some platforms and with some configurations this feature might not be
available or can fail, in these cases a new error code
(CURLM_UNBLOCK_FAILURE) is returned from curl_multi_unblock().

Fixes curl#4418
ngg added a commit to tresorit/curl that referenced this issue Nov 20, 2019
This commit adds curl_multi_unblock() which was previously in the TODO
list.

On some platforms and with some configurations this feature might not be
available or can fail, in these cases a new error code
(CURLM_UNBLOCK_FAILURE) is returned from curl_multi_unblock().

Fixes curl#4418
ngg added a commit to tresorit/curl that referenced this issue Nov 21, 2019
This commit adds curl_multi_unblock() which was previously in the TODO
list.

On some platforms and with some configurations this feature might not be
available or can fail, in these cases a new error code
(CURLM_UNBLOCK_FAILURE) is returned from curl_multi_unblock().

Fixes curl#4418
ngg added a commit to tresorit/curl that referenced this issue Nov 21, 2019
This commit adds curl_multi_unblock() which was previously in the TODO
list.

On some platforms and with some configurations this feature might not be
available or can fail, in these cases a new error code
(CURLM_UNBLOCK_FAILURE) is returned from curl_multi_unblock().

Fixes curl#4418
ngg added a commit to tresorit/curl that referenced this issue Nov 22, 2019
This commit adds curl_multi_unblock() which was previously in the TODO
list.

On some platforms and with some configurations this feature might not be
available or can fail, in these cases a new error code
(CURLM_UNBLOCK_FAILURE) is returned from curl_multi_unblock().

Fixes curl#4418
ngg added a commit to tresorit/curl that referenced this issue Nov 22, 2019
This commit adds curl_multi_unblock() which was previously in the TODO
list.

On some platforms and with some configurations this feature might not be
available or can fail, in these cases a new error code
(CURLM_UNBLOCK_FAILURE) is returned from curl_multi_unblock().

Fixes curl#4418
ngg added a commit to tresorit/curl that referenced this issue Nov 22, 2019
This commit adds curl_multi_unblock() which was previously in the TODO
list.

On some platforms and with some configurations this feature might not be
available or can fail, in these cases a new error code
(CURLM_UNBLOCK_FAILURE) is returned from curl_multi_unblock().

Fixes curl#4418
ngg added a commit to tresorit/curl that referenced this issue Nov 22, 2019
This commit adds curl_multi_unblock() which was previously in the TODO
list.

On some platforms and with some configurations this feature might not be
available or can fail, in these cases a new error code
(CURLM_UNBLOCK_FAILURE) is returned from curl_multi_unblock().

Fixes curl#4418
ngg added a commit to tresorit/curl that referenced this issue Nov 22, 2019
This commit adds curl_multi_unblock() which was previously in the TODO
list.

On some platforms and with some configurations this feature might not be
available or can fail, in these cases a new error code
(CURLM_UNBLOCK_FAILURE) is returned from curl_multi_unblock().

Fixes curl#4418
ngg added a commit to tresorit/curl that referenced this issue Nov 23, 2019
This commit adds curl_multi_unblock() which was previously in the TODO
list.

On some platforms and with some configurations this feature might not be
available or can fail, in these cases a new error code
(CURLM_UNBLOCK_FAILURE) is returned from curl_multi_unblock().

Fixes curl#4418
ngg added a commit to tresorit/curl that referenced this issue Nov 23, 2019
This commit adds curl_multi_wakeup() which was previously in the TODO
list under the curl_multi_unblock name.

On some platforms and with some configurations this feature might not be
available or can fail, in these cases a new error code
(CURLM_WAKEUP_FAILURE) is returned from curl_multi_wakeup().

Fixes curl#4418
ngg added a commit to tresorit/curl that referenced this issue Nov 23, 2019
This commit adds curl_multi_wakeup() which was previously in the TODO
list under the curl_multi_unblock name.

On some platforms and with some configurations this feature might not be
available or can fail, in these cases a new error code
(CURLM_WAKEUP_FAILURE) is returned from curl_multi_wakeup().

Fixes curl#4418
ngg added a commit to tresorit/curl that referenced this issue Nov 23, 2019
This commit adds curl_multi_wakeup() which was previously in the TODO
list under the curl_multi_unblock name.

On some platforms and with some configurations this feature might not be
available or can fail, in these cases a new error code
(CURLM_WAKEUP_FAILURE) is returned from curl_multi_wakeup().

Fixes curl#4418
ngg added a commit to tresorit/curl that referenced this issue Nov 24, 2019
This commit adds curl_multi_wakeup() which was previously in the TODO
list under the curl_multi_unblock name.

On some platforms and with some configurations this feature might not be
available or can fail, in these cases a new error code
(CURLM_WAKEUP_FAILURE) is returned from curl_multi_wakeup().

Fixes curl#4418
ngg added a commit to tresorit/curl that referenced this issue Nov 24, 2019
This commit adds curl_multi_wakeup() which was previously in the TODO
list under the curl_multi_unblock name.

On some platforms and with some configurations this feature might not be
available or can fail, in these cases a new error code
(CURLM_WAKEUP_FAILURE) is returned from curl_multi_wakeup().

Fixes curl#4418
ngg added a commit to tresorit/curl that referenced this issue Nov 24, 2019
This commit adds curl_multi_wakeup() which was previously in the TODO
list under the curl_multi_unblock name.

On some platforms and with some configurations this feature might not be
available or can fail, in these cases a new error code
(CURLM_WAKEUP_FAILURE) is returned from curl_multi_wakeup().

Fixes curl#4418
ngg added a commit to tresorit/curl that referenced this issue Nov 25, 2019
This commit adds curl_multi_wakeup() which was previously in the TODO
list under the curl_multi_unblock name.

On some platforms and with some configurations this feature might not be
available or can fail, in these cases a new error code
(CURLM_WAKEUP_FAILURE) is returned from curl_multi_wakeup().

Fixes curl#4418
bagder added a commit that referenced this issue Nov 25, 2019
This commit adds curl_multi_wakeup() which was previously in the TODO
list under the curl_multi_unblock name.

On some platforms and with some configurations this feature might not be
available or can fail, in these cases a new error code
(CURLM_WAKEUP_FAILURE) is returned from curl_multi_wakeup().

Fixes #4418
Closes #4608
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.