Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add xfer_buf to multi handle #12805

Closed
wants to merge 7 commits into from
Closed

Add xfer_buf to multi handle #12805

wants to merge 7 commits into from

Conversation

icing
Copy link
Contributor

@icing icing commented Jan 26, 2024

  • can be borrowed by transfer during recv-write operation
  • needs to be released before borrowing again
  • adjustis size to data->set.buffer_size
  • used in transfer.c readwrite_data()

@bagder
Copy link
Member

bagder commented Jan 27, 2024

Shouldn't this also remove the buffer from the easy handle?

@icing
Copy link
Contributor Author

icing commented Jan 27, 2024

Shouldn't this also remove the buffer from the easy handle?

When I rebase on the other merges...

@icing
Copy link
Contributor Author

icing commented Jan 27, 2024

now with the download buffer removed everywhere. Curious if this survives the torture tests, because conncache shutdown might now try to allocate the xfer_buf, if it had not been used before.

@bagder
Copy link
Member

bagder commented Jan 27, 2024

conncache shutdown might now try to allocate the xfer_buf, if it had not been used before.

Yes that sounds possible. Like if you only do curl -I ftp://host, as then it never actually did a transfer but the connection will be put in the cache.

But;: this is a transfer buffer now. Surely the connection shutdown no longer needs the buffer?

@icing
Copy link
Contributor Author

icing commented Jan 27, 2024

conncache shutdown might now try to allocate the xfer_buf, if it had not been used before.

Yes that sounds possible. Like if you only do curl -I ftp://host, as then it never actually did a transfer but the connection will be put in the cache.

But;: this is a transfer buffer now. Surely the connection shutdown no longer needs the buffer?

True, good point.

lib/conncache.c Outdated Show resolved Hide resolved
lib/conncache.c Outdated Show resolved Hide resolved
lib/multi.c Outdated Show resolved Hide resolved
@bagder
Copy link
Member

bagder commented Jan 28, 2024

First: I want to wait with merging this until after 8.6.0. For safety.

Then:

I was thinking: before this PR, the download buffer is only allocated while in use. It is freed again from the easy handle after the transfer is complete (in the DONE state), which makes keeping N easy handles around not use much memory, even if those handles are used with the easy interface.

This PR puts the freeing of the buffer back to when the easy handle is cleaned up, making easy handles use much more memory post a completed transfer.

Do you think it would make sense to have the multi handle free the download buffer once there is no transfer in progress anymore? It would risk having to re-alloc it again immediately when new transfers are added, but it would make the easy interface use much more like before and might even make sense for the multi interface use cases too: simply use less memory when no transfers are running.

@icing
Copy link
Contributor Author

icing commented Jan 29, 2024

First: I want to wait with merging this until after 8.6.0. For safety.

Agree.

Then:

I was thinking: before this PR, the download buffer is only allocated while in use. It is freed again from the easy handle after the transfer is complete (in the DONE state), which makes keeping N easy handles around not use much memory, even if those handles are used with the easy interface.

This PR puts the freeing of the buffer back to when the easy handle is cleaned up, making easy handles use much more memory post a completed transfer.

Do you think it would make sense to have the multi handle free the download buffer once there is no transfer in progress anymore? It would risk having to re-alloc it again immediately when new transfers are added, but it would make the easy interface use much more like before and might even make sense for the multi interface use cases too: simply use less memory when no transfers are running.

We could, when entering DONE for a transfer, check if there are then no more active transfers at the multi handle and then free the buffer. That would put the memory situation for single, easy transfers back to how it was before.

@icing
Copy link
Contributor Author

icing commented Jan 29, 2024

Added xfer_buf free when multi_done() detects there aren't any active transfers any more.

lib/multi.c Outdated Show resolved Hide resolved
@bagder

This comment was marked as outdated.

@bagder
Copy link
Member

bagder commented Feb 6, 2024

Tweaked:

diff --git a/docs/libcurl/opts/CURLOPT_BUFFERSIZE.md b/docs/libcurl/opts/CURLOPT_BUFFERSIZE.md
index 1faebeef54..b4759ea653 100644
--- a/docs/libcurl/opts/CURLOPT_BUFFERSIZE.md
+++ b/docs/libcurl/opts/CURLOPT_BUFFERSIZE.md
@@ -40,10 +40,15 @@ buffer size allowed to be set is 1024.
 DO NOT set this option on a handle that is currently used for an active
 transfer as that may lead to unintended consequences.
 
 The maximum size was 512kB until 7.88.0.
 
+Starting in libcurl 8.7.0, there is just a single transfer buffer allocated
+per multi handle. This buffer is used by all easy handles added to a multi
+handle no matter how many parallel transfers there are. The buffer remains
+allocated as long as there are active transfers.
+
 # DEFAULT
 
 CURL_MAX_WRITE_SIZE (16kB)
 
 # PROTOCOLS

@icing
Copy link
Contributor Author

icing commented Feb 6, 2024

Added this, thanks! Also rebased without problems.

@icing icing changed the title Add xfer_buf to multi handle Add xfer_buf to multi handle Feb 9, 2024
@bagder bagder closed this in 476adfe Feb 9, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging this pull request may close these issues.

2 participants