Join GitHub today
GitHub is home to over 40 million developers working together to host and review code, manage projects, and build software together.
Sign upcurl: support parallel transfers #3804
Conversation
This comment has been minimized.
This comment has been minimized.
The codacy warnings are just silly. |
This comment has been minimized.
This comment has been minimized.
You can turn that one off via Code patterns -> Flawfinder. |
This comment has been minimized.
This comment has been minimized.
I can? I can't find anywhere to do that! |
This comment has been minimized.
This comment has been minimized.
After "Login with GitHub", "Code patterns" appears at the menu on the left. |
This comment has been minimized.
This comment has been minimized.
I would just use --parallel and nix the short option, this seems like use will be almost always in scripts. I don't see anyone at the command line running
|
This comment has been minimized.
This comment has been minimized.
I had one unreproducible hang while I was playing around with this, I'm not sure what caused it but during the investigation I found an off-by-one bug so I'm going to attribute it to that unless it happens again.
That resource no longer exists, so I expect the tool to terminate almost immediately. The first time it didn't but every time after that it did (regardless of the fix). |
This comment has been minimized.
This comment has been minimized.
thanks for the fixes @jay! |
This comment has been minimized.
This comment has been minimized.
jumoog
commented
Apr 29, 2019
I saw your YouTube video.. you can do a ETA if you request the file size of all urls first |
This comment has been minimized.
This comment has been minimized.
No, I can't. Primarily, lots of transfers libcurl can do simply do not know the size before-hand no matter what we do. |
This comment has been minimized.
This comment has been minimized.
I've now moved a few of the remaining action items to get them done after the first merge, so that we can do the development of those things more parallel and with cooperation. As long as the non-parallel use cases still work exactly like before and are stable it shouldn't be a problem to proceed like this. |
This comment has been minimized.
This comment has been minimized.
My --write-outs are being overwritten by the progress meter. Simple example:
In parallel_transfers after a transfer finishes post_transfer is called for that transfer, which does the write-out. However after that progress meter is called again which overwrites the line. Lines 2011 to 2040 in 80f044f
|
This comment has been minimized.
This comment has been minimized.
Any suggestion on what to do with |
This comment has been minimized.
This comment has been minimized.
I'm not sure. Delay until finished? |
This comment has been minimized.
This comment has been minimized.
I suppose it goes into the whole "how to show what happened for each transfer" discussion too. If you do 20 transfers, how are those |
This comment has been minimized.
This comment has been minimized.
If they're done in the order they were given they could make some sense. Or there could be a transfer id like "%{transfer_id}" |
834a55a
to
4c082ef
This is done by making sure each individual transfer is first added to a linked list as then they can be performed serially, or at will, in parallel.
This comment has been minimized.
This comment has been minimized.
I intend to merge this within shortly. |
This comment has been minimized.
This comment has been minimized.
This broke the autobuilds. Unfortunately, it's not easy to see why in this huge diff:
|
easysrc_cleanup is only defined when CURL_DISABLE_LIBCURL_OPTION is defined, and prior to this change would be called regardless. Bug: curl#3804 (comment) Reported-by: Marcel Raad Closes #xxxx
This comment has been minimized.
This comment has been minimized.
|
easysrc_cleanup is only defined when CURL_DISABLE_LIBCURL_OPTION is not defined, and prior to this change would be called regardless. Bug: #3804 (comment) Reported-by: Marcel Raad Closes #4142
easysrc_cleanup is only defined when CURL_DISABLE_LIBCURL_OPTION is not defined, and prior to this change would be called regardless. Bug: curl#3804 (comment) Reported-by: Marcel Raad Closes curl#4142
bagder commentedApr 24, 2019
•
edited
This is done by making sure each individual transfer is first added to a linked list as then they can be performed serially, or at will, in parallel.
Status
-Z
/--parallel
)--parallel-max
)--retry
still works for serial transfersStill TODO post first merge
--retry
work for parallel