Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

How to get the list of URLs that have not been executed yet when I use MultiCurl::stop() #802

Closed
simp-lee opened this issue Jul 8, 2023 · 3 comments
Labels

Comments

@simp-lee
Copy link

simp-lee commented Jul 8, 2023

Question 1:
How to get the list of URLs that have not been executed yet when I use the $multi_curl->stop() method like below?

// $multi_curl->getFailedUrls();

// URLs that need to be retrieved.
['https://httpbin.org/status/error_url_1']
['https://httpbin.org/status/error_url_2']
['https://httpbin.org/status/pending_url_1']
['https://httpbin.org/status/pending_url_2']
$multi_curl = new MultiCurl();
$multi_curl->error(function ($instance) use ($multi_curl) {
    $multi_curl->stop();
});

// Assuming failed URLs
$multi_curl->addGet('https://httpbin.org/status/error_url_1');
$multi_curl->addGet('https://httpbin.org/status/error_url_2');

// Assuming successful URLs
$multi_curl->addGet('https://httpbin.org/status/successful_url_1');
$multi_curl->addGet('https://httpbin.org/status/successful_url_2');

// Pending URLs (that have not been executed yet).
$multi_curl->addGet('https://httpbin.org/status/pending_url_1');
$multi_curl->addGet('https://httpbin.org/status/pending_url_2');

$multi_curl->start();

Question 2:
How can I ensure that the required URLs (urls_need_to_be_executed) are executed completely,
and stop the execution once all the sub-URLs derived from these URLs (urls_need_to_be_executed) have been executed?
When I use the following approach and place the "stop" method after "$multi_curl->addGet('pending_url_1') ...", the program only executes one data entry and immediately stops without executing any of the sub-URLs(pending_url_1, pending_url_2).

$response = [];
$multi_curl ->success(function ($instance) use (&$response, $multi_curl){
    // I can retrieve the list of URLs that will be executed next by using $instance->response. 
    // Then, I add these URLs to the execution queue using the addGet method.
    // Once all the URLs in the list have been executed, stop the execution.

    $multi_curl->addGet('pending_url_1');
    $multi_curl->addGet('pending_url_2');
    // more urls to be executed...

    $multi_curl->stop();
});

$multi_curl->addGet('url_1_need_to_be_executed');
$multi_curl->addGet('url_2_need_to_be_executed');
$multi_curl->addGet('url_3_need_to_be_executed');
// I want these URLs to be executed completely before $multi_curl->stop().
// Because these URL tasks are intended to generate subtask links in the $multi_curl->success() method.

$multi_curl->start();
@zachborboa
Copy link
Member

zachborboa commented Jul 15, 2023

@zachborboa
Copy link
Member

For question 2, calling Curl::stop() inside the success callback can be removed.

Without removing the stop() call as in the example, the addGet() requests inside the success callback are queued, but the call to stop() immediately attempts to stop queued requests and active requests.

With the Curl::stop() inside the success callback removed:

$response = [];
$multi_curl ->success(function ($instance) use (&$response, $multi_curl){
    $multi_curl->addGet('pending_url_1');
    $multi_curl->addGet('pending_url_2');
    // more urls to be executed...
});

$multi_curl->addGet('url_1_need_to_be_executed');
$multi_curl->addGet('url_2_need_to_be_executed');
$multi_curl->addGet('url_3_need_to_be_executed');

$multi_curl->start();

@simp-lee
Copy link
Author

Thank you very much for your help.

Regarding the first question, your examples are written very clearly and are highly applicable. Well done!

As for the second question, I overcomplicated things, and you are right - simply removing the stop() call will do.

Thank you again for your assistance.
It's a fantastic class.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Development

No branches or pull requests

2 participants