403 URLs arenot removed from caching table

[Mai-Saad]

Before submitting an issue please check that you’ve completed the following steps:

• Made sure you’re on the latest version => 3.12.6.1
• Used the search feature to ensure that the bug hasn’t been reported before

Describe the bug
Using Ultimate Member plugin, will add URL to cache table like https://new.rocketlabsqa.ovh/void(0);, this URL isnot removed once processed

To Reproduce

• Ultimate Member plugin is activated
• WP sitemap is disabled (URLs fetched from home page)
  Steps to reproduce the behavior:

1. install and activate WPR
2. Check cache table => mentioned URL is added and once processed will fail

Expected behavior
403 URLs shall be removed as same as 404 URLs

Screenshots
If applicable, add screenshots to help explain your problem.

Additional context
Add any other context about the problem here.

We shouldn't fetch this kind of URL.

Backlog Grooming (for WP Media dev team use only)

• Reproduce the problem
• Identify the root cause
• Scope a solution
• Estimate the effort

[mehedihasanziku]

#5543 maybe it should tackle also with this case

[jeawhanlee]

Reproduce the problem ✅

I was able to reproduce this issue on NGINX.

Identify the root cause ✅

We are not removing urls that don't have status of 200

Scope a solution ✅

We can make a request to get the response status of the url and bail out if not 200 in preload_url

wp-rocket/inc/Engine/Preload/Controller/PreloadUrl.php

Line 61 in ce9d862

public function preload_url( string $url ) {

$response = wp_remote_get( $url );

if ( is_wp_error( $response ) ) {
	$this->query->delete_by_url( $url );
	return;
}

$response_code = wp_remote_retrieve_response_code( $response );

if ( 200 !== $response_code ) {
	$this->query->delete_by_url( $url );
	return;
}

Estimate the effort ✅

[S]

[jeawhanlee]

Reproduce the problem ✅

I was able to reproduce this issue on NGINX.

Identify the root cause ✅

We are not removing urls that don't have status of 200

Scope a solution ✅

We can make a request to get the response status of the url and bail out if not 200 in preload_url

wp-rocket/inc/Engine/Preload/Controller/PreloadUrl.php

Line 61 in ce9d862

public function preload_url( string $url ) {

$response = wp_remote_get( $url );

if ( is_wp_error( $response ) ) {
	$this->query->delete_by_url( $url );
	return;
}

$response_code = wp_remote_retrieve_response_code( $response );

if ( 200 !== $response_code ) {
	$this->query->delete_by_url( $url );
	return;
}

Estimate the effort ✅

[S]

Following a point from @mostafa-hisham, the above grooming might be resource consuming on the customer site as we will be doing 2 requests to each url:

• one to check for status code.
• and if response code is 200 another again to preload.

As suggested by @mostafa-hisham we can also have a compatibility class for ultimate members and exclude urls like that void(0);
or I'm thinking we can utilize this request here and check the response code and perform necessary actions, don't know how well that will work.

@piotrbak @wp-media/php what do you think?

[piotrbak]

@jeawhanlee I believe that both options are not good enough:

1. As Mostafa mentioned, it'll create additional request
2. We're sending not-blocking request, it means that we're not waiting for the response

Maybe it's good time to move this array to the backend?

wp-rocket/inc/Engine/Preload/Controller/CheckExcludedTrait.php

Line 33 in f20c8f3

$regexes = (array) apply_filters( 'rocket_preload_exclude_urls', [ $pagination_regex ] );

Pinging @mostafa-hisham