Delete search index entries under more specific conditions

[fritzmg]

In a Contao instance of a customer where indexing of protected pages is enabled I noticed the following issue: if the URL of protected and already indexed page is requested without a valid login for that page, the URL gets deleted from the search index.

To fix this I first thought of ignoring the 401 and 403 status code in our SearchIndexListener for the delete operation. However, then I realized that there are other status codes where the same holds true: a 503 status code is only temporary and any URL responding momentarily with that status code should not be removed from the index (neither would Google, they would only remove the URL from the index if that status code persists over a longer period of time).

Then I realized the same holds true for the 500 status code. Any error happening under a specific URL might only be temporary and thus the URL should not be removed from the index (neither would Google, they would only remove the URL from the index if that status code persists over a longer period of time).

Thus I then decided to completely revamp the conditions under which a URL should be deleted from the index:

• If the status code is 404 or 410, always delete from the index.
• If the response is succesful and the X-Robots-Tag contains noindex, always delete from the index.
• If the response is succesful and the HTML contains the meta robots tag with noindex, always delete from the index.
• Otherwise never delete from the index automatically, as any other state might just be temporary for any given URL.

[ausi]

If the X-Robots-Tag contains noindex, always delete from the index.

Shouldn’t we also check for $response->isSuccessful() additionally here? Meaning if the response is successful AND X-Robots-Tag contains noindex we should delete from the index, but not otherwise.

[fritzmg]

Shouldn’t we also check for $response->isSuccessful() additionally here? Meaning if the response is successful AND X-Robots-Tag contains noindex we should delete from the index, but not otherwise.

Hmm, do you mean in case a controller only sends this header during a 500 response for example? I guess that makes sense.

[Toflar]

LGTM

[leofeyer]

Thank you @fritzmg.