Return noindex, nofollow when wpseo_robots is false

[mmikhan]

Context

• When add_filter( 'wpseo_robots', '__false' ); is set, noindex, nofllow isn't shown but only max-image-preview:large is being shown, which is wrong. When the relevant filter is set to false, it shall output noindex, nofollow to discourage search engine crawling the entire site.

Summary

This PR can be summarized in the following changelog entry:

• Fixes a bug where passing __false to the wpseo_robots filter would result in a max-image-preview:large robots meta directive instead of the desired noindex, nofollow directive.

Test instructions

Test instructions for the acceptance test before the PR gets merged

This PR can be acceptance tested by following these steps:

• Set add_filter('wpseo_robots', '__return_false'); on functions.php file.
• Check the robots meta output on the front-end of the website to look for noindex, nofllow.

Test instructions for QA when the code is in the RC

• QA should use the same steps as above.

Quality assurance

• I have tested this code to the best of my abilities
• I have added unittests to verify the code works as intended

Fixes #18483

[jdevalk]

LGTM, needs acceptance testing.

Also, let's check our code for cases where we return false on this and why we do.

[enricobattocchi]

Acceptance test OK! Changing target to 19.8 to include this in our next release

[LoganStrike]

FYI, the code snippets on the following page from a Yoast support engineer show how to remove the Yoast robots tag by returning false. But the change in this PR resulted in a noindex,nofollow robots tag when returning false, instead of suppressing the robots tag altogether.

https://gist.github.com/amboutwe/0c71e42aa164238007d7ea88f174a93f

This is a huge breaking change considering the impact on SEO for sites that previously used this filter to remove the Yoast robots tag. Those sites/pages went from having no robots tag to having a noindex,nofollow robots tag.