XML Sitemap Not Found Notification When Sitemap Exists(v151114) #618

patdumond · 2015-11-19T20:14:01Z

Reference:
https://websharks.zendesk.com/agent/tickets/9511
https://websharks.zendesk.com/agent/tickets/9568
https://websharks.zendesk.com/agent/tickets/9567
https://websharks.zendesk.com/agent/tickets/9573
https://websharks.zendesk.com/agent/tickets/9583

In addition to the five tickets listed above, I've experienced this on four of my sites, including my non-Multisite test site.

After updating ZenCache Pro to v151114, this notification is displayed:

ZenCache says... The Auto-Cache Engine is currently configured with an XML Sitemap location that could not be found. We suggest that you install the Google XML Sitemaps plugin. Or, empty the XML Sitemap field and only use the list of URLs instead. See: Dashboard → ZenCache → Auto-Cache Engine → XML Sitemap URL.

Each of the above users said that they had Google XML Sitemaps installed and I verified that a valid sitemap was found on each site. Same with my four sites except I use Better Google XML Sitemaps. I have been able to get rid of the message on all but 1 of my sites by:

deleting the sitemap name
saving changes
typing the sitemap name back in
saving changes

However, I have one site that just refuses to admit that a sitemap exists and a few of the Zendesk users have reported that the above steps did not work for them either.

raamdev · 2015-11-19T21:28:55Z

@patdumond Thank you very much for documenting this here.

@jaswsinc I have a feeling some of these reports (especially any persistent reports of the XML Sitemap missing when it in fact exists) might be related to misconfigured servers. I've seen a fairly common issue where local DNS is not configured properly which results in the hostname for the site hosted on the server not being accessible from the server. In that scenario, ZenCache would attempt to visit the XML Sitemap and it would timeout/fail, reporting a false error (well, somewhat false; if ZenCache can't load the XML Sitemap, then the Auto-Cache Engine won't be able to find it, which means that not being able to load the XML Sitemap is a problem, no matter the scenario).

We could get more descriptive with why ZenCache failed to locate the XML Sitemap (did it return a 404? was the server unable to resolve the hostname? etc.) and then link to a KB Article that documents explanations for each error (e.g., "contact your web hosting company if your server cannot resolve its own URL"). (There's a GitHub issue open here related to improving that error notice: #616)

jaswrks · 2015-11-19T21:52:57Z

@raamdev writes...

We could get more descriptive with why ZenCache failed to locate the XML Sitemap (did it return a 404? was the server unable to resolve the hostname? etc.)

Agree.

@patdumond writes...

However, I have one site that just refuses to admit that a sitemap exists and a few of the Zendesk users have reported that the above steps did not work for them either.

On that site, if you can run the following test it might shed light.

Create this PHP test file in the root of your WordPress installation please.
/xml-sitemap-test.php

<?php
require_once dirname(__FILE__).'/wp-load.php';
header('Content-Type: text/plain; charset=utf-8');
print_r(wp_remote_request('http://example.com/sitemap.xml'));

In the above code snippet, change http://example.com/sitemap.xml to your Sitemap URL.
Open http://example.com/xml-sitemap-test.php in a browser and paste the output for us.
Delete /xml-sitemap-test.php after you're done.

patdumond · 2015-11-19T22:28:42Z

Well, I went to run that test on my stubborn site and the message had disappeared. I know it was still there yesterday afternoon, because I logged in specifically to check that before closing up everything for the night.

ufufufu · 2015-11-19T22:39:16Z

Just run the code above, and below is the output:-

jaswrks · 2015-11-19T22:53:42Z

@ufufufu Very helpful. Thanks!

If you wouldn't mind, can you run one more test, but this time with this alternate code snippet that uses wp_remote_head() instead please.

<?php
require_once dirname(__FILE__).'/wp-load.php';
header('Content-Type: text/plain; charset=utf-8');
print_r(wp_remote_head('http://example.com/sitemap.xml'));

ufufufu · 2015-11-19T23:02:59Z

The result has considerably different formatting (no line-breaks):-

Array ( [headers] => Array ( [date] => Thu, 19 Nov 2015 23:01:12 GMT [server] => Apache [expires] => Thu, 19 Nov 1981 08:52:00 GMT [cache-control] => no-store, no-cache, must-revalidate, post-check=0, pre-check=0 [pragma] => no-cache [x-pingback] => https://imouto.my/xmlrpc.php [x-robots-tag] => noindex [set-cookie] => Array ( [0] => ct_cookies_test=5566f72b110eb5a7b6503a216e09f6de; path=/ [1] => PHPSESSID=6ad7227k91ibmsa934alrfgai5; path=/ ) [strict-transport-security] => max-age=15768000 [public-key-pins] => max-age=5184000; pin-sha256=2qKyaG0PHPExrUszbEua2SPHmHze8rNa1RNK2pHwfsI=; pin-sha256=Z3TyA+moOEVIq1FRV6C8RAkOJBiX22a80bT3BIs8o80=; pin-sha256=L0YY0bTrFt+E5DCWQYYZq4YSF5ORfMnF4trDzWBjP28= [connection] => close [content-type] => text/xml; charset=utf-8 ) [body] => [response] => Array ( [code] => 200 [message] => OK ) [cookies] => Array ( [0] => WP_Http_Cookie Object ( [name] => ct_cookies_test [value] => 5566f72b110eb5a7b6503a216e09f6de [expires] => [path] => / [domain] => imouto.my ) [1] => WP_Http_Cookie Object ( [name] => PHPSESSID [value] => 6ad7227k91ibmsa934alrfgai5 [expires] => [path] => / [domain] => imouto.my ) ) [filename] => )

jaswrks · 2015-11-19T23:10:24Z

@ufufufu Thank you. Both of those tests show that WordPress is properly identifying that you do in fact have https://imouto.my/sitemap.xml. Couple of questions for you.

Do the test results change at all when you alter the scheme; i.e., http:// vs. https:// for the URL?
Can you confirm that you are in fact still seeing the warning notice in ZenCache regarding a missing sitemap?

jaswrks · 2015-11-19T23:20:37Z

Also, are you running a WordPress Multisite Network or a standard installation?

ufufufu · 2015-11-19T23:26:25Z

If I use http:// instead, I will get the output below:-

WP_Error Object
(
    [errors] => Array
        (
            [http_request_failed] => Array
                (
                    [0] => Could not resolve host: imouto.mysitemap.xml; Name or service not known
                )

        )

    [error_data] => Array
        (
        )

)

This is by using the...

<?php
require_once dirname(__FILE__).'/wp-load.php';
header('Content-Type: text/plain; charset=utf-8');
print_r(wp_remote_request('http://imouto.my/sitemap.xml'));

...script.

BTW, unless I also clear the opcache. the script's output will be cached.

Also, I've configured Apache to always redirect to https and remove the www. portion of the URL using the code below in Apache configuration.

RewriteEngine On
RewriteCond %{HTTP_HOST} ^www.(.+)$ [NC]
RewriteRule ^(.*)$ https://imouto.my$1 [R=301,L]

RewriteEngine On
RewriteCond %{SERVER_PORT} !^443$
RewriteRule ^/(.*) https://%{HTTP_HOST}$1 [NC,R=301,L]

Therefore, if I load http://imouto.my/sitemap.xml on my browser, it will automatically redirect to https://imouto.my/sitemap.xml

Unless I disable the Autocache engine completely, I will still see the error message.
This is a standard installation.

jaswrks · 2015-11-19T23:30:38Z

If I use http:// instead, I will get the output below

Thank you very much for that info. What URL is defined in your WordPress General Options? i.e., what do you have here please? I'm curious to see if perhaps ZenCache is attempting to access the http:// variation by mistake.

jaswrks · 2015-11-19T23:33:04Z

@ufufufu I see that the error report you pasted shows:

Could not resolve host: imouto.mysitemap.xml

That is incorrect altogether, because that's an invalid host name.

Looking at your .htaccess file, I see this line that is causing that.

RewriteRule ^/(.*) https://%{HTTP_HOST}$1 [NC,R=301,L]

It is stripping the / away and creating an invalid host name in the redirection.

It should be:

RewriteRule (.*) https://%{HTTP_HOST}$1 [NC,R=301,L]

Or this would also work:

RewriteRule ^(.*)$ https://imouto.my$1 [R=301,L]

jaswrks · 2015-11-19T23:45:11Z

@raamdev One possible reason for this could be Multisite Network installations. I'm seeing this block of code where ZenCache is actually trying to find an XML Sitemap for each child blog in the network, and that's not always going to be the case; i.e., the Main Site in the Network may be the only location for the XML sitemaps. In such a case, ZenCache could be alerting site owners about child blogs that really don't matter to them anyway, and that could lead to some of this confusion.

ufufufu · 2015-11-20T00:27:53Z

Yeah, after editing the relevant .conf file, now both http:// and https:// has the same output.

In 'General Options', both fields contains the correct info.

jaswrks · 2015-11-20T00:37:26Z

@ufufufu Thanks! And you're not running a Multisite Network, right?

ufufufu · 2015-11-20T00:38:44Z

Yes, only a single standard installation.

raamdev · 2015-11-22T00:31:50Z

Related: #616

raamdev · 2015-11-22T00:43:54Z

@jaswsinc writes...

One possible reason for this could be Multisite Network installations. I'm seeing this block of code where ZenCache is actually trying to find an XML Sitemap for each child blog in the network, and that's not always going to be the case; i.e., the Main Site in the Network may be the only location for the XML sitemaps. In such a case, ZenCache could be alerting site owners about child blogs that really don't matter to them anyway, and that could lead to some of this confusion.

So should we change the behavior so that by default only the Main Site is used for an XML Sitemap and then have an option (that appears when Network Activated) that allows a site owner to specifically enable "Check Each Child Blog For an XML Sitemap"?

If we did that, we'd also need to update the XML Sitemap error message to mention the domain (or maybe the full path) where the XML Sitemap check failed, to make debugging this issue for a Multisite Admin a lot easier ("which Child Blog is missing an XML Sitemap?"). And in that case, it should probably also be possible for a Multisite Admin to dismiss certain notices, if, for example, they already know that a specific Child Blog doesn't have and won't have a XML Sitemap (that would be work related to #616).

However, I'm a bit concerned about feature creep here. It seems like it would be better to just disable XML Sitemaps for Child Blogs (or disable error reporting for Child Blog XML Sitemaps) and leave the additional work to a feature request, if such improvements are requested.

jaswrks · 2015-11-23T22:30:48Z

So should we change the behavior so that by default only the Main Site is used for an XML Sitemap and then have an option (that appears when Network Activated) that allows a site owner to specifically enable "Check Each Child Blog For an XML Sitemap"?

Yes, I agree with that approach. I don't think that's any big deal. A simple checkbox to flick that on seems worth it to me; i.e., not feature creep IMO.

or disable error reporting for Child Blog XML Sitemaps

That, in concert with the checkbox, sounds like a good idea to me. We can just disable error reporting for child blogs and let the log file suffice in that instance. If a network admin really wants to look closer they can go to the log for this particular aspect. Seems good enough to me.

…and wpsharks/comet-cache#618

raamdev · 2015-12-03T13:40:06Z

Next Pro Release Changelog:

Enhancement: The Auto-Cache Engine diagnostic reporting for XML Sitemap-related errors has been greatly improved. The Dashboard notice now explains exactly what error is occurring when the Auto-Cache Engine attempts to verify that the XML Sitemap exists. Props @jaswsinc. See Issue #615 and Issue #618.
Multisite Enhancement: The Auto-Cache Engine has a new configuration option when running on WP Multisite Networks that allows you to define whether or not the Auto-Cache Engine should look for XML Sitemaps on each of the child blogs. This now defaults to being off; i.e., by default, child blogs are no longer checked. Props @jaswsinc. See Issue #618.
Multisite Enhancement: The Auto-Cache Engine XML Sitemap diagnostic reporting is now always disabled for child blogs and only errors related to a primary sitemap location are displayed on the Dashboard. Props @jaswsinc. See Issue #618.

raamdev · 2015-12-21T01:22:17Z

ZenCache Pro v151220 has been released and includes changes worked on as part of this GitHub Issue. See the release announcement for further details.

This issue will now be locked to further updates. If you have something to add related to this GitHub Issue, please open a new GitHub Issue and reference this one (#618).

raamdev added bug needs testing needs research auto-cache engine labels Nov 19, 2015

raamdev added this to the Next Release (Pro) milestone Nov 19, 2015

raamdev mentioned this issue Nov 22, 2015

Auto-Cache Engine XML Sitemap Error Notice Only Shows Once Every 15 Minutes #616

Closed

This was referenced Dec 3, 2015

Warning about lack of xml sitemap when the sitemap exists #615

Closed

PR: feature/615 wpsharks/comet-cache-pro#192

Closed

jaswrks pushed a commit to wpsharks/comet-cache-pro that referenced this issue Dec 3, 2015

Diagnostic reporting and new MS option. See: wpsharks/comet-cache#615 …

900d965

…and wpsharks/comet-cache#618

raamdev closed this Dec 3, 2015

wpsharks locked and limited conversation to collaborators Dec 21, 2015

wpsharks / comet-cache

XML Sitemap Not Found Notification When Sitemap Exists(v151114) #618

XML Sitemap Not Found Notification When Sitemap Exists(v151114) #618

patdumond commented Nov 19, 2015

raamdev commented Nov 19, 2015

jaswrks commented Nov 19, 2015

patdumond commented Nov 19, 2015

ufufufu commented Nov 19, 2015

jaswrks commented Nov 19, 2015

ufufufu commented Nov 19, 2015

jaswrks commented Nov 19, 2015

jaswrks commented Nov 19, 2015

ufufufu commented Nov 19, 2015

jaswrks commented Nov 19, 2015

jaswrks commented Nov 19, 2015

jaswrks commented Nov 19, 2015

ufufufu commented Nov 20, 2015

jaswrks commented Nov 20, 2015

ufufufu commented Nov 20, 2015

raamdev commented Nov 22, 2015

raamdev commented Nov 22, 2015

jaswrks commented Nov 23, 2015

raamdev commented Dec 3, 2015

raamdev commented Dec 21, 2015

wpsharks / comet-cache

XML Sitemap Not Found Notification When Sitemap Exists(v151114) #618

XML Sitemap Not Found Notification When Sitemap Exists(v151114) #618

Comments

patdumond commented Nov 19, 2015

raamdev commented Nov 19, 2015

jaswrks commented Nov 19, 2015

patdumond commented Nov 19, 2015

ufufufu commented Nov 19, 2015

jaswrks commented Nov 19, 2015

ufufufu commented Nov 19, 2015

jaswrks commented Nov 19, 2015

jaswrks commented Nov 19, 2015

ufufufu commented Nov 19, 2015

jaswrks commented Nov 19, 2015

jaswrks commented Nov 19, 2015

jaswrks commented Nov 19, 2015

ufufufu commented Nov 20, 2015

jaswrks commented Nov 20, 2015

ufufufu commented Nov 20, 2015

raamdev commented Nov 22, 2015

raamdev commented Nov 22, 2015

jaswrks commented Nov 23, 2015

raamdev commented Dec 3, 2015

raamdev commented Dec 21, 2015