Discussion pagination is broken for crawlers (Google)

[gwillem]

Explanation

The noscript "next page" url is invalid, thus preventing crawlers to index followup posts.

Proof

$ curl -s 'https://discuss.flarum.org/d/7585-freeflarum-com-now-open-for-beta-access' | grep 'Next Page'
            <a href="https://discuss.flarum.org/d/7585?id=7585-freeflarum-com-now-open-for-beta-access&amp;page=2">Next Page &raquo;</a>
$ curl -s 'https://discuss.flarum.org/d/7585?id=7585-freeflarum-com-now-open-for-beta-access&amp;page=2' | grep 'Next Page'
            <a href="https://discuss.flarum.org/d/7585?id=7585&amp;amp%3Bpage=2&amp;page=2">Next Page &raquo;</a>
$ curl -s 'https://discuss.flarum.org/d/7585?id=7585&amp;amp%3Bpage=2&amp;page=2' | grep 'Next Page'
            <a href="https://discuss.flarum.org/d/7585?id=7585&amp;amp%3Bamp%3Bpage=2&amp;amp%3Bpage=2&amp;page=2">Next Page &raquo;</a>

Technical details

• Version of Flarum: beta 7

Likely solution

The & character is erroneously html-escaped. See https://github.com/flarum/core/blob/3dcfe32b27f27f896b3aa22ec4085b6fcac005b9/views/frontend/content/discussion.blade.php#L27

[franzliedke]

I disagree.

Instead of curl -s 'https://discuss.flarum.org/d/7585?id=7585-freeflarum-com-now-open-for-beta-access&page=2' | grep 'Next Page', you should run curl -s 'https://discuss.flarum.org/d/7585?id=7585-freeflarum-com-now-open-for-beta-access&page=2' | grep 'Next Page'.

The result is then correct.

That's what any browser would do, too.

[franzliedke]

That id parameter looks superfluous, though.

[franzliedke]

That will be fixed by improving the URL generator to append additional parameters to the query string.

[gwillem]

Franz, sorry for the erroneous report. However there doesn't seem to be a single "Next page" page indexed by Google. So something else is going wrong then:

https://www.google.com/search?q=site:discuss.flarum.org+inurl:%22page%3D%22

[franzliedke]

That might mean that Google has determined the page parameter does not have any effect on page contents, probably because of the bug you mentioned here.

That is now fixed, once it is rolled out, we should check this in Google's Webmaster Console.
@tobscure Do you have access to that?

[gwillem]

@franzliedke could be, however curl 'https://discuss.flarum.org/d/7585?id=7585-freeflarum-com-now-open-for-beta-access&page=2' does indeed yield the page 2 posts. Oh well, Google's ways are inscrutable.

[tobyzerner]

@franzliedke I've deployed the latest changes to discuss.flarum.org. I've also granted your gmail address access to Google Analytics for discuss.flarum.org. Hopefully with this you can add discuss.flarum.org to Google Webmaster Console using GA as the verification method. Please let me know if it doesn't work.

[franzliedke]

I have access. 😁

I already tweaked some settings, e.g. concerning the page parameter, let's see if this changes anything.

Interestingly, this graph shows the indexing status over the last year:

What happened at the end of last April? 😱

[gwillem]

Possibly penalized for duplicate content..