Scraping problem with indexer

[aw-gerrit]

Use case

https://sag-sh.de

Issue: no hit although term exists on page

The search term „Albert-Schweitzer-Schule“ should throw a hit on following page:
https://sag-sh.de/referenzschulnetzwerk/archiv (but doesen't)

On the page the term exists as follows (span in a)
<a data-v-0c0bb767="" href="http://www.ass-wedel.de/" target="_blank" rel="noopener noreferrer" class="gtl-link"> <span data-v-0c0bb767="" class="font-semibold">Albert-Schweitzer-Schule</span></a>

[jannescb]

This is because

laravel-indexer/src/Commands/CreateIndexCommand.php

Line 73 in f84ff3b

$content = file_get_contents($url);

file_get_contents() wont render Javascript. The table in your example is a Vue-App.

@cbl

How much effort do you think would it take to add an optional chromium feature that could render each URL?

Alternative

We might consider implementing Browsershot and simply do:

Browsershot also can get the body of an html page after JavaScript has been executed:

Browsershot::url('https://example.com')->bodyHtml()

[aw-gerrit]

Could the @bot blade directive also be a solution for this? This would ensure a proper google index as well.

[jannescb]

@aw-gerrit

Yes, if you build a server rendered bot version and the @bot directive will trigger on the user agent of php it would work.