Problem with escaped fragment when fetching some websites

[Antoninum]

Hello !

I come from Wallabag, which is using this project.

I have problem retrieving content from a webpage, because graby automatically adds an ?_escaped_fragment_= at the end of the URL, for crawling AJAX purpose.
That's a problem because the website in question gives a 404 error when detecting this escaped fragment. Probably to avoid being fetched by robots ?

Still, the content seems to be accessible without the fragment.

A solution would be to try to fetch again the URL without this escaped fragment if a 404 error is answered ?

Here is the website, you can test with or without the escaped fragment:
https://dzone.com/
https://dzone.com/?_escaped_fragment_=

Thank you !

Antonin

[j0k3r]

This is related to that issue #19
I thought all Angular websites handled _escaped_fragment_ but it seems that not all of them.

• the smart solution: test with escaped fragment and if it failt, remove it and try again
• the fastest one: remove the Angular detector (which add the escaped fragment)