Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Homepage gets exported twice #12

Closed
sebastiandedeyne opened this Issue Mar 27, 2019 · 4 comments

Comments

Projects
None yet
2 participants
@sebastiandedeyne
Copy link
Member

sebastiandedeyne commented Mar 27, 2019

I think this might be an issue in spatie/crawler.

http://localhost and ``http://localhost/` are both crawled for some reason. The first should suffice.

@pepijn-de-wachter

This comment has been minimized.

Copy link

pepijn-de-wachter commented Mar 27, 2019

Hi,
I cannot seem to figure out how to reproduce this, made a brand new laravel app with 2 extra pages linked from the default welcome blade,
added a logger in CollectionCrawlQueue::hasAlreadyBeenProcessed() (spatie/crawler) which theroretically gets executed for every url the crawler finds.
log output:
[2019-03-27 20:10:56] local.DEBUG: http://localhost/
[2019-03-27 20:10:56] local.DEBUG: http://localhost/home
[2019-03-27 20:10:56] local.DEBUG: http://localhost/test
I also don't see any unexpected results in the dist folder.
Could you perhaps share a repository where this behaviour is noticed and in what way it visualizes?
Possibly os related? (I'm on the sometimes inferior Windows platform)
Kind regards

@sebastiandedeyne

This comment has been minimized.

Copy link
Member Author

sebastiandedeyne commented Mar 28, 2019

First of all, thanks for looking into this!

I was testing on a seperate branch of my personal site: https://github.com/sebastiandedeyne/sebastiandedeyne.com/tree/laravel-export

When I run php artisan export -v, this is what the output looks like:

Starting export...
[http://localhost/]
/ => /index.html
 => /index.html
/about => /about/index.html
/feed => /feed/index.html
/the-great-divide => /the-great-divide/index.html
/an-introduction-to-phoenix-liveview => /an-introduction-to-phoenix-liveview/index.html
/not-all-code-is-the-same => /not-all-code-is-the-same/index.html
/hindsight-2018-edition => /hindsight-2018-edition/index.html
/everyone-has-javascript-right => /everyone-has-javascript-right/index.html
/ia-writer-52-will-embrace-variable-fonts => /ia-writer-52-will-embrace-variable-fonts/index.html
...

The first and second page are the same.

We might do some changes regarding entries in this package later today (we're gonna try to get rid of the necessary localhost prefix internally), so this might take care of itself then...

@pepijn-de-wachter

This comment has been minimized.

Copy link

pepijn-de-wachter commented Mar 28, 2019

@sebastiandedeyne

This comment has been minimized.

Copy link
Member Author

sebastiandedeyne commented Apr 10, 2019

Update: the bug appears to only occur with our InternalClient Guzzle extension. We'll follow this up in #16.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.