Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Unable to scrape URL #386

Closed
wbhob opened this issue Oct 6, 2021 · 2 comments
Closed

Unable to scrape URL #386

wbhob opened this issue Oct 6, 2021 · 2 comments

Comments

@wbhob
Copy link

wbhob commented Oct 6, 2021

I've got a sub stack URL with almost 900 posts and a custom domain. When I try to run the migration tool, it starts failing and saying "Unable to scrape URL".

My hunch is this may be because of rate limiting, so making this a background process with a delay between requests may make this more stable. Happy to debug one-on-one to give more context!
@PaulAdamDavis
Copy link
Member

PaulAdamDavis commented Nov 9, 2021 •
edited
Loading

Hi @wbhob! This has now been fixed. Scraping meta data was hitting a rate limit which is now accounted for. Scraping will be much slower, but now error-free.

It will be available in the next release.

@PaulAdamDavis
Copy link
Member

@wbhob This fix is now released in 0.22.0

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@PaulAdamDavis @wbhob