Skip to content
This repository has been archived by the owner on Feb 7, 2022. It is now read-only.

Optimise the crawler for more stable performance. #5

Open
ArvinZJC opened this issue Jul 15, 2021 · 0 comments
Open

Optimise the crawler for more stable performance. #5

ArvinZJC opened this issue Jul 15, 2021 · 0 comments
Assignees
@ArvinZJC
Labels
enhancement New feature or request help wanted Extra attention is needed
Milestone
Release V0.10.0.

Comments

@ArvinZJC
Copy link
Owner

ArvinZJC commented Jul 15, 2021 •
edited

The current crawling strategy would cause that Sina refuses access to the non-public stock API for 5-60 minutes. Given that the API is the best way to retrieve required stock data at present, I would like to work out some workarounds to bypass the limitation or mitigate this issue.
Any good practice for crawling (e.g., rest for 10 seconds for every 10 queries)?
Anti-anti-crawling?
Online+offline?
@ArvinZJC ArvinZJC added enhancement New feature or request help wanted Extra attention is needed labels Jul 15, 2021
@ArvinZJC ArvinZJC added this to the Release V0.10.0. milestone Jul 15, 2021
@ArvinZJC ArvinZJC self-assigned this Jul 15, 2021
@ArvinZJC ArvinZJC mentioned this issue Jul 15, 2021
Open
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@ArvinZJC ArvinZJC

Labels
enhancement New feature or request help wanted Extra attention is needed
Projects
None yet
Milestone
Release V0.10.0.
Development

No branches or pull requests
1 participant
@ArvinZJC