Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update colombia scraper to no retry after a 404 error #415

Closed
yolile opened this issue Jun 9, 2020 · 1 comment · Fixed by #416 or #424
Closed

Update colombia scraper to no retry after a 404 error #415

yolile opened this issue Jun 9, 2020 · 1 comment · Fixed by #416 or #424

Comments

@yolile
Copy link
Member

yolile commented Jun 9, 2020

The Colombia API changed and now if a link returns 404 it means that there are no more pages
We can update the API to include from_date parameter too

@yolile yolile added this to To do in CDS 2020-05/2021-02 via automation Jun 9, 2020
@yolile yolile self-assigned this Jun 11, 2020
CDS 2020-05/2021-02 automation moved this from To do to Done Jun 17, 2020
@jpmckinney jpmckinney reopened this Jun 22, 2020
CDS 2020-05/2021-02 automation moved this from Done to In progress [6 max] Jun 22, 2020
@jpmckinney
Copy link
Member

Let's reuse (or improve) the logic in BaseSpider for setting the default from/until date, instead of having that logic in the individual spider. The Colombia spider should just be concerned with setting the date_format class variable and with calling strftime.

CDS 2020-05/2021-02 automation moved this from In progress [6 max] to Done Jun 22, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
No open projects
2 participants