Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix(spiders): use date filters as part of file names #1026

Merged
merged 11 commits into from
Sep 20, 2023
Merged

fix(spiders): use date filters as part of file names #1026

Changes from 1 commit
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
8224fb1
fix(spiders): use date filters as part of file names
yolile Sep 19, 2023
c5c8f12
Merge branch 'main' into fix-incremental-file-names
jpmckinney Sep 19, 2023
0477133
kyrgyzstan: Use a comparable name, if from_date isn't set
jpmckinney Sep 19, 2023
a9f2cb4
portugal_base: Use a comparable name, if from_date isn't set
jpmckinney Sep 19, 2023
b5639b4
united_kingdom_fts: Use a comparable name, if until_date isn't set
jpmckinney Sep 19, 2023
172738e
Apply suggestions from code review
yolile Sep 19, 2023
135723e
flake8
yolile Sep 19, 2023
db7d544
fix colombia_api filename
yolile Sep 19, 2023
5c21481
chore(united_kingdom_*): Simplify default until_date
jpmckinney Sep 20, 2023
7066a00
Update kingfisher_scrapy/spiders/portugal_base.py
yolile Sep 20, 2023
ef95e65
united_kingdom_contracts_finder_base: restore callback
yolile Sep 20, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions kingfisher_scrapy/spiders/kyrgyzstan.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,10 +32,10 @@ class Kyrgyzstan(LinksSpider):

def start_requests(self):
jpmckinney marked this conversation as resolved.
Show resolved Hide resolved
url = 'http://ocds.zakupki.gov.kg/api/tendering'
file_name = 'start.json'
if self.from_date:
from_date = self.from_date.strftime(self.date_format)
# The API requires the timezone and seconds in the since parameter.
url = f'{url}?since={from_date}.00%2B06:00'
file_name = f'{from_date}-{file_name}'
yield scrapy.Request(url, meta={'file_name': file_name})
else:
from_date = '1970-01-01T00:00:00'
jpmckinney marked this conversation as resolved.
Show resolved Hide resolved
yield scrapy.Request(url, meta={'file_name': f'{from_date}.json'})