Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix Portugal scraper to download actual records and yield jsons #502

Closed
yolile opened this issue Sep 21, 2020 · 0 comments · Fixed by #503
Closed

Fix Portugal scraper to download actual records and yield jsons #502

yolile opened this issue Sep 21, 2020 · 0 comments · Fixed by #503
Assignees
@yolile
Labels
existing spider
Projects
CDS 2020-05/2021-02

Comments

@yolile
Copy link
Member

yolile commented Sep 21, 2020

http://www.base.gov.pt/api/Record/GetRecords?offset=1 returns release packages with a list of compiled releases, not record packages. http://www.base.gov.pt/api/Record/GetRecordByOCID?ocid= returns a record package.
And

kingfisher-collect/kingfisher_scrapy/spiders/portugal_base.py

Line 25 in 3f18b90

yield self.build_file_from_response(response, data=json_array, data_type=self.data_type)

returns an array, not a json object, so pluck command fails.
@yolile yolile self-assigned this Sep 21, 2020
@yolile yolile added this to To do in CDS 2020-05/2021-02 via automation Sep 21, 2020
@yolile yolile mentioned this issue Sep 21, 2020
Merged
CDS 2020-05/2021-02 automation moved this from To do to Done Sep 21, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@yolile yolile

Labels
existing spider
Projects
No open projects
CDS 2020-05/2021-02
  
Done
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Update portugal to download records and jsons open-contracting/kingfisher-collect
1 participant
@yolile