-
Notifications
You must be signed in to change notification settings - Fork 55
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Can only scrape 10000 items. #19
Comments
yes, if we could sort by last-modfiied date that would be very good. |
thanks for this information. It's very valuable |
looking into this request. |
here's my current workaround. https://github.com/datamade/govinfo/blob/master/govinfo/__init__.py#L32-L61 there's a much more elegant solution is the data was sorted. |
collections response is now sorting by lastModified date. snippet: {
"count": 27027,
"message": null,
"nextPage": "https://api.govinfo.gov/collections/CHRG/1990-01-01T00:00:00Z/?offset=100&pageSize=100",
"previousPage": null,
"packages": [
{
"packageId": "CHRG-103hhrg76069",
"lastModified": "2018-10-30T17:25:19Z",
"packageLink": "https://api.govinfo.gov/packages/CHRG-103hhrg76069/summary",
"docClass": "HHRG",
"title": "Bpa Competitiveness"
},
{
"packageId": "CHRG-103hhrg74075",
"lastModified": "2018-10-30T17:21:12Z",
"packageLink": "https://api.govinfo.gov/packages/CHRG-103hhrg74075/summary",
"docClass": "HHRG",
"title": "Bpa Proposed Flscal Year 1994 Budget"
},
{
"packageId": "CHRG-103hhrg69593",
"lastModified": "2018-10-30T17:17:43Z",
"packageLink": "https://api.govinfo.gov/packages/CHRG-103hhrg69593/summary",
"docClass": "HHRG",
"title": "Indian Tribal Justice Act"
},
{
"packageId": "CHRG-103hhrg74346",
"lastModified": "2018-10-30T13:44:22Z",
"packageLink": "https://api.govinfo.gov/packages/CHRG-103hhrg74346/summary",
"docClass": "HHRG",
"title": "Bpa Electric Power Resources Acquisition"
},
{
"packageId": "CHRG-113shrg28394",
"lastModified": "2018-10-30T13:41:19Z",
"packageLink": "https://api.govinfo.gov/packages/CHRG-113shrg28394/summary",
"docClass": "SHRG",
"title": "Reassessing Solitary Confinement II: The Human Rights, Fiscal, and Public Safety Consequences"
},
{
"packageId": "CHRG-115shrg27017",
"lastModified": "2018-10-30T13:41:16Z",
"packageLink": "https://api.govinfo.gov/packages/CHRG-115shrg27017/summary",
"docClass": "SHRG",
"title": "Nominations of Brock Long, Russell Vought and Neomi Rao"
},
... |
@jonquandt in order use the sorting as a way to get around the 10000 row limit, last modified needs to be sorted from earlier to later. right now it's sorted from later to ealrier. If it's earlier to later, I can just use the last element I scraped before hitting the 10000 limit, and use that as a new start time parameter. |
actually, nevermind. I can make this work. |
Thank you for making this changde |
This may be by intention, but
https://api.govinfo.gov/collections/CHRG/1990-01-01T00:00:00Z/?offset=10000&pageSize=100&api_key=DEMO_KEY
returns
If this policy is intended, it would be good to
Thanks so much.
The text was updated successfully, but these errors were encountered: