Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ckanapi dump datasets --all doesn't work for catalog.data.gov #98

Closed
ekzhu opened this issue Nov 12, 2016 · 6 comments
Closed

ckanapi dump datasets --all doesn't work for catalog.data.gov #98

ekzhu opened this issue Nov 12, 2016 · 6 comments

Comments

@ekzhu
Copy link

ekzhu commented Nov 12, 2016 •
edited
Loading

I am trying to download all metadata from catalog.data.gov. The following command returns an API error.

ckanapi dump datasets --all -O catalog.data.gov.jsonl -p 1 -r http://catalog.data.gov

The error:

Traceback (most recent call last):
  File "/home/ekzhu/anaconda/bin/ckanapi", line 11, in <module>
    sys.exit(main())
  File "/home/ekzhu/anaconda/lib/python2.7/site-packages/ckanapi/cli/main.py", line 128, in main
    return dump_things(ckan, thing[0], arguments)
  File "/home/ekzhu/anaconda/lib/python2.7/site-packages/ckanapi/cli/dump.py", line 60, in dump_things
    names = ckan.call_action(get_thing_list, {})
  File "/home/ekzhu/anaconda/lib/python2.7/site-packages/ckanapi/remoteckan.py", line 83, in call_action
    return reverse_apicontroller_action(url, status, response)
  File "/home/ekzhu/anaconda/lib/python2.7/site-packages/ckanapi/common.py", line 131, in reverse_apicontroller_action
    raise CKANAPIError(repr([url, status, response]))
ckanapi.errors.CKANAPIError: ['http://catalog.data.gov/api/action/package_list', 302, u'<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">\n<html><head>\n<title>302 Found</title>\n</head><body>\n<h1>Found</h1>\n<p>The document has moved <a href="http://catalog.data.gov/api/action/package_search">here</a>.</p>\n</body></html>\n']

It looks like the catalog.data.gov have redirected the package_list to package_search. Is there a way to handle this? Am I missing something?
@wardi
Copy link
Contributor

wardi commented Nov 14, 2016

It's possible (and more efficient) to dump all datasets with package_search, but you have to be careful about duplicate or missing records that may appear in the results if datasets are being modified while you're dumping them.

Are you interested in helping to implement this? It shouldn't be too difficult.

@ekzhu
Copy link
Author

ekzhu commented Nov 14, 2016

I looked at the code in dump.py. It seems that it doesn't support making multiple paginated requests. catalog.data.gov has 193483 metadata so a single ckan.call_action with package_search is probably not going to return the complete result.

http://catalog.data.gov/api/action/package_search?rows=200&start=0

Does the current code support paginated requests somewhere?

@wardi
Copy link
Contributor

wardi commented Nov 14, 2016

No, using package_search is very different than the *_list, *_show that dump.py uses, so it won't be much help. getting dataset results from package_search should be very fast though, so you don't need dump.py's multiple-worker support. How about just writing it as a separate sub-command?

Also note that older versions of CKAN don't return the same data from package_search that they do from package_show, so it's useful to keep the current dump datasets implementation around for a little while.

@ekzhu
Copy link
Author

ekzhu commented Nov 14, 2016

Okay. Since it's convenient I will ask another question here. How do I know which version of the API am I calling. Is this

http://catalog.data.gov/api/action/package_search?rows=200&start=0

the newest API or do I have to explicitly add /3/:

http://catalog.data.gov/api/3/action/package_search?rows=200&start=0

Using /3/ may not work for some endpoints which don't have version 3.

@ekzhu ekzhu mentioned this issue Nov 15, 2016
Open
@wardi
Copy link
Contributor

wardi commented Nov 15, 2016

in ckanapi so far I've omitted the api version when calling ckan because I've always wanted the latest version of the API.

@varuneranki varuneranki mentioned this issue May 30, 2018
Open
@wardi
Copy link
Contributor

wardi commented Nov 17, 2020

note that we now have ckanapi search datasets for getting packages with package_search, closing this issue

@wardi wardi closed this as completed Nov 17, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@wardi @ekzhu