Consider options for opening APIs without tokens

[scolapasta]

In 4.0, we require tokens for all APIs even if searching published dataverses and datasets. Going forward, we want to think of the best way to open this up (rate limts? guest token?)

[bencomp]

As I just suggested on IRC: how about making this part of configuration of a Dataverse installation? If possible, making it configurable per endpoint could be useful.

I would very likely open up the Search API, likely to open the dataset metadata (native) endpoint and possibly open up the dataverse (native) endpoint.

If "free access" enabled, not including an API key should be interpreted as if a guest API key was used.

I don't know about best practices for rate limiting, but the basis for it could be configurable as well (e.g. by IP, per day/week/month). There is an issue for rate limiting logic, #1339, in which @kcondon writes:

Limiting access might involve varying degrees of options: general api access on/off switch, per api, and/or whitelist/blacklist of ip addresses/ users. The last might be integrated with groups and permissions.

[bencomp]

As I just suggested on IRC: how about making this part of configuration of a Dataverse installation? If possible, making it configurable per endpoint could be useful.

I would very likely open up the Search API, likely to open the dataset metadata (native) endpoint and possibly open up the dataverse (native) endpoint.

If "free access" enabled, not including an API key should be interpreted as if a guest API key was used.

I don't know about best practices for rate limiting, but the basis for it could be configurable as well (e.g. by IP, per day/week/month). There is an issue for rate limiting logic, #1339, in which @kcondon writes:

Limiting access might involve varying degrees of options: general api access on/off switch, per api, and/or whitelist/blacklist of ip addresses/ users. The last might be integrated with groups and permissions.

[pdurbin]

If a Dataverse installation someday runs in a mode where the Search API doesn't require a token, a proof of concept Android app I wrote could be used without needing he modifications I opened this issue about: IQSS/dataverse-android#1 :)

[pdurbin]

If a Dataverse installation someday runs in a mode where the Search API doesn't require a token, a proof of concept Android app I wrote could be used without needing he modifications I opened this issue about: IQSS/dataverse-android#1 :)

[karissa]

Unfortunately the search requires an api key, unlike figshare, socrata labs, datahub (ckan) and others... so we're going to hold off on integrating it into http://github.com/karissa/dps

[karissa]

Unfortunately the search requires an api key, unlike figshare, socrata labs, datahub (ckan) and others... so we're going to hold off on integrating it into http://github.com/karissa/dps

[pameyer]

The eUtils guidelines (http://www.ncbi.nlm.nih.gov/books/NBK25497/#chapter2.Usage_Guidelines_and_Requiremen) might be worth a look.

Requiring authentication (token or otherwise) for read-access APIs seems unusual, at least based on other repository APIs I've used (eUtils, RCSB). Is there a significant effect on system load?

[pameyer]

The eUtils guidelines (http://www.ncbi.nlm.nih.gov/books/NBK25497/#chapter2.Usage_Guidelines_and_Requiremen) might be worth a look.

Requiring authentication (token or otherwise) for read-access APIs seems unusual, at least based on other repository APIs I've used (eUtils, RCSB). Is there a significant effect on system load?

[pdurbin]

Yesterday at https://botbot.me/freenode/dat/msg/75629403/ I asked @joehand to weigh in on what @karissa wrote at #1838 (comment) saying, "Is requiring an API Token a show stopper for including Dataverse installations in PublicBits?"

He responded, "I think it's not a complete blocker now. That original idea was more user driven requests closer to one time http requests. But now I think we'll be doing something closer to centralized harvesting."

This reminds me of https://share.osf.io , of course, (and related issue #2291), where SHARE is using an API token to query the Dataverse Search API periodically. (Now that OAI-PMH is supported as of Dataverse 4.5 I suppose they could switch to that.)

Anyway, as I originally expressed at #1809 (comment) I still feel that requiring API tokens will prevent potential collaborators from even trying out Dataverse APIs. I like how the GitHub API doesn't require any authentication. "For unauthenticated requests, the rate limit allows you to make up to 10 requests per minute" according to https://developer.github.com/v3/search/#rate-limit . At least that let's you play with the API a bit before creating an account.

For the record, a few days after @karissa made her comment, I pushed a branch at c876fee with a configuration option called :SearchApiTokenlessGuestAllowed but it doesn't do any fancy rate limiting or anything (#1339 and also mentioned in passing at #3153). It just opens up the Search API like it was before we decided to close it at #1809 (comment) . I'd like to remind us all, however, that we don't the most consistency when it comes to which APIs require a token or not. For example, exporting datasets as JSON (or XML, I believe) doesn't require a token for published datasets. I actually favor more openness like this for greater hackability. I'm just saying that the rules are a bit inconsistent. That's why issues like #1809 are still open.

Incidentally, it was a pleasure to meet @joehand yesterday. I gave him a demo of @pameyer 's https://data.sbgrid.org at http://irclog.iq.harvard.edu/dataverse/2016-10-28#i_44033 😄

[pdurbin]

Yesterday at https://botbot.me/freenode/dat/msg/75629403/ I asked @joehand to weigh in on what @karissa wrote at #1838 (comment) saying, "Is requiring an API Token a show stopper for including Dataverse installations in PublicBits?"

He responded, "I think it's not a complete blocker now. That original idea was more user driven requests closer to one time http requests. But now I think we'll be doing something closer to centralized harvesting."

This reminds me of https://share.osf.io , of course, (and related issue #2291), where SHARE is using an API token to query the Dataverse Search API periodically. (Now that OAI-PMH is supported as of Dataverse 4.5 I suppose they could switch to that.)

Anyway, as I originally expressed at #1809 (comment) I still feel that requiring API tokens will prevent potential collaborators from even trying out Dataverse APIs. I like how the GitHub API doesn't require any authentication. "For unauthenticated requests, the rate limit allows you to make up to 10 requests per minute" according to https://developer.github.com/v3/search/#rate-limit . At least that let's you play with the API a bit before creating an account.

For the record, a few days after @karissa made her comment, I pushed a branch at c876fee with a configuration option called :SearchApiTokenlessGuestAllowed but it doesn't do any fancy rate limiting or anything (#1339 and also mentioned in passing at #3153). It just opens up the Search API like it was before we decided to close it at #1809 (comment) . I'd like to remind us all, however, that we don't the most consistency when it comes to which APIs require a token or not. For example, exporting datasets as JSON (or XML, I believe) doesn't require a token for published datasets. I actually favor more openness like this for greater hackability. I'm just saying that the rules are a bit inconsistent. That's why issues like #1809 are still open.

Incidentally, it was a pleasure to meet @joehand yesterday. I gave him a demo of @pameyer 's https://data.sbgrid.org at http://irclog.iq.harvard.edu/dataverse/2016-10-28#i_44033 😄

[T-Hormesch]

Hello,

I got a question regarding the tokens that are necessary to use the API of Harvard Dataverse.

I developed a free desktop application that is able
to search Harvard Dataverse by use of my own API token.

My question is:
Is it still allowed
(regarding the Harvard Dataverse General Terms of Use and the Harvard Dataverse API Terms of Use)
to spread copies of that application to other users,
regarding the fact that all these copies then also will access the Harvard Dataverse API
by usage of my own API token?

I read the Harvard Dataverse General Terms of Use and the Harvard Dataverse API Terms of Use
and I couldn't really find something that would forbid this.

I already contacted the Harvard Dataverse team per E-Mail, asking the same question.
They told me that they are "considering removing this limitation"
(meaning the limitation to provide a token when searching through Harvard Dataverse using the API).
Additionally, I was asked to post this whole story also here.

[T-Hormesch]

Hello,

I got a question regarding the tokens that are necessary to use the API of Harvard Dataverse.

I developed a free desktop application that is able
to search Harvard Dataverse by use of my own API token.

My question is:
Is it still allowed
(regarding the Harvard Dataverse General Terms of Use and the Harvard Dataverse API Terms of Use)
to spread copies of that application to other users,
regarding the fact that all these copies then also will access the Harvard Dataverse API
by usage of my own API token?

I read the Harvard Dataverse General Terms of Use and the Harvard Dataverse API Terms of Use
and I couldn't really find something that would forbid this.

I already contacted the Harvard Dataverse team per E-Mail, asking the same question.
They told me that they are "considering removing this limitation"
(meaning the limitation to provide a token when searching through Harvard Dataverse using the API).
Additionally, I was asked to post this whole story also here.

[pdurbin]

@T-Hormesch it's awesome that you've created a free desktop application that is able to search Harvard Dataverse. (Maybe we could list it at http://guides.dataverse.org/en/4.6.1/api/apps.html .) It should work on all the other installations of Dataverse on the map at http://dataverse.org . I'm not a lawyer and opening the ticket at https://help.hmdc.harvard.edu/Ticket/Display.html?id=249710 was the right thing to do to get in touch with people from Harvard. You could consider allowing your users to paste in their own API token. I know this is a pain and it's why IQSS/dataverse-android#1 is still open for an Android app I knocked together over winter break years ago.

I think it would be nice if the Dataverse Search API was like the GitHub Search API, allowing anonymous access for the first few requests and then requiring a token for additional requests (throttling, as described at #1339).

[pdurbin]

@T-Hormesch it's awesome that you've created a free desktop application that is able to search Harvard Dataverse. (Maybe we could list it at http://guides.dataverse.org/en/4.6.1/api/apps.html .) It should work on all the other installations of Dataverse on the map at http://dataverse.org . I'm not a lawyer and opening the ticket at https://help.hmdc.harvard.edu/Ticket/Display.html?id=249710 was the right thing to do to get in touch with people from Harvard. You could consider allowing your users to paste in their own API token. I know this is a pain and it's why IQSS/dataverse-android#1 is still open for an Android app I knocked together over winter break years ago.

I think it would be nice if the Dataverse Search API was like the GitHub Search API, allowing anonymous access for the first few requests and then requiring a token for additional requests (throttling, as described at #1339).