New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unable to obtain full list of funders #179

Closed
dlenhardt opened this Issue Jan 19, 2017 · 5 comments

Comments

Projects
None yet
5 participants
@dlenhardt

dlenhardt commented Jan 19, 2017

The /funders resource component reports there are currently nearly 14,000 funders:

http://api.crossref.org/funders?rows=0

The /funders resource component can be traversed by using the rows and offset parameters. For example:

http://api.crossref.org/funders?rows=5&offset=5

The rows parameter has a limit of 1000 rows, and the offset parameter has a limit of 10,000 rows. Therefore, one can access up to 11,000 funders. Is there a way to access funders beyond that number? The cursor parameter does not work with the funders resource component.

@AshleyMoore

This comment has been minimized.

Show comment
Hide comment
@AshleyMoore

AshleyMoore Feb 1, 2017

I am also finding this issue. I would like to return all works by a single funder. I am unable to step through records once limit is reached as I am unable to use the cursor parameter.

AshleyMoore commented Feb 1, 2017

I am also finding this issue. I would like to return all works by a single funder. I am unable to step through records once limit is reached as I am unable to use the cursor parameter.

@dhimmel

This comment has been minimized.

Show comment
Hide comment
@dhimmel

dhimmel Mar 22, 2017

I'm having a similar issue with trying to query the journals component. There are currently 52135 journals, but the offset parameter is limited to 10000.

cursor solves this issue, but is only available for the works component. It would be nice to see cursor enabled for more components.

Tagging active maintainers: @kmeddings @gbilder @kjw. Unless we're missing something, this seems like a serious issue... a large portion of funders and journals are unavailable to exhaustive queries.

dhimmel commented Mar 22, 2017

I'm having a similar issue with trying to query the journals component. There are currently 52135 journals, but the offset parameter is limited to 10000.

cursor solves this issue, but is only available for the works component. It would be nice to see cursor enabled for more components.

Tagging active maintainers: @kmeddings @gbilder @kjw. Unless we're missing something, this seems like a serious issue... a large portion of funders and journals are unavailable to exhaustive queries.

@dhimmel

This comment has been minimized.

Show comment
Hide comment
@dhimmel

dhimmel Mar 22, 2017

https://api.crossref.org/journals?rows=1&offset=20000 illustrates the issue.

{
  "status": "ok",
  "message-type": "journal-list",
  "message-version": "1.0.0",
  "message": {
    "items-per-page": 1,
    "query": {
      "start-index": 10000,
      "search-terms": null
    },
    "total-results": 52135,
    "items": [
      {
        "title": "Recherche et Applications en Marketing (English Edition)",
        "publisher": "SAGE Publications",
        "ISSN": [
          "2051-5707",
          "2051-5707"
        ],
        "flags": null,
        "coverage": null,
        "breakdowns": null,
        "counts": null,
        "last-status-check-time": null
      }
    ]
  }
}

The user is expecting the 20001st journal, but instead received the 10001st journal.

As a temporary fix until cursor is more broadly available, would it be possible to lift the offset limit?

dhimmel commented Mar 22, 2017

https://api.crossref.org/journals?rows=1&offset=20000 illustrates the issue.

{
  "status": "ok",
  "message-type": "journal-list",
  "message-version": "1.0.0",
  "message": {
    "items-per-page": 1,
    "query": {
      "start-index": 10000,
      "search-terms": null
    },
    "total-results": 52135,
    "items": [
      {
        "title": "Recherche et Applications en Marketing (English Edition)",
        "publisher": "SAGE Publications",
        "ISSN": [
          "2051-5707",
          "2051-5707"
        ],
        "flags": null,
        "coverage": null,
        "breakdowns": null,
        "counts": null,
        "last-status-check-time": null
      }
    ]
  }
}

The user is expecting the 20001st journal, but instead received the 10001st journal.

As a temporary fix until cursor is more broadly available, would it be possible to lift the offset limit?

@dhimmel

This comment has been minimized.

Show comment
Hide comment
@dhimmel

dhimmel Mar 22, 2017

Also I think it makes sense to throw an error when the user attempts an offset that is too large. Currently, the behavior silently introduces bugs and is hard to diagnose.

dhimmel commented Mar 22, 2017

Also I think it makes sense to throw an error when the user attempts an offset that is too large. Currently, the behavior silently introduces bugs and is hard to diagnose.

@dhimmel

This comment has been minimized.

Show comment
Hide comment
@dhimmel

dhimmel Jun 6, 2017

Just reran the query from above and got:

{
  "status":"ok",
  "message-type":"journal-list",
  "message-version":"1.0.0",
  "message":{
    "items-per-page":1,
    "query":{
      "start-index":20000,
      "search-terms":null
    },
    "total-results":52135,
    "items":[
      {
        "title":"International Journal of Family Therapy",
        "publisher":"Springer-Verlag",
        "ISSN":[
          "0148-8384"
        ],
        "flags":null,
        "coverage":null,
        "breakdowns":null,
        "counts":null,
        "last-status-check-time":null
      }
    ]
  }
}

So issue appears to be fixed. From other issues:

From #220 (comment) by @danielopitz:

The fix seems to be implemented in the code already (see recent commits in the CrossRef/cayenne repository, but the code wasn't deployed yet.

From #208 (comment) by @kjw:

Fix is due to be deployed in the next release, in the next couple of days.

The commits that fix this appear to be CrossRef/cayenne@f0127be. Thanks @kjw! Do you want to close this issue since it's now resolved? Also for the future, consider linking to issue in commit message. For example, if you add Closes CrossRef/rest-api-doc#179, GitHub will automatically link and close the issue. Cheers.

dhimmel commented Jun 6, 2017

Just reran the query from above and got:

{
  "status":"ok",
  "message-type":"journal-list",
  "message-version":"1.0.0",
  "message":{
    "items-per-page":1,
    "query":{
      "start-index":20000,
      "search-terms":null
    },
    "total-results":52135,
    "items":[
      {
        "title":"International Journal of Family Therapy",
        "publisher":"Springer-Verlag",
        "ISSN":[
          "0148-8384"
        ],
        "flags":null,
        "coverage":null,
        "breakdowns":null,
        "counts":null,
        "last-status-check-time":null
      }
    ]
  }
}

So issue appears to be fixed. From other issues:

From #220 (comment) by @danielopitz:

The fix seems to be implemented in the code already (see recent commits in the CrossRef/cayenne repository, but the code wasn't deployed yet.

From #208 (comment) by @kjw:

Fix is due to be deployed in the next release, in the next couple of days.

The commits that fix this appear to be CrossRef/cayenne@f0127be. Thanks @kjw! Do you want to close this issue since it's now resolved? Also for the future, consider linking to issue in commit message. For example, if you add Closes CrossRef/rest-api-doc#179, GitHub will automatically link and close the issue. Cheers.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment