Unable to obtain full list of funders

[dlenhardt]

The /funders resource component reports there are currently nearly 14,000 funders:

http://api.crossref.org/funders?rows=0

The /funders resource component can be traversed by using the rows and offset parameters. For example:

http://api.crossref.org/funders?rows=5&offset=5

The rows parameter has a limit of 1000 rows, and the offset parameter has a limit of 10,000 rows. Therefore, one can access up to 11,000 funders. Is there a way to access funders beyond that number? The cursor parameter does not work with the funders resource component.

[AshleyMoore]

I am also finding this issue. I would like to return all works by a single funder. I am unable to step through records once limit is reached as I am unable to use the cursor parameter.

[dhimmel]

I'm having a similar issue with trying to query the journals component. There are currently 52135 journals, but the offset parameter is limited to 10000.

cursor solves this issue, but is only available for the works component. It would be nice to see cursor enabled for more components.

Tagging active maintainers: @kmeddings @gbilder @kjw. Unless we're missing something, this seems like a serious issue... a large portion of funders and journals are unavailable to exhaustive queries.

[dhimmel]

https://api.crossref.org/journals?rows=1&offset=20000 illustrates the issue.

{
  "status": "ok",
  "message-type": "journal-list",
  "message-version": "1.0.0",
  "message": {
    "items-per-page": 1,
    "query": {
      "start-index": 10000,
      "search-terms": null
    },
    "total-results": 52135,
    "items": [
      {
        "title": "Recherche et Applications en Marketing (English Edition)",
        "publisher": "SAGE Publications",
        "ISSN": [
          "2051-5707",
          "2051-5707"
        ],
        "flags": null,
        "coverage": null,
        "breakdowns": null,
        "counts": null,
        "last-status-check-time": null
      }
    ]
  }
}

The user is expecting the 20001st journal, but instead received the 10001st journal.

As a temporary fix until cursor is more broadly available, would it be possible to lift the offset limit?

[dhimmel]

Also I think it makes sense to throw an error when the user attempts an offset that is too large. Currently, the behavior silently introduces bugs and is hard to diagnose.

[dhimmel]

Just reran the query from above and got:

{
  "status":"ok",
  "message-type":"journal-list",
  "message-version":"1.0.0",
  "message":{
    "items-per-page":1,
    "query":{
      "start-index":20000,
      "search-terms":null
    },
    "total-results":52135,
    "items":[
      {
        "title":"International Journal of Family Therapy",
        "publisher":"Springer-Verlag",
        "ISSN":[
          "0148-8384"
        ],
        "flags":null,
        "coverage":null,
        "breakdowns":null,
        "counts":null,
        "last-status-check-time":null
      }
    ]
  }
}

So issue appears to be fixed. From other issues:

From #220 (comment) by @danielopitz:

The fix seems to be implemented in the code already (see recent commits in the CrossRef/cayenne repository, but the code wasn't deployed yet.

From #208 (comment) by @kjw:

Fix is due to be deployed in the next release, in the next couple of days.

The commits that fix this appear to be CrossRef/cayenne@f0127be. Thanks @kjw! Do you want to close this issue since it's now resolved? Also for the future, consider linking to issue in commit message. For example, if you add Closes CrossRef/rest-api-doc#179, GitHub will automatically link and close the issue. Cheers.