500 Internal Server Error

[electronsandstuff]

Hey folks, thanks for the great resource! I encounter an HTTP 500 error when I use the API at the following URL.

https://opencitations.net/index/coci/api/v1/references/10.1103/physrevaccelbeams.22.033401

I saw this github repo linked on the COCI API reference, but let me know if this isn't the right spot to report issues.

It looks like information does exist on this DOI since when I use the search feature on the homepage, citations are returned.

[AshtonSBradley]

Just want to add that all DOI requests from shortcodes are now failing after a lengthy timeout.

hellemo/ShortCodes.jl#11

[AshtonSBradley]

Does this require a token now? When I try to get one, I get this, so I am at a loss as to how to proceed. Are http get API calls supported at all anymore?

[essepuntato]

Hi all,

The HTTP 500 Error you have got was a temporary problem with our API – we got too many requests when @electronsandstuff tried, which resulted in a 500. I've just tried to call the API with the URL specified at the beginning of this issue, and it returned a result as expected.

Instead, @AshtonSBradley, for what concerns the access token: the API works even if you do not have an access token, even if we kindly ask users to get and use one - more information can be found at our blog.

Back to the @AshtonSBradley code linked in the previous comment: we are indeed monitoring the operation "metadata" in the last weeks since it seems to be very slow also from our end. We have yet to identify the issue for that. As an alternative (which will be the stable way for doing that from now on), I suggest using OpenCitations Meta API (https://w3id.org/oc/meta/api/v1) for retrieving metadata of a given DOI. E.g.:

https://w3id.org/oc/meta/api/v1/metadata/doi:10.1103/physrevaccelbeams.22.033401

I hope it may help.
Have a nice day :-)

S.

[ivanhb]

Hi all,
We have updated our system and improved it in order to handle the reported problems/errors.
The performances have been significantly improved.

I hope this will prevent future problems like the ones you have encountered,
Please let us know in case of any other problems,
Have a nice day,
Ivan

[AshtonSBradley]

Is there a reason that the meta API has more incomplete metadata on doi requests? At present it seems quite frequent that the metadata is incomplete for records that had full metadata under the old API

hellemo/ShortCodes.jl#13

[essepuntato]

Hi @AshtonSBradley,

Copying @arcangelo7 and @ivanhb here. Can you provide some examples?
Have a nice day :-)

S.

[AshtonSBradley]

Hi @AshtonSBradley,

Copying @arcangelo7 and @ivanhb here. Can you provide some examples? Have a nice day :-)

S.

Hi @arcangelo7,

There is an issue here hellemo/ShortCodes.jl#13

some more examples that worked under the old API and now throw the same error

10.1103/PhysRev.65.117
10.1103/PhysRevA.107.010101
10.1016/S0167-2789(00)00094-4

[AshtonSBradley]

Hi @essepuntato
I gave some examples above. They are quite common. In practice I find that where previously opencitations was very reliable with broad coverage, now the coverage has narrowed enough that I can't use it as a practical tool anymore. Can you tell me anything about plans (or otherwise) to widen the metadata coverage in the new API to match the previous reliability?

[essepuntato]

Hi @AshtonSBradley,

I've checked all the examples, and what is happening is clear from our side. Thus, to provide a bit of context, before all metadata were retrieved on the fly by calling external APIs, all the information about citation counts was handled in-house. This means you retrieved the metadata for the abovementioned cases even when no citation information was available, since we called Crossref to get them.

However, since February, we have been revising the ingestion workflow of OpenCitations, and that has resulted in some specific activities:

1. The metadata of all the entities involved in the citations we store comes from OpenCitations Meta (https://opencitations.net/meta), our in-house solution for storing basic bibliographic metadata of such entities. Currently, we have only metadata of bibliographic resources involved in citations – meaning that if no one cites a resource for which you are asking the metadata or does not cite anyone, it is not included currently.
2. In February, we started to re-engineer the ingestion process to make it more smooth now that we do include several sources (not only Crossref) for feeding our collections. Such modification took four months of work; during this period, we could not update any of the citation indexes. However, we are releasing this new ingestion code and process in June and will be back on track with regular updates. In addition to Crossref, you can retrieve citation data included in DataCite, NIH-OCC, and OpenAIRE.
3. We have started a process to align our data to OpenAlex, to improve the interoperability of these two open datasets.
4. We are updating the technical infrastructure (i.e. servers) to offer better services (new APIs are on the way that will include the possibility of text search for titles and author names on the collections) and to programme an extension of the current coverage of bibliographic resources we host. The goal, thus, is to include any bibliographic resource we can take from the sources independently from being cited or not.

Thus, the reason for not getting metadata for the examples above depends mainly on 2). Indeed, in Crossref right now, these entities include citation information to other entities, and they will be ingested in the next ingestion of Crossref planned for July/August.

I hope that this explains the current situation.
Thanks again for using OpenCitations data!