HTTP 500 Error with DOI

[stefanbringuier]

I'm noticing issues since the bump in HTTP version in v0.3.3 commit d1a4991. The response times out due to:

HTTP.Exceptions.StatusError(500, "GET", "/index/api/v1/metadata/10.1186/1756-8722-6-59", HTTP.Messages.Response:

"""
HTTP/1.1 500 Internal Server Error
Content-Type: application/json
Set-Cookie: webpy_session_id=502faf465e77c8c7eae682df0304f4716085fd6c; Path=/; httponly
Content-Length: 43
Date: Thu, 02 Feb 2023 18:59:12 GMT
Server: lighttpd/1.4.53
HTTP status code 500: Internal Server Error"""

)

However, I'm suspecting its not HTTP.jl but changes to the opencitations REST API ? Currently

ShortCodes.jl/src/doi.jl

Line 52 in b0e8bf2

r = HTTP.get("https://opencitations.net/index/api/v1/metadata/$(doi)")

is using metadata in the link, but it looks like it may need to be references, I'm not sure. Thanks for the useful package!

[hellemo]

Thanks for reporting. It should be the right one according to the documentation:
https://opencitations.net/index/coci/api/v1#/metadata/{dois}
I'm guessing it's a server issue, will check back in a couple of days.

[stefanbringuier]

@hellemo I'm still experiencing this issue, is it the same on your end?

[hellemo]

I still have the same problem. I checked with opencitations, and they will look into it. They say it only happens occationally, but for me it is very consistent since you raised the issue.

[AshtonSBradley]

shortcodes DOI gives me this

Failed to show value:

HTTP.Exceptions.StatusError(500, "GET", "/index/api/v1/metadata/10.1137/141000671", HTTP.Messages.Response:

"""

HTTP/1.1 500 Internal Server Error

Content-Type: application/json

Set-Cookie: webpy_session_id=1729cd4b01e1be0b30683b759306df1dadca9775; Path=/; httponly

Content-Length: 43

Date: Thu, 16 Feb 2023 21:31:37 GMT

Server: lighttpd/1.4.53

HTTP status code 500: Internal Server Error""")

[AshtonSBradley]

I still have the same problem. I checked with opencitations, and they will look into it. They say it only happens occationally, but for me it is very consistent since you raised the issue.

yeah, it seems to be permanent for DOI. For twitter it is intermittent, but frequent.

[AshtonSBradley]

I still have the same problem. I checked with opencitations, and they will look into it. They say it only happens occationally, but for me it is very consistent since you raised the issue.

do they have an issue open for it?

[hellemo]

do they have an issue open for it?

This looks like a relevant issue:
opencitations/api#7

[AshtonSBradley]

this is a real bummer... so much that now depends on ShortCodes... how can I help fix this?

[AshtonSBradley]

Maybe i misunderstand the issue, but can I suggest rolling back the HTTP version bump for ShortCodes, d1a4991, until opencitations/api#7 is fixed?

[stefanbringuier]

this is a real bummer... so much that now depends on ShortCodes... how can I help fix this?

This is indeed unfortunate. I've tried HTTP.get calls to a few of the different REST API flavors; all time out. I'm wondering if obtaining a API access token would solve this; although its not required.

Maybe i misunderstand the issue, but can I suggest rolling back the HTTP version bump for ShortCodes, d1a4991, until opencitations/api#7 is fixed?

I don't think this was the issue but you could test it out by creating a Julia project environment:

julia>]
(@v1.8) pkg> activate --temp
(@v1.8) pkg> add ShortCodes@0.3.3

[AshtonSBradley]

thanks for the info, I will try it with a token

[AshtonSBradley]

bit of a newb question, but any advice as to how to insert a token to ShortCodes API calls?

[stefanbringuier]

bit of a newb question, but any advice as to how to insert a token to ShortCodes API calls?

As far as I can see in doi.jl there are no functions that support receiving the header (i.e token) keyword for HTTP.get. Therefore, I think what you would need to do is import ShortCodes and add methods that receive two variables with the second being a header/token input for the following functions:

ShortCodes.jl/src/doi.jl

Line 51 in b0e8bf2

@memoize function fetch_metadata(doi)

ShortCodes.jl/src/doi.jl

Line 63 in b0e8bf2

@memoize function fetch_shortdoi(doi::String)

ShortCodes.jl/src/doi.jl

Line 72 in b0e8bf2

@memoize function expand(doi::ShortDOI)

You could also just fork and modify the functions to take a keyword instead.

As a follow-up I tried out my suggestion for using v0.3.3 and I still was getting issues.

[AshtonSBradley]

This points to a new API going forward. Do you see any issues with moving ShortCodes to this?

opencitations/api#7 (comment)

[hellemo]

Thanks, moving to the new API should be doable, initial work here: #12 if anyone wants to try it out.
Apart from testing, some more work will be needed to add the citation counts back, they are not included in the new metadata API call.

[AshtonSBradley]

I think this should reopen as papers with no citations error at present:

using ShortCodes
DOI("10.1103/PhysRev.65.117")

Failed to show value:

BoundsError: attempt to access 0-element JSON3.Array{Union{}, Vector{UInt8}, Vector{UInt64}} at index [1]

throw_boundserror(::JSON3.Array{Union{}, Vector{UInt8}, Vector{UInt64}}, ::Tuple{Int64})@abstractarray.jl:703
checkbounds@abstractarray.jl:668[inlined]
getindex(::JSON3.Array{Union{}, Vector{UInt8}, Vector{UInt64}}, ::Int64)@JSON3.jl:158
var"##fetch_metadata_unmemoized"(::String)@doi.jl:68
#5@Memoize.jl:62[inlined]
get!@iddict.jl:178[inlined]
fetch_metadata@Memoize.jl:61[inlined]
var"##fetch_metadata_unmemoized"(::ShortCodes.DOI{String})@doi.jl:63
#3@Memoize.jl:62[inlined]
get!(::ShortCodes.var"#3#4"{ShortCodes.DOI{String}}, ::IdDict{Any, Any}, ::Any)@iddict.jl:178
fetch_metadata(::ShortCodes.DOI{String})@Memoize.jl:61
getproperty(::ShortCodes.DOI{String}, ::Symbol)@doi.jl:26
emph_author(::ShortCodes.DOI{String})@doi.jl:104
show(::IOContext{IOBuffer}, ::MIME{Symbol("text/html")}, ::ShortCodes.DOI{String})@doi.jl:43
show_richest(::IOContext{IOBuffer}, ::Any)@PlutoRunner.jl:1186
show_richest_withreturned@PlutoRunner.jl:1100[inlined]
format_output_default(::Any, ::Any)@PlutoRunner.jl:1000
#format_output#61@PlutoRunner.jl:1017[inlined]
#53@PlutoRunner.jl:909[inlined]
var"#with_io_to_logs#115"(::Bool, ::Base.CoreLogging.LogLevel, ::typeof(Main.PlutoRunner.with_io_to_logs), ::Main.PlutoRunner.var"#53#56"{Module, ShortCodes.DOI{String}, Dict{Tuple{UInt64, Int64}, Int64}})@PlutoRunner.jl:2424
(::Main.PlutoRunner.var"#119#120"{Bool, Base.CoreLogging.LogLevel, Main.PlutoRunner.var"#53#56"{Module, ShortCodes.DOI{String}, Dict{Tuple{UInt64, Int64}, Int64}}})()@PlutoRunner.jl:2448
with_logstate(::Function, ::Any)@logging.jl:511
with_logger@logging.jl:623[inlined]
#with_logger_and_io_to_logs#118@PlutoRunner.jl:2447[inlined]
var"#formatted_result_of#50"(::Bool, ::typeof(Main.PlutoRunner.formatted_result_of), ::Base.UUID, ::Base.UUID, ::Bool, ::Vector{String}, ::Nothing, ::Module)@PlutoRunner.jl:908
top-level scope@WorkspaceManager.jl:527

DOI("10.1103/PhysRevA.107.010101")

Failed to show value:

BoundsError: attempt to access 0-element JSON3.Array{Union{}, Vector{UInt8}, Vector{UInt64}} at index [1]

throw_boundserror(::JSON3.Array{Union{}, Vector{UInt8}, Vector{UInt64}}, ::Tuple{Int64})@abstractarray.jl:703
checkbounds@abstractarray.jl:668[inlined]
getindex(::JSON3.Array{Union{}, Vector{UInt8}, Vector{UInt64}}, ::Int64)@JSON3.jl:158
var"##fetch_metadata_unmemoized"(::String)@doi.jl:68
#5@Memoize.jl:62[inlined]
get!@iddict.jl:178[inlined]
fetch_metadata@Memoize.jl:61[inlined]
var"##fetch_metadata_unmemoized"(::ShortCodes.DOI{String})@doi.jl:63
#3@Memoize.jl:62[inlined]
get!(::ShortCodes.var"#3#4"{ShortCodes.DOI{String}}, ::IdDict{Any, Any}, ::Any)@iddict.jl:178
fetch_metadata(::ShortCodes.DOI{String})@Memoize.jl:61
getproperty(::ShortCodes.DOI{String}, ::Symbol)@doi.jl:26
emph_author(::ShortCodes.DOI{String})@doi.jl:104
show(::IOContext{IOBuffer}, ::MIME{Symbol("text/html")}, ::ShortCodes.DOI{String})@doi.jl:43
show_richest(::IOContext{IOBuffer}, ::Any)@PlutoRunner.jl:1186
show_richest_withreturned@PlutoRunner.jl:1100[inlined]
format_output_default(::Any, ::Any)@PlutoRunner.jl:1000
#format_output#61@PlutoRunner.jl:1017[inlined]
#53@PlutoRunner.jl:909[inlined]
var"#with_io_to_logs#115"(::Bool, ::Base.CoreLogging.LogLevel, ::typeof(Main.PlutoRunner.with_io_to_logs), ::Main.PlutoRunner.var"#53#56"{Module, ShortCodes.DOI{String}, Dict{Tuple{UInt64, Int64}, Int64}})@PlutoRunner.jl:2424
(::Main.PlutoRunner.var"#119#120"{Bool, Base.CoreLogging.LogLevel, Main.PlutoRunner.var"#53#56"{Module, ShortCodes.DOI{String}, Dict{Tuple{UInt64, Int64}, Int64}}})()@PlutoRunner.jl:2448
with_logstate(::Function, ::Any)@logging.jl:511
with_logger@logging.jl:623[inlined]
#with_logger_and_io_to_logs#118@PlutoRunner.jl:2447[inlined]
var"#formatted_result_of#50"(::Bool, ::typeof(Main.PlutoRunner.formatted_result_of), ::Base.UUID, ::Base.UUID, ::Bool, ::Vector{String}, ::Nothing, ::Module)@PlutoRunner.jl:908
top-level scope@WorkspaceManager.jl:527