Needed to restart ORCA server after some number of plots

[dgleich]

I encountered an issue where I'd receive a client socket timeout while generating an animation using Plots.jl, PlotlyJS.jl and ORCA.jl.

Here is a smaller version of the example I have that crashes, I tried to simplify but was unable to simplify.

using Plots
using ORCA
using Random
using LinearAlgebra
plotlyjs()
ezsurf!(x, y, f) = begin
    X = repeat(x', length(y), 1)
    Y = repeat(y, 1, length(x))
    # Evaluate each f(x, y)
    Z = map((x,y) -> f([x,y]), X, Y)
    plot!(x, y, Z, st=:surface, alpha=0.4)
end
npts = 100
A = randn(npts,2)
b = randn(npts)
nsamp = 10
k = 25
quadratic_leastsq(A,b) = x-> log10(0.5*norm(A*x - b)^2)/size(A,1)
plot(framestyle=:none)
for j=1:k
  subset = randperm(size(A,1))[1:nsamp] # choose 25 of 200 samples.
  ezsurf!(-3:0.05:3,-50:0.25:250,
    quadratic_leastsq(A[subset,:],b[subset,:]))
end
plot!(colorbar=false,size=(900,900))
##
anim = @animate for i=2:2:360
  plot!(camera=(i,30))
end

If I change the final animation to

anim = @animate for i=2:2:360
  if rem(i,15) == 0
    ORCA.restart_server()
  end
  plot!(camera=(i,30))
end

Note that I had to change rem(i,15) to rem(i,20) or rem(i,10) depending on exactly which plot I was generating.

Then it succeeds.

Possibly also related
JuliaPlots/Plots.jl#2055

[sglyon]

Hmm, thanks for opening. to be totally honest, I'm not sure what would cause this error or how to fix it beyond some sort of manual restart of the server like you do hear?

Perhaps we could automate what you have done here by doing a try/catch inside this package to listen for the client socket timeout and if we get one then restart the server? Do you think that would work? if so, would you be willing to write it up in a PR and use your example as a test case?

[dgleich]

As a temporary pragmatic measure, testing for this timeout and restarting would suffice. It may be a bit before I have time to make that pull request. The only thing I worry about is getting into a restart loop that wouldn't terminate, so I like the try-catch idea just to do it once.

Alternatively, it may be useful to try and track down the resource limit in ORCA itself to see if there would be something else that may be reportable from there. e.g. these reports in the orca process seem somehow related plotly/orca#188

Doing so would involve manually interacting with the ORCA process and enabling verbose reporting. (Or maybe there is some way to log orca messages?)

e.g. what I would do if I had time would be to...

• Store the JSON to generate the plot that ORCA.jl feeds to the server for each image
• Try manually feeding that to the process and get any additional state.

[sglyon]

great ideas. Whenever you have time to make an attempt at that I'll gladly review with you.

Thanks

[dgleich]

Okay, it definitely seems like something in the ORCA process. I managed to debug this a bit... (much easier than expected!)

Steps to debug and get more info.

• Get the path to the ORCA process. (I got this from ps aux | grep orca) -- you want the one with the "--port" command
• Kill any existing orca processes
• Start up ORCA yourself

Then you can overwrite the default savefig from which you can avoid the ensure_server call to skip starting up ORCA.jl's own instance :)

using PlotlyBase, JSON, HTTP

function PlotlyBase.savefig(io::IO,
        p::Plot; format=nothing, scale=nothing,
        width=nothing, height=nothing
    )
    if format === nothing
        error("Must set format when writing to iostream")
    end

    println("Using my function, ", string(format))

    # end early if we got json or html
    format == "json" && return JSON.print(io, p)

    # construct payload
    payload = Dict{Any,Any}(:figure => p, :format=>format)
    scale !== nothing && setindex!(payload, scale, :scale)
    width !== nothing && setindex!(payload, width, :width)
    height !== nothing && setindex!(payload, height, :height)

    # make request to server
    res = HTTP.post(
        "http://localhost:7982", Dict(),
        JSON.json(payload),
        status_exception=false
    )

    # save if success, otherwise report error
    if res.status == 200
        write(io, res.body)
    else
        error(String(res.body))
    end

    return nothing
end

This will get called if you run a plot now.. (you can verify with the output)

Anyway, this shows that Orca is throwing a memory error after 19 plots.

Ideally, this suggests that there'd be some way to reset Orca, but it seems like restarting is almost the official recommendation here.

My quick recommendation: it seems the pragmatic solution is best :), check for the timeout error, restart and retry...

[sglyon]

Thanks for debugging!

I wonder what the people over at the orca repo would say? I'm going to ping a couple of them @antoinerg, @nicolaskruchten (if I missed anyone please tag them here too!)

[sglyon]

Thank you for the great detective work @dgleich

Due to the problems with this package, we have decided to go another direction. The savefig functionality offered by this package has been superseded by new routines built in to PlotlyBase.jl (and thus PlotlyJS.jl) itself.

ORCA.jl is now deprecated and should not be used going forward.

Thank you