Allow handling requests in multiple tasks

[denglerchr]

I'm using a Julia webserver to provide some data to a Grafana dashboard. Opening a dashboard immediately sends multiple requests to the server, some of which take >1s to complete, therefore I was trying to make the request handling work in multiple threads, as currently one can see the panels being filled with a delay one after the other. I couldnt figure it out though, it seems that HTTP.jl requires the response of one request being sent before accepting a new request. My guess is it can't be done currently and would require new functionality, if Im wrong please correct me. Therefore I would like to propose to add this functionality in some way for some future version of HTTP.
I could imagine something like this

# julia -t 8
using HTTP

const requestChannel = Channel{HTTP.Request}(8)

# "autorespond" option doesnt exist yet
HTTP.serve(; autorespond = false) do request
    put!(requestChannel, request)
    return nothing
end

function handleRequests()
    while true
        request = take!(requestChannel)
        sleep(5) # work being done
        response = HTTP.Response(200)
        HTTP.respond(request, response) # this function doesnt exist
    end
end

for i = 1:8
    Threads.@spawn handleRequests()
end

[fredrikekre]

Reqests are already handled in their own tasks (@async and not Threads.@spawn though), so you can already do something like

HTTP.serve(...) do request
    t = Threads.@spawn handle_request(request)    
    return fetch(t)
end

I think?

[denglerchr]

I actually tested that, as well as a few other things, but it never seems to run in parallel. I ran this code with 4 threads, then open 2 browser windows, go to localhost:8083 and refresh both at ~the same time. The 5 second timer elapses twice, the requests are not executed in parallel it seems.

using HTTP, Sockets, Dates

function handlerequest(request)
    sleep(5)
    return HTTP.Response(200, "Done, $(Dates.now())")
end

HTTP.serve(Sockets.localhost, 8083; stream = false) do request
    t = Threads.@spawn handlerequest(request)
    return fetch(t)
end

Edit: update example to show the 5s delay

[fredrikekre]

I see. I played around a bit and I think the problem is that tasks may spawn on thread 1 so the "server code" is blocked by computations. To make sure to only spawn on "workers threads" you can use the ThreadPools.jl package. For example using the spawnbg function:

run.jl:

@assert Threads.nthreads() > 1

using HTTP, Sockets, ThreadPools


function handle_request(_)
    sleep(5)
    return HTTP.Response(200, "Hello from thread #$(Threads.threadid()).")
end

HTTP.serve(Sockets.localhost, 8083; access_log=combined_logfmt) do request
    t = ThreadPools.spawnbg() do
        handle_request(request)
    end
    return fetch(t)
end

started as

$ julia --threads=4 run.jl

and then in another session

julia> @time @sync begin
           for i in 1:9
               @async begin
                   r = HTTP.get("http://localhost:8083")
                   println(String(r.body))
               end
           end
       end
Hello from thread #2.
Hello from thread #3.
Hello from thread #4.
Hello from thread #2.
Hello from thread #3.
Hello from thread #4.
Hello from thread #3.
Hello from thread #2.
Hello from thread #4.
 15.044383 seconds (3.77 k allocations: 483.294 KiB, 0.13% compilation time)

so 9 requests to 3 workers threads give expected 15 seconds.

[fredrikekre]

@quinnj discusses a slightly different setup here: https://youtu.be/uLhXgt_gKJc?t=9543 (from 2:39:00).

[denglerchr]

Thanks for the help, it seems that my testing method in the browser was the problem. When using 2 separate windows of chromium, the requests are handled in sequence, but testing it from Julia using HTTP.get, it all worked as expected, even my initial code is then working, using Base.Threads. I actually get 5 seconds all on thread 2:

using HTTP, Sockets, Dates

function handlerequest(request)
    sleep(5)
    return HTTP.Response(200, "Done on thread $(Threads.threadid()), $(Dates.now())")
end

HTTP.serve(Sockets.localhost, 8083; stream = false) do request
    t = Threads.@spawn handlerequest(request)
    return fetch(t)
end

returns

Done on thread 2, 2022-01-24T13:55:05.045
Done on thread 2, 2022-01-24T13:55:05.045
Done on thread 2, 2022-01-24T13:55:05.045
Done on thread 2, 2022-01-24T13:55:05.045
Done on thread 2, 2022-01-24T13:55:05.045
Done on thread 2, 2022-01-24T13:55:05.045
Done on thread 2, 2022-01-24T13:55:05.045
Done on thread 2, 2022-01-24T13:55:05.045
Done on thread 2, 2022-01-24T13:55:05.045
  5.026398 seconds (4.13 k allocations: 365.882 KiB, 0.15% compilation time)

Ill close, since it clearly works, not sure what Chromium is doing though