asyncmap never ends with HTTP.get #909
Description
Hello guys. I believe I found a bug in the asyncmap or the HTTP package. My goal is to test a list of 4881 proxies and find the ones that are working. But asyncmap never seems to finish.
To compare the performance I ran the asyncmap function with two functions. One that has a sleep(5) and one that makes a HTTP.get request with readtimeout=5, retry=false.
The execution time of the function with sleep(5) was 248.891672 seconds. The execution of HTTP.get never ends, it gets slower and slower until it almost stops.
Here is my versioninfo():
Julia Version 1.8.0
Commit 5544a0fab76 (2022-08-17 13:38 UTC)
Platform Info:
OS: Linux (x86_64-linux-gnu)
CPU: 8 × Intel(R) Core(TM) i7-4870HQ CPU @ 2.50GHz
WORD_SIZE: 64
LIBM: libopenlibm
LLVM: libLLVM-13.0.1 (ORCJIT, haswell)
Threads: 8 on 8 virtual cores
Environment:
JULIA_NUM_THREADS = auto
Here is my list of proxies:
https://paste.debian.net/plain/1251148
Here is my test code:
using Base.Threads
using HTTP
using CSV
using Tables
const conf = (readtimeout=5, retry=false)
const url = "https://www.google.com/"
const proxies = CSV.File("proxies.csv") |> Tables.matrix |> vec
function testSleep(proxy)
try
sleep(5)
if resp.status == 200
println(threadid(), " ok ", proxy)
return true
else
println(threadid(), " bad ", proxy)
return false
end
catch
println(threadid(), " fail ", proxy)
return false
end
end
function testHTTP(proxy)
try
resp = HTTP.get(url, proxy="http://$proxy"; conf...)
if resp.status == 200
println(threadid(), " ok ", proxy)
return true
else
println(threadid(), " bad ", proxy)
return false
end
catch
println(threadid(), " fail ", proxy)
return false
end
end
@spawn @time asyncmap(testHTTP, proxies) # never ends
@spawn @time asyncmap(testSleep, proxies) # 248.891672 seconds (19.84 M allocations: 1.270 GiB, 0.23% gc time, 4.33% compilation time: 1% of which was recompilation)