Ratelimiter state machine errors #517
Comments
|
Please retry with the latest master. I haven't seen this error myself before nor could I reproduce it, but it should deal with automatically requeueing requests that were closed abruptly. |
Thanks for the quick reply! I'll test it out and let you know |
@jchristgit I tested and in logs I see now this error: Date: 2023-06-29T14:03:02.570
Level: error
State machine 'Elixir.Nostrum.Api.Ratelimiter' terminating
Last event = {
info, {
gun_down, <0.5653.0>, http2, closed, [#Ref<0.2297931708.4051173377.228586>]
}
}
Server state = {
connected, {
conn => <0.5653.0>,
inflight => #{},
outstanding => {
<<"/users/@me">> => {
initial, {
[{
#{body => <<>>,
headers => [{<<content-type>>, <<application/json>>}],
method => get,
params => [],
route => <<"/users/@me">>
},
{<0.6601.0>, #Ref<0.2297931708.4051173377.228585>}
}], []
}
}
},
remaining_in_window => 48,
running => #{}
}
}
Reason for termination = error: {badkey, #Ref<0.2297931708.4051173377.228586>}
Callback modules = ['Elixir.Nostrum.Api.Ratelimiter']
Callback mode = state_functions
Stacktrace = [
{erlang, map_get, [#Ref<0.2297931708.4051173377.228586>, #{}], [{error_info, #{module => erl_erts_errors}}]},
{'Elixir.Nostrum.Api.Ratelimiter', '-connected/3-fun-1-', 3, [{file, "lib/nostrum/api/ratelimiter.ex"}, {line, 881}]},
{'Elixir.Enum', '-map/2-lists^map/1-0-', 2, [{file, "lib/enum.ex"}, {line, 1658}]},
{'Elixir.Nostrum.Api.Ratelimiter', connected, 3, [{file, "lib/nostrum/api/ratelimiter.ex"}, {line, 878}]},
{gen_statem, loop_state_callback, 11, [{file, "gen_statem.erl"}, {line, 1426}]},
{proc_lib, init_p_do_apply, 3, [{file, "proc_lib.erl"}, {line, 240}]}
]
Time-outs: {1, [{{timeout, reset_bot_calls_window}, expired}]} |
|
I pushed another commit to master, with the same caveats: I cannot really test it. Are you on an unreliable network? Or do you have another clue why these may keep appearing? |
@jchristgit Thank you! I'll test it out tomorrow and let you know. The elixir app is deployed to AWS, so there shouldn't be any problem with the network. |
@jchristgit Sometimes after To work around this I have wrapped it in a Task like this: defp warm_up(msg_or_interaction, retries \\ 3) do
t1 = System.monotonic_time(:millisecond)
log(msg_or_interaction, "WARM UP STARTING...")
task = Task.async(fn -> Nostrum.Api.get_current_user() end)
case Task.yield(task, 5000) || Task.shutdown(task) do
nil ->
log(msg_or_interaction, "WARM UP ERROR: Timeout reached.", type: :error)
if retries > 0 do
log(msg_or_interaction, "WARM UP TIMEOUT: Retrying...", type: :error)
warm_up(msg_or_interaction, retries - 1)
else
log(msg_or_interaction, "WARM UP TIMEOUT: No more retries.", type: :error)
end
{:ok, result} ->
# handle the result
handle_result(result, msg_or_interaction, retries)
end
t2 = System.monotonic_time(:millisecond)
log(msg_or_interaction, "Time spent warming up #{t2 - t1}ms.")
endHere are some log excerpts: |
|
I cannot reproduce this here, not with spamming the endpoint nor with
running it one by one like you do.
Do you have a proxy sitting in front of nostrum? Some outgoing firewall?
|
@jchristgit The elixir app is deployed in kubernetes cluster in AWS. |
|
Sorry but I have no environment where I can reproduce this, at all. Since you mentioned that the only difference between stage and prod is the AWS NAT Gateway I cannot help but assume that the AWS NAT Gateway is at fault, and nostrum works properly here. Perhaps the NAT Gateway is messing around with HTTP/2 connections or it is doing some other strange form of introspection on the request. In nostrum the bot gateway request does not do anything out of the ordinary in terms of HTTP requests. |
|
@jchristgit My current workaround is having a job that makes an API call every couple of minutes so it makes sure that the connection doesn't get idle. Thank you for quick responses and the fixes, really appreciated! 💙 I am closing the issue. |
I got these Ratelimiter errors in the logs after upgrading to latest master.
Edit: these seems to be frequent in the logs
Nostrum version:
"nostrum": {:git, "https://github.com/Kraigie/nostrum.git", "49a2944d7e1a953a4cafae18b97e9379da36f3bb", []},, tag version:nostrum 0.9.0-alpha3Elixir and otp versions:
Traceback
The text was updated successfully, but these errors were encountered: