Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WebUI not loading Queues #296

Open
jirevwe opened this issue Mar 22, 2023 · 13 comments
Open

WebUI not loading Queues #296

jirevwe opened this issue Mar 22, 2023 · 13 comments

Comments

@jirevwe
Copy link

jirevwe commented Mar 22, 2023

Hi 👋🏿,

I'm on running asynq: 0.23.0 and asynqmon: 0.7.1 in my production environment and I have been unable to access the Web UI, when this happened in the past running flushall in redis fixed it (got the idea from #149), but I can't afford to do that now as I am storing other things in redis.

What do you think caused this (so I can prevent this from happening in the future) and how can I get it to work without clearing all the keys?

I attached a screenshot of the error. Thanks in advance.

Screenshot from 2023-03-22 17-24-38

@netixx
Copy link

netixx commented Sep 13, 2023

I have the same issue, I ran flushall, but it didn't solve my issue either.

@MrRobo-t
Copy link

MrRobo-t commented Sep 14, 2023

Facing the same issue.. It momentarily shows the active task.. That's all I get.. There is no archive task coming in..
@hibiken Can you please help on this? I have build using the docker image .. I am getting graph for task processed though.. The db is deployed on aws elasticache redis..

@netixx
Copy link

netixx commented Sep 14, 2023

I looked at the call graph for the /queues route, and given the error is lua running in redis, my guess is that the issue is in this script:

var memoryUsageCmd = redis.NewScript(`
local sample_size = tonumber(ARGV[2])
if sample_size <= 0 then
    return redis.error_reply("sample size must be a positive number")
end
local memusg = 0
for i=1,2 do
    local ids = redis.call("LRANGE", KEYS[i], 0, sample_size - 1)
    local sample_total = 0
    if (table.getn(ids) > 0) then
        for _, id in ipairs(ids) do
            local bytes = redis.call("MEMORY", "USAGE", ARGV[1] .. id)
            sample_total = sample_total + bytes
        end
        local n = redis.call("LLEN", KEYS[i])
        local avg = sample_total / table.getn(ids)
        memusg = memusg + (avg * n)
    end
    local m = redis.call("MEMORY", "USAGE", KEYS[i])
    if (m) then
        memusg = memusg + m
    end
end
for i=3,6 do
    local ids = redis.call("ZRANGE", KEYS[i], 0, sample_size - 1)
    local sample_total = 0
    if (table.getn(ids) > 0) then
        for _, id in ipairs(ids) do
            local bytes = redis.call("MEMORY", "USAGE", ARGV[1] .. id)
            sample_total = sample_total + bytes
        end
        local n = redis.call("ZCARD", KEYS[i])
        local avg = sample_total / table.getn(ids)
        memusg = memusg + (avg * n)
    end
    local m = redis.call("MEMORY", "USAGE", KEYS[i])
    if (m) then
        memusg = memusg + m
    end
end
local groups = redis.call("SMEMBERS", KEYS[7])
if table.getn(groups) > 0 then
	local agg_task_count = 0
	local agg_task_sample_total = 0
	local agg_task_sample_size = 0
	for i, gname in ipairs(groups) do
		local group_key = ARGV[4] .. gname
		agg_task_count = agg_task_count + redis.call("ZCARD", group_key)
		if i <= tonumber(ARGV[3]) then
			local ids = redis.call("ZRANGE", group_key, 0, sample_size - 1)
			for _, id in ipairs(ids) do
				local bytes = redis.call("MEMORY", "USAGE", ARGV[1] .. id)
				agg_task_sample_total = agg_task_sample_total + bytes
				agg_task_sample_size = agg_task_sample_size + 1
			end
		end
	end
	local avg = agg_task_sample_total / agg_task_sample_size
	memusg = memusg + (avg * agg_task_count)
end
return memusg
`)

Since the issue is the "bytes", then I guess more specifically here:

local bytes = redis.call("MEMORY", "USAGE", ARGV[1] .. id)
            sample_total = sample_total + bytes

running asynqmon v0.7.2
and asynq v0.24.1

This part of the code is asynq code (internal/rdb/inspect.go). All calls to CurrentStats should be impacted as well.

@Psuta90
Copy link

Psuta90 commented Oct 12, 2023

any fix for this issue

@Psuta90
Copy link

Psuta90 commented Oct 12, 2023

image

i got the same issue

@Psuta90
Copy link

Psuta90 commented Oct 12, 2023

the redis connected but cannot get the queues

@Ianmuhia
Copy link

@Psuta90 share logs from the asynqmon server.

@Psuta90
Copy link

Psuta90 commented Oct 13, 2023

@Psuta90 share logs from the asynqmon server.

image

i try to scheduled task is success in terminal manualjobs is succes but in asynqmon is not showing the queues

in the sidebar server is showing but i click is not working like previous picture
image

@Psuta90
Copy link

Psuta90 commented Oct 13, 2023

image

@Psuta90
Copy link

Psuta90 commented Oct 13, 2023

@Psuta90 share logs from the asynqmon server.

"time":"2023-10-13T07:13:46.906285637Z","id":"","remote_ip":"172.18.0.1","host":"localhost:3005","method":"GET","uri":"/monitoring/tasks/api/queues/default/active_tasks?page=1&size=100","user_agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/117.0.0.0 Safari/537.36","status":500,"error":"","latency":491845,"latency_human":"491.845µs","bytes_in":0,"bytes_out":23}

@wahyudibo
Copy link

I also encountered the same issue. The asynqmon can connect to redis but unable to load queue data. I'm using asynq v0.24.1 and asynqmon v0.7.2. Here's the log from asynqmon

2023/10/26 03:38:00 Failed to collect metrics data: failed to get queue info: UNKNOWN: UNKNOWN: redis eval error: ERR user_script:30: attempt to perform arithmetic on local 'bytes' (a boolean value) script: 0c80fc1868e3aae205155ae2017ce995a3a7cec0, on @user_script:30.

thanks

@wahyudibo
Copy link

Just want to give an update. my asynq and asynqmon setup works now after i run go mod tidy

@piavgh
Copy link

piavgh commented Nov 10, 2023

Just want to give an update. my asynq and asynqmon setup works now after i run go mod tidy

does not work for me.
It looks like v0.24 of asynq is not compatible with the asyncmon web UI v0.7.2

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

7 participants