Support for filtering jobs

[xdc0]

It would be nice to filter jobs in a particular state given some criteria. The most useful use case in my opinion is searching for failed jobs whose first error matches some search term.

[krazyjakee]

Great idea.

Sadly, redis does not support value searches but we could hand the entire job set over to the client in the form of a clusterize table which can support hundreds of thousands of rows without a performance hit. A form could then be constructed to perform the filtering and update results in the table. It would need a lot of testing.

If a client-side solution doesn't work, the server would need to create an internal database that syncs with redis and could be queried via ajax. We could also keep and improve pagination this way.

[bradvogel]

Some of our queues are 3GB, so sending all that data to the client for filtering won't work.

Why not have the server just read each job from redis, perform the filter, and then stream results to the client?

[krazyjakee]

@bradvogel ok, 3GB is a great example to think about. It's worth testing at least but I doubt we could have such functionality as an autocomplete. I guess we'd need a syncd database loaded with indexes to achieve that.

[bradvogel]

Nah - it'd have to stream each job (key) from redis and run the filter in the backend, then stream it to the frontend if it matches.

[krazyjakee]

@bradvogel ah! I hear you. Yeah I'm picturing loading indicator, progress bar, options to filter by job data by values & response content. Could be a great addition.

[xdc0]

Note, the intent of this feature request is to only provide basic filtering capabilities when looking at a queue in a particular state, although it would be nice to think on solving that problem in a way that it may open new features as the ones that @krazyjakee is suggesting. But I think it'd be better to solve a small problem at a time.

I'm thinking that a solution for this that is efficient for very large queues is to use SCAN cursors. I'm not sure if bee supports job listing with a cursor, the idea is that since you have to do a full scan for filtering anyway, we can pull 10 jobs in some state like we do with pagination, and then apply the filter on these 10 jobs. Then, if 3 jobs were filtered out of these 10 jobs, then get the next set of 10 jobs with the SCAN cursor until you have a set of 10 jobs.

Pagination can become tricky though because we can't know how many pages a particular search has without doing a full scan which is expensive. I'm thinking that we can replace pagination with a cursor hash that represents the filter and the redis SCAN cursor together, then a "next page" button or a "load more" button will fetch more jobs until the cursor is exhausted. The tradeoff is that we can't know how many results a filter has but I believe that it is fine to exclude traditional pagination when doing searches

[alexisohayon]

Hello!

Is there still a feature ongoing for this issue? :)
Would be very handy!

Thanks for the good work :)

[tevaum]

Any news on this issue?

[yaboi]

ping

[skeggse]

While I understand that this ticket has a lot of interest, please refrain from simply bumping - it's not a priority for us, and we don't really have resources to fully support this project. I'm locking and restricting to collaborators. The best way to get progress on this ticket would be to submit a pull request! Linked it to this ticket :)