Expose basic feed queue stats from Status service

[humphd]

In #2395 we are adding an expanded status service dashboard. One thing I want to include in it is the ability to see the current status of the feed queue. That includes things like:

• how many jobs are in the queue?
• how many are active?
• how many have failed?
• how many are delayed?
• how many are waiting?

We use Bull + Redis to manage our feed queue. Bull has a nice API for querying the status for a queue.

Let's add a new route to the Status service that returns details about the current state of the feed queue.

GET /feeds

It can return a complex JS object with various details queried from the queue.

[humphd]

@TueNguyen2911 is going to work on this.

[humphd]

Now that the posts service has the feeds included, I wonder if we should put the new route on that service at /feeds/info, and have it return a JS object with details from the Bull Queue.

@TueNguyen2911 I'd tackle this in pieces (multiple PRs). Get the backend stuff working then add frontend.

[humphd]

We'd need to essentially copy something like https://github.com/Seneca-CDOT/telescope/blob/29951b7b60fa33d83e14b05f24d4a4c2722f9ef7/src/backend/lib/queue.js into the post service so we can query the feed queue and send back JSON data. This one wouldn't need all these event handlers, and could instead use the Bull API for things like this, etc.

Let me know what help you need. @manekenpix knows this code as well (not sure who else does...)

[TueeNguyen]

Got it! I'll ask if something comes up

[TueeNguyen]

This is my idea to tackle with this issue. I wonder if it's headed into the correct direction

First of all, the queue returned from createQueue(name) backend/lib/queue.js is a Bull queue with a ioredis client connection.

I probably can create a new Bull queue with same ioredis connection in the post service using the same logic from backend/lib/queue.js and query the queue's feed using Bull API.

[humphd]

Yes, you're right. The "backend" of the feed is in our Redis database, and you're basically attaching (another) Bull client to it.

I wonder if this should go in the Posts service, or in the Parser service. We don't currently use the parser service, so probably we should start with the change to Posts, and migrate there later if necessary.

[TueeNguyen]

I never looked at the Parser service, it looks like it has what is needed to query the queue's status.

Anyways, I think I'll try adding the change into the Posts service.

[TueeNguyen]

Hi @humphd, I've added a draft PR.