UI + Rest API + Webhooks Anyone?

[ashconnell]

We found pg-boss to be a great solution as it allows us to use Postgres and Node for our app and our queue! There's nothing worse than needing to manage another DB eg Redis, or adding another language to your stack.

I immediately noticed a few things that would improve the way we use it so we've decided to build a self-contained server that does all of the following:

1. Runs pg-boss
2. Exposes a Rest API for jobs, eg POST /jobs and DELETE /jobs/:jobId etc
3. Offloads jobs back to our app API servers via webhook (each job is processed elsewhere so that this server remains lazer focused)
4. Exposes a UI for viewing, filtering and searching jobs + shows charts for the number of jobs processed in the past

Our app servers now enqueue jobs by making a request to the pg-boss server, and also process them (when the time comes) via webhook.

We've done a first pass at all of the above and it's looking promising. The whole thing (API, UI and Processing) is powered by Next.js so the code is pretty lightweight.

I'm wondering if this is something others might be interested in?

One thing i'm not sure about is whether people would like the way it processes via webhooks instead of doing the processing on the box. We decided to do this for 2 reasons:

1. Our app servers already have the code/packages needed to process our jobs
2. We like the separation of concerns. The job server doesn't know anything about our domain. It's like having your own Amazon SQS

Thoughts? Questions?

[timgit]

Glad to hear you're finding this as useful as we do! 👍

I would echo your efforts here. I've had a few requests over the years for a UI side project. I've built a simple bar chart in our application to get a good visual of activity, and I'm also hosting my own api for interacting with fetch, complete, cancel and fail operations.

To build the chart, I created an api route that returns a memcache from monitor-states event payload, which I would recommend over any user-initiated queries against the job table.

[ashconnell]

We switched scope on this a little bit and decided to build a dedicated UI without the other bells and whistles. The monitoring, webhooks, slack notifications and api are now handled in another layer.

It's now a simple node server and frontend that gives us the much needed visibility into the queue we really needed, and makes it easier to open source.

Here's some screenshots (we also have charts but their not finalized):

create-job-2.mp4

To summarize what this is:

• Everything runs as a single node server / docker container
• You configure env variables with PG_URI and PG_BOSS_SCHEMA etc
• The server exposes an API to the client for viewing/editing stats and jobs etc
• The client lets you view and interact with your queue, and create jobs etc

Currently thinking of calling this pg-boss-ui but open to suggestions

[timgit]

Love it!

[tcoats]

Hi Folks,

We've implemented a UI for Boss. It requires two components, an API to provide secure access to the Boss queue via a websocket, and UI which is static.

https://github.com/y5labs/boss-api
https://github.com/y5labs/boss-control

The API has been released as a docker container:
https://quay.io/repository/y5labs/boss-api

We're running the UI here:
https://boss-control.y5.nz

When hosted securely the UI can only connect to secure APIs.

To run locally you'll want to pull the two repositories, spin up a PostgreSQL server.

Boss API requires environment variables:

BOSS_PG_HOST=localhost
BOSS_PG_DATABASE=postgres
BOSS_PG_USER=postgres
BOSS_PG_PASSWORD=postgres
BOSS_PG_PORT=5432
BOSS_PG_SCHEMA=public
BOSS_APPLICATION_NAME=boss-test
MIGRATE_ON_STARTUP=1

And can be installed using

npm i
npm run dev

An API key and secret need to be generated. The API runs a telnet server internally. These commands will generate a new key and secret.

telnet localhost
help
api_key_new

Secrets are not stored securely in the database. In a future update we'll salt them.

The UI can be run using these commands

npm i
npm run dev
(New terminal)
npm start

Feedback is welcome.

[phips28]

@ashconnell that looks really nice! Do you have a public repo for that? I would also use that and contribute if I can.

@tcoats great work! can you provide some screenshots to show the functionality of your UI?
edit: I have set it up locally to test :)

I would prefer to see the existing queues with the metrics and then look into each queue to get the jobs. I like the overview in bull-monitor (as we also use bull in other projects) https://s-r-x.github.io/bull-monitor/

[tcoats]

@phips28 Not sure if you've found it yet but you can click on each collection of jobs - e.g. from your screenshot you can click on the fetch_binance_10 active column that currently shows one job with the pulse icon. The detail view shows activity on that single queue and a searchable list of jobs.

We use bull too!. Looking at Bull Monitor the difference is that Boss Control doesn't show all queues when viewing the details of a single queue. We could add this.

Feedback is welcome, thank you. My original design is on Excalidraw if it's of interest.

[timgit]

I'd like to include links to these projects in the readme if you guys are interested ilo of keeping this issue open.

[tcoats]

@timgit sounds good to me. I'll need to update our packages to match 7alpha.

[timgit]

This issue has been linked from the docs now.

[dturton]

We switched scope on this a little bit and decided to build a dedicated UI without the other bells and whistles. The monitoring, webhooks, slack notifications and api are now handled in another layer.

It's now a simple node server and frontend that gives us the much needed visibility into the queue we really needed, and makes it easier to open source.

Here's some screenshots (we also have charts but their not finalized):

create-job-2.mp4

To summarize what this is:

• Everything runs as a single node server / docker container
• You configure env variables with PG_URI and PG_BOSS_SCHEMA etc
• The server exposes an API to the client for viewing/editing stats and jobs etc
• The client lets you view and interact with your queue, and create jobs etc

Currently thinking of calling this pg-boss-ui but open to suggestions

Hi @ashconnell is this available in a public repo?