Monitoring dashboard

[nathancahill]

Wanted to open discussion here around a monitoring server. Idea for an API:

const ow = new OpenWorkflow({ backend })
ow.serve({
    port: 4594,
    // host, etc
})

Sketch of an HTTP server with Hono:

import { Hono } from 'hono'

const app = new Hono()

app.get('/api/metrics', (c) => {
    return this.backend.getWorkflowMetrics(params: GetWorkflowMetricsParams)
})

app.get('/api/workflows', (c) => {
    return this.registeredWorkflows
})

app.get('/api/workflows/runs', (c) => {
    return this.backend.listWorkflowRuns(params: ListWorkflowRunsParams)
})

app.get('/api/workflows/runs/:id', (c) => {
    return this.backend.getWorkflowRun(params: GetWorkflowRunParams)
})

// Serve SPA frontend built with vite
app.use('/*', serveStatic({ root: './dist/public' }))
app.get('*', serveStatic({ path: '/index.html', root: './dist/public' }))

Adds listWorkflowRuns() and getWorkflowMetrics() to the backend interface. Probably want to explore adding listWorkflows() and getWorkflow() as well (registeredWorkflows is stored in memory rather than persisted to the backend).

Could interface with a future CLI rather than spawning from JS:

$ ow serve --port 4594

[mariusflorescu]

I believe the idea of a monitoring dashboard is very nice, but what you suggest here is a monitoring API.

What could be done is to expose in the backend some functions that can be transformed into an API by the developer, but not something that openworflow should provide.

A monitoring dashboard would be really nice, especially for development purposes.

Having something like npx ow dev that opens a full-stack application (Next.js, Tanstack start) where you can view your workflows, runs, steps, do some retries, view errors, etc... would be really useful (and also maybe triggering workflows on demand).

But, in order to achieve this many things should be done prior to that, like:

Extending the backend

For this to be achieved, there are many methods that should be available from the class instance

Configuration file

Having a configuration file like openworkflow.config.ts would be really useful for the CLI, because it can provide useful information like:

• DATABASE_URL -> by default OpenWorflow could look for a process.env.DATABASE_URL, but you could just pass it if that's not what you have in the app
• port -> for the port where the app could start
• I guess also directory for workflows would make sense (to know when to refresh the dashboard)

Something like this:

// openworkflow.config.ts

export default {
  // the monitoring dashboard would look for changes here and refresh if needed
  dir: 'src/workflows'
  // optional, by default the monitoring dashboard would look for `DATABASE_URL`
  environment: {
    databaseUrl: process.env.MY_OTHER_DATABASE_URL
  }
  // also optional, if the port that OpenWorkflow tries to use is unavailable
  development: {
    port: 1234
  }

A CLI that spins up the web app

Once those are done, what's left IMO would be just to create the API calls (wrappers over the OpenWorkflow class) and UI for it, since it should have access already to the DB.

[nathancahill]

OpenWorkflow is backend agnostic, so the data must come from the OpenWorkflow client rather than a direct connection to a database.

Indeed we need to add a few more methods to the backend interface: listWorkflowRuns() and getWorkflowMetrics() at a minimum.

Once the data is exposed a frontend is fairly straightforward.

As far as a CLI and config file go, it makes sense to expose the ow client as a named export from openworkflow.config.ts, which can then be imported and interfaced with from the CLI and doesn't limit the project structure to any certain patterns:

// openworkflow.config.ts

import { ow } from "$lib/openworkflow"

export default {
    ow,
    port: 1234
}

Then the implementation of the CLI is simple too:

// bin.ts

import { ow, port } from resolveConfigFile('./openworkflow.config.ts')

if (cmd === 'serve') {
    ow.serve({ port })
}

[jamescmartinez]

I like how you all are thinking about this, it's pretty clever. A few things I am pretty sure about:

• I like the idea of npx openworkflow dev (or dashboard or ui?) that spins up a UI where you can see workflows, runs, steps, errors, retries, and even trigger workflows
• OpenWorkflow should stay backend agnostic, so I agree the source of truth should be the OpenWorkflow client, not outside DB queries
• we definitely need more read APIs on the backend interface: listWorkflowRuns and getWorkflowMetrics or similar. Deciding on what we want to show in the dashboard and working backwards from there would be a great first step

One thing I am cautious about is putting ow.serve directly on the core client if it means pulling an HTTP framework into every app that uses OpenWorkflow. My instinct is that if we have to add dependencies, it probably belongs in a small companion package or as a dev only dependency, but I am open to being convinced otherwise since that makes the DX worse.

It's also likely I'm missing something clever we can do here, too. I need to look at what other projects do.

Curious to hear what you both think!

[nathancahill]

The HTTP dependency can be a peer dependency with a dynamic import. Calling serve without it installed would show a message to install the package.

[jamescmartinez]

The HTTP dependency can be a peer dependency with a dynamic import. Calling serve without it installed would show a message to install the package.

clearly I'm over here overthinking things! This seems like a good approach.

[mariusflorescu]

so the data must come from the OpenWorkflow client rather than a direct connection to a database.

I said, that the backend must be extended (the openworkflow class), so you can build APIs that wrap functionality, and not querying the database directly at your end.

The HTTP dependency can be a peer dependency with a dynamic import. Calling serve without it installed would show a message to install the package.

@jamescmartinez @nathancahill I think this is a bad idea.
You misunderstood me.
The monitoring dashboard should not be part of the openworkflow package or of the backend. It must be part of a @openworkflow/cli package, such that we don't get limited by what we can use or not.

One thing I am cautious about is putting ow.serve directly on the core client if it means pulling an HTTP framework into every app that uses OpenWorkflow.

If we look at how other solutions do this (inngest, trigger), they do it in the CLI package because there you don't need to care about dependencies being shipped to the package. That's why I said a openworkflow.config.ts is needed, because the CLI will automatically look for that file in order to search for on what port should it open, what env variable to use and how to do HMR.

So basically the CLI will do things like:

• npx @openworkflow/cli init -> can install your dependency in package.json + create the bare minimum config file
• npx @openworkflow/cli dev -> start up monitoring app, which should be a fullstack app with UI and all the data fetching, in order to make it easy for the users to see what went wrong and etc...
• other commands like npx @openworkflow/cli getWorkflows --namespace-id "NAMESPACE_ID_HERE"

If we do it in the core package, we become very limited + bad DX for users to install peer deps. The CLI package won't ever be installed by them so they don't care about dependencies, size and other things like that.

[jamescmartinez]

The monitoring dashboard should not be part of the openworkflow package or of the backend. It must be part of a @openworkflow/cli package, such that we don't get limited by what we can use or not.

I think it ends up working the same way, right? Say we build it as a separate @openworkflow/cli package, we can still have a bin script with a dynamic import that prompts you to install @openworkflow/cli if it isn't installed, or uses it if it is, keeping dependencies out of the main package.

[mariusflorescu]

@jamescmartinez yes, the first time you run npx @openworkflow/cli {command} you'll be prompted, but this is much more common for nowadays packages rather than having to install peer deps into your project. Anyhow, @openworkflow/cli won't ever appear into your package.json deps.

For me, as someone who used tools that do tasks/workflows this is the most common way I've seen it implemented.

[nathancahill]

I think we're conflating the two features, one is exposing data for a dashboard to consume and the other is a CLI tool (including serve command). No matter how the CLI tool is built or installed we need OW to expose the data first.

[jamescmartinez]

I'm working on adding lists + pagination to the Backend interface; mentioned in passing here. Metrics will come later.

We also need to sort out how we want to package any external APIs/UIs/CLIs before anyone starts building them.

[nathancahill]

We also need to sort out how we want to package any external APIs/UIs/CLIs before anyone starts building them.

Took a first pass here, very open to ideas and suggestions: #30

[mariusflorescu]

I think we're conflating the two features, one is exposing data for a dashboard to consume and the other is a CLI tool (including serve command). No matter how the CLI tool is built or installed we need OW to expose the data first.

Yeap, we need to expose data but a good question is how we exactly do that.

I'm looking at your referenced PR and my first inconvenience is that the config files receives an ow instance. This is not ideal since projects use config files outside of src folder.

EDIT1:

"Install @openworkflow/dashboard to enable observability:\n\n" +
          "npm install @openworkflow/dashboard\n",

I don't see it working well this way. I'd be pissed as a developer to see this. The CLI should have dashboard as a dependency so the user does not need to install the dashboard, so when you do server everything just works. It's enough for a user to install the CLI.