StepsTrack is a tool built to help tracking, visualizing and inspecting intermediate steps in a complex pipeline-based application. It automatically captures and stores the intermediate data, results and execution times of each steps in a pipeline, visualizing the execution details and allowing easier debug or analysis in a monitoring dashboard. It is originally developed as a go-to tool to inspect and analyze runtime data of an agentic RAG pipeline.
Background of StepsTrack
StepsTrack is a lightweight inspection and debugging tool originally built to monitor an agentic Retrieval-Augmented Generation (RAG) pipeline running in a production environment—where visibility, performance, and stability are critical.
When chaining multiple LLM agents with custom logic and dynamic inputs, non-deterministic nature of LLM outputs of each steps often lead to dynamic route of logics and behaviors. I needed a inspection tool but the existing tools didn’t provide the granularity I needed to trace what happened inside each step of the pipeline.
So I built StepsTrack to do just that: trace, inspect, and understand every step of each request. It helped me quickly spot bottlenecks, unexpected behaviors and performance drags, and address them effectively.
I'm open-sourcing it in the hope that it helps others building and operating complex LLM pipelines.
Contributions welcome!
This repository is a monorepo containing following packages:
- Typescript library that provides basic tracker and chart generation function for your pipeline
- Dashboard that visualizes and allows you to monitor tracked data for analysis.
- Tracking: Define steps in pipeline to track intermediates data, results and execution time
- Visualizing: Exporting the details and generating visualizations
- Event Emitting: Listen to step events for real-time monitoring and custom handling
- ES6 Decorators: Easy integration with ES6 decorators
Monitor and analyze pipeline executions through an interactive web interface
- Detailed Steps Insepection
- Real-time Execution Monitoring
- Gantt Chart Visualization for pipeline
- Step Execution Stats
Note: StepsTrack is designed for any pipeline-based / multi-steps logic, especially agentic LLM pipelines
npm install --save steps-trackCreate a pipeline and track steps with nested, sequential, or parallel logic:
import { Pipeline, Step } from 'steps-track';
const pipeline = new Pipeline('my-pipeline');
await pipeline.track(async (st: Step) => {
// Track a simple step
await st.step('step1', async (st: Step) => {
// Logic for step1
st.record('key', 'value'); // Record data for analysis
});
// Track nested steps
await st.step('parent-step', async (st: Step) => {
await st.step('child-step-1', async (st: Step) => {
// Child step logic
});
await st.step('child-step-2', async (st: Step) => {
// Child step logic
});
});
// Track parallel steps
await Promise.all([
st.step('parallel-1', async (st: Step) => { /* ... */ }),
st.step('parallel-2', async (st: Step) => { /* ... */ })
]);
});Generate visual outputs to understand and analyze execution flow:
// Generate a Gantt chart Buffer using quickchart.io
const ganttChartBuffer = await pipeline.ganttQuickchart();
// Generate a Gantt chart HTML file with Google Charts
const ganttChartHtml = await pipeline.ganttGoogleChartHtml();
// Generate an execution graph URL
const executionGraphUrl = pipeline.executionGraphQuickchart();
// Get the hierarchical output of all steps
const stepsHierarchy = pipeline.outputHierarchy();Sample Gantt Chart
Sample Execution Graph
Sample Hierarchy Output
json
{
"name": "document-parse",
"key": "document-parse",
"time": { "startTs": 1739357985509, "endTs": 1739357990192, "timeUsageMs": 4683 },
"records": {},
"substeps": [
{
"name": "preprocess",
"key": "document-pipeline.preprocess",
"time": { "startTs": 1739357985711, "endTs": 1739357986713, "timeUsageMs": 1002 },
"records": {
"pageCount": 3
},
"result": [ "page_1_content", "page_2_content"],
"substeps": []
},
{
"name": "parsing",
"key": "document-pipeline.parsing",
"time": { "startTs": 1739357985711, "endTs": 1739357990192, "timeUsageMs": 4481 },
"records": {},
"substeps": [
{
"name": "page_1",
"key": "document-pipeline.parsing.page_1",
"time": { "startTs": 1739357987214, "endTs": 1739357990192, "timeUsageMs": 2978 },
"records": {},
"result": "page_1_content",
"substeps": []
},
{
"name": "page_2",
"key": "document-pipeline.parsing.page_2",
"time": {
"startTs": 1739357987214, "endTs": 1739357989728, "timeUsageMs": 2514 },
"records": {},
"result": "page_2_content",
"substeps": []
}
]
},
{
"name": "sample-error",
"key": "document-pipeline.sample-error",
"time": { "startTs": 1739357990192, "endTs": 1739357990192, "timeUsageMs": 0},
"records": {},
"error": "Sample Error",
"substeps": []
}
]
}StepsTrack also provides Event Emitting listeners and ES6 Decorators support for easier integration. For more detailed usages, check out the Basic Usage and Advanced Usage guides.
StepsTrack includes a dashboard that provides several features for monitoring and analyzing pipeline executions.
During pipeline initialization, define how you would want to store logs as persistent data, to be able to read by the dashboard later on. Currently supported storing in file-system or Redis. See Advanced Usage for more details.
// Set up persistent storage for the dashboard
const pipeline = new Pipeline('my-pipeline', {
autoSave: true,
storageAdapter: new FileStorageAdapter('/path/to/data')
});# The image loads data from "/app/steps-data" by default.
docker run -p 3000:3000 -v /path/to/data:/app/steps-data lokwkin/steps-track-dashboardDetails of a pipeline run. From here you can examine all the steps running in the pipeline, their auto-captured data and results as well as the time usage information.
The dashboard includes auto-refreshing option, allowing you to monitor real-time pipeline runs.
Gantt Chart for visualizing the time usages of each steps in a pipeline run. You can see real-time progress of the pipeline, highlighted by status of running / success / failed.
Step Execution Stats. Aggregated from past run histories with basic statistical information for performance analyzing.
- Decorator support for easier integration.
- Generate speed analysis stats from multiple runs.
- Add Redis / File support for persistent data storage.
- Dashboard to monitor execution logs and results.
- Support real-time step exdcution monitoring.
- Convert into mono-repo and split dashboard as independent dockerized module
- Use GoogleChart / QuickChart instead of local chart.js generation
- Enhance StepsTrack Monitoring Dashboard UI/UX
- Allow importing external logs into dashboard
- Optional LLM-extension that optimize for LLM response and usage tracking
- Use memory-store instead of storing nested steps class
- More robust file locking for FileStorageAdapter
- Python version of steps tracker
MIT © lokwkin