You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Is your feature request related to a problem? Please describe.
For perpetually running pipelines in particular, its hard to get a sense of how much work the pipeline is doing against the database. It would be ideal if the user
Describe the solution you'd like
Expand the nodestream run command to configure and run a prometheus metrics server:
nodestream run <<pipeline>> --prometheus-server-addr 127.0.0.1:8080
We already collect metrics as part of the PipelineContext api accessible via get_context() in nodestream.pipeline.meta. This means that we can reuse a lot of existing logic for Prometheus. As well as pipeline or application level metrics, it would be ideal to include some process/system level metrics such as memory and cpu usage.
Describe alternatives you've considered
Currently there is max memory usage logged as part of the message but that isn't really good enough. the only other alternative is to monkey patch the system, or use custom components for everything.
Additional context
n/A
The text was updated successfully, but these errors were encountered:
Is your feature request related to a problem? Please describe.
For perpetually running pipelines in particular, its hard to get a sense of how much work the pipeline is doing against the database. It would be ideal if the user
Describe the solution you'd like
Expand the
nodestream run
command to configure and run a prometheus metrics server:nodestream run <<pipeline>> --prometheus-server-addr 127.0.0.1:8080
We already collect metrics as part of the
PipelineContext
api accessible viaget_context()
innodestream.pipeline.meta
. This means that we can reuse a lot of existing logic for Prometheus. As well as pipeline or application level metrics, it would be ideal to include some process/system level metrics such as memory and cpu usage.Describe alternatives you've considered
Currently there is max memory usage logged as part of the message but that isn't really good enough. the only other alternative is to monkey patch the system, or use custom components for everything.
Additional context
n/A
The text was updated successfully, but these errors were encountered: