pipeline as yaml? #112
-
Howdy! I was wondering if there is any way to define a pipeline as yaml or json? I work for this monitoring project: https://github.com/netdata/netdata and i'd love to build a sort of pandas or pdpipe collector that could just take in a pipeline as config. Idea is that this might lower entry barrier for some users who are familiar with pandas and could just define a pdpipeline as part of the config. At moment i have created something like this: https://github.com/andrewm4894/netdata/blob/pandas-collector/collectors/python.d.plugin/pandas/pandas.chart.py#L46 So user just defines the pandas steps to run: https://github.com/andrewm4894/netdata/blob/pandas-collector/collectors/python.d.plugin/pandas/pandas.conf#L81 But i dont think the eval/exec approach is great and would like to avoid it if i could. Was thinking if i could generate a pdpipe pipeline from some yaml or json then that could be cleaner in thatb we could have something like a pdpipe collector where users just point it at some url that has some csv/json they want to convert into metrics for monitoring and they juist define some pdpipe steps in the config in some way. Library looks cool btw! |
Beta Was this translation helpful? Give feedback.
Replies: 3 comments
-
I think that a very interesting idea, but that it is quite a big feature. I'd love to see some initial code contribution for you, if you're up to it. I also recommend taking a look at the following three projects, as they all build upon |
Beta Was this translation helpful? Give feedback.
-
I've opened a feature request here: |
Beta Was this translation helpful? Give feedback.
-
cool - upvoted the feature request. cheers. |
Beta Was this translation helpful? Give feedback.
I've opened a feature request here:
#113