Stream Inspector

[neovintage]

The goal is to give folks the ability to see what's happening inside the stream. This is not like tail. This is more like peek. Pull out some information from the stream so that a developer can see the data and data types.

• docs: Stream inspector design #692
• UI changes feat(ui): stream inspector #764
• Stream Inspector: Destinations #700
• Stream Inspector: Sources #701
• Stream Inspector: Processors #702

[lyuboxa]

Inspector will be nice, but I will raise you a REPL for this, it will be magnificent.

[neovintage]

@lyuboxa what would a repl look like? Would you have to create a "conduit cli" to make this possible?

[lyuboxa]

I think any tooling that can be used to inspect data real time and does not involve explicit parsing will be useful. When @hariso and I were discussing this something like this came to mind:

$ conduit-cli
> pipelines
<... print list of pipelines>
> c  = pipeline["abc"].connector["source"]
> c.peek(1) 
-> <message in structured format>
> c.processors["anonymize"].Peek(1)
-> <message in structured format> 

I think an interactive REPL will be useful, a console or whatever you want to call it.

[lovromazgon]

@lyuboxa A REPL doesn't sound very composable to me 😕 Why not rather provide a simple CLI command to get the records? This would let you compose it with other tools like grep & co. Example:

$ conduit-cli inspect id:of:my:component --limit 2
<message>
<message>

Sidenote: I think it's a bit off-topic to talk about a CLI/REPL in this issue, it sounds like a whole separate feature by itself.

[lyuboxa]

A REPL doesn't sound very composable to me .

Does it have to be? REPL are interactive tools for evaluation, not unix commands. Both options have their merits.
Former is more suitable for interactive work (which CLI is not tbh, but it does need a way to enter an interactive session).
However, if you only want to dump data and move on, then this is likely not the most optimal solution.

Sidenote: I think it's a bit off-topic to talk about a CLI/REPL in this issue, it sounds like a whole separate feature by itself.

Yup for sure.

[hariso]

Sidenote: I think it's a bit off-topic to talk about a CLI/REPL in this issue, it sounds like a whole separate feature by itself.

Yup for sure.

Here's a separate issue for that: #697.

[hariso]

All the tasks are done, closing the issue.