logger as part of immutable/versioned pipeline definition

[TimoKramer]

We spoke about this issue already extensively and I kind of get the idea behind it but I still think it would be better to handle the pipeline definition immutably and specify the logger inside the pipeline definition.

It avoids the complexity of having to specify the logger on each run which makes it complicated for the user (interface). The interface would have to store a default somewhere to make it more user-friendly which breaks the idea behind this concept anyway.

[lispyclouds]

What would be a way to represent this in the pipeline spec? lets start there and think through it.

[TimoKramer]

so... as I see it in etcd the history of KVs is only preserved until compaction so this is not a persistent way of storing the combination of run and revision. We could make the key of a pipeline spec a combination of group/name/revision something like bob.pipeline/dev:pipeline1/00001. then we could extend the run spec with :bob.pipeline/revision and get rid of the :bob.pipeline.run/logger. backwards compatibility is something i did not figure out by now.

When switching to NATS it might be the point where this breaking change could be introduced. AFAIS there is some kind of revision history per key in NATS JetStream but it's capped at 64 revisions. We could probably manually put a version into the value document as well.

[TimoKramer]

that would give us the opportunity to set a logger per pipeline and not per run. and the GUI/TUI would not have to store a default itself.