Cannot delete a misconfigured output connector

[ryzhyk]

Steps to reproduce

• Create a pipeline using a program with at least one output
• Connect any output Kafka connector to the pipeline
• Modify the program to remove the output view, making the connector "orphan"
• Go back to the pipeline editor and try to delete the connector. The connector doesn't get deleted and an error message shows up: Could not attach 1 connector(s): No tables/views named PREFERRED_VENDOR found.

[lalithsuresh]

@snkas I believe you encountered something similar with a dangling connector too right?

[snkas]

I don't think so, what I encountered was more related to the UI when you add/remove connectors and that it only updates when the pipeline is stopped and started (which is expected behavior).

I think this issue does raise an interesting question regarding program versioning. What is expected behavior when you edit or delete a program which a pipeline uses right now (irrespective if it is running or not)? The most straightforward would be a "no you can't, a created pipeline depends on this, as such any edits are not possible". Possibly with an option to duplicate the program to create a new one.

Similarly, would this method of handling also be used for running pipelines: if you wish to edit them, they must be stopped first? In any case right now, the pipeline must be stopped and restarted for changes to take effect.

[gz]

this issue does raise an interesting question regarding program versioning. What is expected behavior when you edit or delete a program which a pipeline uses right now (irrespective if it is running or not)? The most straightforward would be a "no you can't, a created pipeline depends on this, as such any edits are not possible". Possibly with an option to duplicate the program to create a new one

we have mostly well defined semantics for this: e.g., the program & all connector configuration get versioned at pipeline deploy time. afterwards you can edit program connectors and evolve them independently (while pipeline is running) -- but the changes should not affect a running (or crashed and recovered pipeline). the next time you stop and then want to run a pipeline again, the UI should tell you that you made changes and show you a diff (which gets applied if you decide to run again)

[Karakatiza666]

@ryzhyk @lalithsuresh
After an SQL program gets modified and recompiled to remove a view which orphans corresponding connector, the /pipelines/{pipeline_id} endpoint still returns the now-voided connection to non-existent view in descriptor.attached_connectors array field. Is this an expected behavior of backend and should UI send a separate request to remove the stale connection from the pipeline?

[ryzhyk]

Yes, I think this is expected behavior. The UI should not remove this connector automatically. The idea is that, as the user edits the program, some connections can become temporarily or permanently invalid. We should show them as orphans (i.e., connectors not attached to the pipeline) to the user and give them an opportunity to (1) delete no longer needed connectors (this is the part that doesn't work), (2) fix up the program to add missing views, or (3) re-attach orphan connectors to a different table or view (e.g., if the view was renamed).
@gz , I remember you did some work on this recently. How do you reckon the UI should handle this?