StreamPipes Connect: Ease deletion of adapters

[dominikriemer]

Body

Users should not be able to delete adapters which are used by an existing pipeline.
The current dialog prevents deletion of such adapters, but has some drawbacks:

• There is no hint to the name of the pipeline which uses the adapter, just the number
  [#1936] Adapter Deletion: Show the Names of All Pipelines Using the Adapter and Allow One Click Deletion #2070
• From a usability point of view, it would be better to ask users if the related pipelines should be deleted as well. In this case, adapters and pipelines can be deleted with a single click without needing to navigate to the pipeline view, find the right pipeline, delete the pipeline, and go back to the adapter view.
  [#1936] Adapter Deletion: Show the Names of All Pipelines Using the Adapter and Allow One Click Deletion #2070
• It is possible that the user who owns an adapter has shared the adapter with other users and the adapter owner does not have access rights to this pipeline. Pipelines that are created by other users using the adapter should not be deleted, in this case an error message should appear.

StreamPipes Committer

I acknowledge that I am a maintainer/committer of the Apache StreamPipes project.

[muyangye]

Hi @dominikriemer @bossenti may I work on this issue? I have traced through the code a bit and I am confident I can do the first 2 improvements (hint the names and ask users if related pipelines should be deleted as well). I have not looked into the 3rd improvement yet but anyway I think it can only be done after the 2nd improvement is done. Thanks!

[bossenti]

Hi @muyangye,

yes, that would be awesome! I agree with you that an iterative approach probably works best here. Please be aware of @dominikriemer PR which is currently still WIP but has some implications for this issue: #1979. So please have a look there before starting on this issue.

Please feel free to ask for advice in case you don't know how to proceed or you are facing any errors. We are happy to help 🙂

[dominikriemer]

Hi @muyangye that would be awesome!
Please ask if you need any help.

[muyangye]

Thanks for reminding me the WIP PR, that is indeed very relevant to this issue. One quick question, how can I see my log messages in module streampipes-rest and module streampipes-storage-couchdb? I used to do "System.out.println()" but this time no message is shown on the consoles of all-pipeline-elements-jvm or connect-adapters probably because they are not the correct consoles that have my log. I also tried using the log4j Logger but I still don't know where to locate my log message. Can you please tell me where can I find the log for these 2 modules as this information is valuable for me to proceed? Thank you very much!

[bossenti]

Just to be sure we guide you in the right direction: Do you run the backend (StreamPipes Core) locally in your IDE and not in docker?

[muyangye]

I think I run it in Docker. The way I run Streampipes now is this:

1. streampipes up -d
2. In Intellij, manually click the debug/run button of configurations connect-adapters, all-pipeline-elements, and ui, so all consoles are of these 3 configurations

As a result I am pretty sure I run the backend in Docker. Do I have to run the backend locally? I just tried to do that but it says couchdb port 5984 refused and I think it is because locally I didn't install couchdb. Is there a way to see the log in Docker or do you recommend running it locally?

[bossenti]

Yes, if you want to change aspects in the backend it is required to run it locally to see your changes in action.

You should be able to do so by following these steps:

1. streampipes down
2. streampipes env --set basic: This setup only runs the prerequisites of StreamPipes, e.g., the message broker
3. streampipes up -d
4. Run StreamPipesCoreApplication in streampipes-service-core in your IDE. Ideally, the IDE detects the required environment variables from the .env file in streampipes-service-core/development. In case not, please ensure their availability manually.

With this setup, you should also be able to see your log statements in your IDE's output 🙂

[bossenti]

You can also have a look at our documentation in Confluence, but I think I mentioned all required steps

[muyangye]

Thank you very much for your reply! This time I can indeed see the log locally🙂! However, whenever I start any pipelines (e.g. run connect-adapters or all-pipeline-elements-jvm), core/backend is doing a migration which fails every time due to a connection time out:


I then tried to reset Streampipes' environment to pipeline-element by streampipes env --set pipeline-element. Unfortunately this error is reproduced in Docker too:

Have you encountered this before or have any ideas? And I am sorry to bug you once again for my environment's question but I really couldn't find a solution myself🙏.

[dominikriemer]

Hi @muyangye you can try to start with a clean system when switching to the basic environment.
So just stop all containers, remove the spnet docker network and all volumes and start with the basic environment again.
This should give you a clean system. When you start the core and extensions locally, do not forget to add environment variables to allow these locally running services to communicate with the other services running in docker (the variables in the development/.env folder of the service)

[bossenti]

There is even a convenience command for cleaning the existing system: streampipes clean -v ( to be run after streampipes down)

[muyangye]

I really appreciate both of your help! After cleaning the system I can finally start implementing this important improvement🙂.

[muyangye]

Should we close this issue as E2E tests that cover all possible scenarios are implemented? If there are problems reported we can reopen it.

[bossenti]

Is the last point of the bullet list also fulfilled?

If so, I fully agree with you

Thank you a lot for your dedicated support for this feature 🙏🏼

[muyangye]

Is the last point of the bullet list also fulfilled?

If so, I fully agree with you

Thank you a lot for your dedicated support for this feature 🙏🏼

Yes the last point is also done. So let's close it.

And you are welcome! Thank you for your guidance throughout the setup 🙏🏼!