Coceptual Question:- Doubts regarding Connector Source, Connector Sink, Storage and Eventmesh Runtime.

[VishalMCF]

Search before asking

• I had searched in the issues and found no similar issues.

Question

Will add the details soon.

[VishalMCF]

Hello Team,
I was going through the docs regarding setting up of eventmesh runtime using an eventmesh storage and eventmesh runtime. In the docs, after setting up RocketMQ as our storage layer we are configuring our eventmesh runtime by adding dependency of eventmesh-connector-rocketmq. Why we are not loading the plugin of eventmesh-storage-rocketmq if we want to store the events in the storage layer?
Also, how are connectors used exactly in production? Are they part of the same Eventmesh application instance (eventmesh-runtime) or they are running as a separate application? If they are running as a separate application are they embedded in a (go, java, rust) application server?
In the code base, I did find that using the config we can run any connector as a server both as sink and source based on the option given in the server-config.yml file.
In the architecture which I have attached above it shows that the connector and runtime are running as a separate application.
Let's say I want to implement the following architecture using eventmesh what should be my approach to set up the important components?

As per my understanding as of now, I need to do the following steps:-

1. Set up the instances of Apache Pulsar, MongoDB, and PostgreSQL.
2. The GO CLI application should contain some logic that pushes some data to the MongoDB instance.
3. One instance of Eventmesh source connector should be launched. It should have a dependency on eventmesh-connector-mongodb.
4. One instance of Eventmesh sink connector should be running which should have the plugin of eventmesh-connector-spring.
5. The Spring Boot application should be launched and it should be configured to listen to the events coming from the sink. (How is it done? Is there any documentation for that?) It should contain some business logic to store data in the PostgreSql and respond to the UI
6. The Eventmesh Runtime should be running and its storage should be configured to Apache Pulsar. Although here I am having doubts because storage is pulsar why do need to configure pulsar-connector as per the documentation?
7. Also is it not possible for an application can directly send events to the eventmesh without having any connector layer i between?
   I am not sure whether my doubts are justified to be asked here as an issue. But it is quite confusing how eventmesh is used in production as there are very less articles present online

[Pil0tXia]

1. eventmesh-storage-rocketmq is loaded by default in eventmesh-runtime/build.gradle at L43.
2. Connectors can be deployed on any machine, including the machine on which EventMesh Runtime has already been started.
3. It is planned to integrate Connectors into K8s.
4. EventMesh supports the architecture you have provided. You will need to deploy the MongoDB source connector, the Spring sink connector, and integrate the eventmesh-storage-pulsar dependency into the EventMesh Runtime plugin binary file.

[VishalMCF]

Okay Thanks @Pil0tXia I will try to implement to the above architecture and understand the core concepts of Apache Eventmesh.

[Pil0tXia]

Here is my response to the part you added after the architecture image:

1. The documentation for the Spring connector is still being improved, and you can expect to see usage instructions as early as next month. If you prefer not to use the connector, you can utilize the EventMesh SDK in your application. Please refer to the following link for more information: EventMesh SDK for Java.
2. There is no need to deploy Pulsar connector.

[VishalMCF]

Great! Thanks for your quick help. @Pil0tXia

[pandaapo]

@VishalMCF

In the docs, after setting up RocketMQ as our storage layer we are configuring our eventmesh runtime by adding dependency of eventmesh-connector-rocketmq.
...
Although here I am having doubts because storage is pulsar why do need to configure pulsar-connector as per the documentation?

Could you please indicate which documents describe these operations? They need to be modified.

The Spring Boot application should be launched and it should be configured to listen to the events coming from the sink. (How is it done? ... It should contain some business logic to store data in the PostgreSql and respond to the UI

Spring Sink Connector supports user to receive messages through method annotated by @EventMeshListener. At present, the functionality is relatively basic, the annotated method can only have one formal parameter representing the received message.
After receiving the message, users need to handle the business logic themselves.

[VishalMCF]

@pandaapo Please check here:-
https://eventmesh.apache.org/docs/instruction/runtime#113-build-and-load-plugins