Welcome to eCAL Discussions!

[rex-schilasky]

👋 Welcome!

We’re using Discussions as a place to connect with other members of our community. We hope that you:

• Ask questions you’re wondering about.
• Share ideas.
• Engage with other community members.
• Welcome others and are open-minded. Remember that this is a community we
  build together 💪.

To get started, comment below with an introduction of yourself and tell us about what you do with this community.

[pratikmankawde]

Hi,
How can we guarantee quality of service at subscriber's end? Meaning, there should not be any loss of message.
Is there already a way to do it? Or is there a place to implement at receive queue?

[rex-schilasky]

Hi, eCAL is not providing any kind of quality of service because this would make the protocol more complicated and would finally decrease the overall performance. We run large software projects at Continental based on eCAL in the area of autonomous driving with many distributed software algorithms and hundreds of pub/sub interfaces.

If you face message drops on the subscribers side then there could be multiple reasons. One of them is that you subscriber is just to slow to process the incoming data send by the high frequency publisher. So even eCAL would use queuing it would not help in this use case, the queue would just overflow and you would finally work on outdate data from the queues end ..

IF you need a kind of queue because only SOMETIMES your subscriber is not able to process a message, then you can just write the incoming message in your own process internal queue. This can be done in a subscriber message receive callback.
In another thread you can then access that data, empty the queue or react on overflows. If this mechanism is still to slow to just copy the incoming message in a queue, then you should check to slow down your publication or put both components in ONE process and avoid ipc for that specific connection.

Can you give some more details on your use case ? We talk about C++ ? What is the send frequency of your publisher ? What size has your message ? Do you use a callback on publisher side ? Do you process the data INSIDE that callback or in another thread ?

[pratikmankawde]

I will look through these. Thanks for sharing. :-)
Do you know of any other framework what supports pub/sub with QOS? I loved the the 'topic' feature in eCal. It's almost like zero setup and all nodes in the network can talk to each other so easily. We want exactly that.

[rex-schilasky]

Did you checkout ROS2 - https://docs.ros.org/en/rolling/ ? Or DDS systems like CycloneDDS - https://github.com/eclipse-cyclonedds/cyclonedds ? They support all kind of QOS and Pub/Sub for sure !

[pratikmankawde]

Thanks for sharing. I will check them.

[raphaelzulliger]

I'm also interested in eCAL. I found this an interesting discussion, sicne I kind of have the same questions.

Two things:

1. Why not "create our own sync mechanism" by having two topics forth and back
2. Why don't services help with QOS?

In more detail

1. What I do not understand is why you (@rex-schilasky) mention "Or you create your own sync mechanism like a feedback message" but you add that "but that makes not that much sense":

• Having two topics, e.g. "Request" and "Acknowledge"
• The sender only sends the next messages once it received an acknowledge via "Acknowledge" topic

By this, it would be implementable, wouldn't it? Are you saying that this "doesn't make sense" because then we have to implement a timeout/retry/flush/... logic ourselves (which is not as trivial as it first seems). And you consider that bad, since there are other frameworks out there already implementing this?

What I like about eCAL is the distributed/multicast feature along with its protocol agnostic feature: Protobuf vs. whatever. So maybe the "drawback" of implementing our own QOS on top of it would be a price we'd like to pay?!

2. Initially, having read about "Services", I thought I'll get the wanted QOS, kind of. But you seem to be skeptical about that one as a viable solution as well. Why? (The docs on Services seems a bit sparse so I may not have understood how they actually work)

[rex-schilasky]

Hi Raphael,

1. What I do not understand is why you (@rex-schilasky) mention "Or you create your own sync mechanism like a feedback message" but you add that "but that makes not that much sense":

• Having two topics, e.g. "Request" and "Acknowledge"
• The sender only sends the next messages once it received an acknowledge via "Acknowledge" topic

By this, it would be implementable, wouldn't it? Are you saying that this "doesn't make sense" because then we have to implement a timeout/retry/flush/... logic ourselves (which is not as trivial as it first seems). And you consider that bad, since there are other frameworks out there already implementing this?

Yes, to implement this will increase complexity and decrease performance for sure. Think about connections of multiple publisher to multiple subscriber running distributed on different hosts (and so using different transport mechanisms). Maybe some of them should process every single message but some of them are slow processing for some reasons and not interested on every message (visualization nodes ..).

So, eCAL itself is not providing a feedback mechanism to guarantee the highest possible "best effort" performance. But for sure such a mechanism can be implemented on user level, you are right - but then it's your job to not slow down you whole system ;-).

The two eCAL C++ samples latency_snd and latency_rec_cb are using such a feedback connection by the way to fully snyc publisher and subscriber without loosing a single message.

What I like about eCAL is the distributed/multicast feature along with its protocol agnostic feature: Protobuf vs. whatever. So maybe the "drawback" of implementing our own QOS on top of it would be a price we'd like to pay?!

Thank you for your feedback, we really appreciate this ! To use different protocols was (besides performance) another design goal.

1. Initially, having read about "Services", I thought I'll get the wanted QOS, kind of. But you seem to be skeptical about that one as a viable solution as well. Why? (The docs on Services seems a bit sparse so I may not have understood how they actually work)

Services are a different communication pattern and no replacement for a publish subscribe pattern. For large systems with hundreds or thousands of data connections and many of them between one sources to one or multiple sinks, services are no alternative IMHO. They create latency everywhere and can cause bad timeouts or deadlocks in some scenarios.

[AntoineBassoul]

Hello eCAL !

I know ecal is great to use as a real time pub/sub library.
I'd be interested in directly writting & reading eCAL data from ecal rec files to do deterministic simulation and build custom applications that read ecal data as fast as possible (I hope that's somehow clear, i'd like to do something similar to what is done in the "yaaf simulator binding").
I didn't see any documentation regarding this, is there something available in the API ?

Thanks and Best,
Antoine (now at https://navya.tech/en/)

[rex-schilasky]

Hi and thank you for the nice words !

The eCAL HDF5 C++ and python API is open source (YAAF is an internal frame generator).

Here is a sample how to read a measurement (message encoding google protobuf) in C++:
https://github.com/continental/ecal/blob/master/samples/cpp/measurement/measurement_read/src/measurement_read.cpp

and this one for Python:
https://github.com/continental/ecal/blob/master/samples/python/measurement_read/measurement_read.py

This is the google unit test:
https://github.com/continental/ecal/blob/master/testing/contrib/ecalhdf5/hdf5_test/src/hdf5_test.cpp

So finally you can integrate the C++ HDF5 measurement interface into your algorithm frame and stimulate your interface in a deterministic way independently from real-time.

Best, ReX

[66hades]

Hello,
I'm a Systems Test Engineer learning to use eCAL on development vehicles to evaluate ADAS features.
I tried to start a new discussion thread, but apparently do not have permissions to do that, so I'm piggybacking on this thread.
Question: Is there an eCAL equivalent to ROS's rqt_graph? Oftentimes it is more understandable to see a node graph of various processes and the topics they're using. I can only find the Monitor App which only provides lists, and unless I'm missing something not a graph.
Thanks, Brian

[rex-schilasky]

Hi Brian,
currently there is no equivalent tool like rqt_graph for eCAL. Since the monitoring information is quite simple to snapshot via python script one option is to use python packages for node plotting and create a graph out of this. Would that be a help for you ? Maybe we can add a script showing how to do so ...
Best, ReX

[66hades]

Hello Rex,
That would be helpful! I'm familiar with C++ & Python, any hints on where to get the current list of nodes & topics, a good graph visualizer repo, and a script to incorporate it into eCAL would be extremely appreciated.
Thanks, Brian

[rex-schilasky]

Hi Brian, I made a first prototype for generating a graph for the ecal monitoring. Just take a look here ecal_mongraph. Would be nice if we could improve that together.

[davidtwomey]

Hi,

Relatively new to ecal but very much enjoying the development experience so far!
I have a quick question:
Before creating a service, I want to check if a service with the same service name already exists (on the same host).
Is there a recommended way to do this?

Thanks

[rex-schilasky]

Hi,

you can use the monitoring API shown in this sample monitoring_reg. In the next version 5.10 you can just start a client and attach a "connect event" to check if there is a service with the same name connecting to you. Like shown here .

Enjoy eCAL, ReX

[long131ss]

hi, I integrated the ecal framework into Kinetic ros,and it can works on my system.
But I found the cpu usage ratio is high. for example, I just subsciber a topic /scam with the frequency 10hz, the message size is 3kb,
another topic /map ,with frequency 0.2hz,message size is 35kb, I can find the cpu ratio is about 10%, as a comparison, I also tested another pub/sub framework zero-zmq ,I found the cpu usage is very low ,about 3%。
Question: do you test the cpu usage? or although ecal with share memory is fast ,but it consumes the cpu more much?
If I want run the ecal framework with relatively low cpu usage ,what can i do?

[rex-schilasky]

hi, I integrated the ecal framework into Kinetic ros,and it can works on my system.

But I found the cpu usage ratio is high. for example, I just subsciber a topic /scam with the frequency 10hz, the message size is 3kb,

another topic /map ,with frequency 0.2hz,message size is 35kb, I can find the cpu ratio is about 10%, as a comparison, I also tested another pub/sub framework zero-zmq ,I found the cpu usage is very low ,about 3%。

Question: do you test the cpu usage? or although ecal with share memory is fast ,but it consumes the cpu more much?

If I want run the ecal framework with relatively low cpu usage ,what can i do?

Can you please share some details about your platform ?

[long131ss]

Hi, thanks for your reply. I tested it on the platform rockchip.x3399 with arm x64 and 1G memory. do you have any suggestions for reduce of the cpu usage?

[rex-schilasky]

Can you please make a test without publication. Just create all entities (pub/sub) and start your processes ?
There is a background registration layer and I want to figure out how much it consumes on your platform.
We run the middleware mainly on large server systems with some more cpu power ...

[long131ss]

maybe the ecal framework is more suite to the large message transport ,such as automatic driving . And now i want to apply it to my Robot, the sensor message size is less than 256kb.

[long131ss]

Can you please make a test without publication. Just create all entities (pub/sub) and start your processes ?
There is a background registration layer and I want to figure out how much it consumes on your platform.
We run the middleware mainly on large server systems with some more cpu power ...

thanks for your kind reply. i will tell you test result once i get back my work. Because our system is strictly to the cpu usage, so it is important to me.

[long131ss]

Hi, done as you said, just create all entities and no publication, the CPU usage is about 5%.

[davidtwomey]

Hi @rex-schilasky,

Another quick question for you. Am I right in saying there is currently no way for a python Client to interact with a c++ protobuf Service?
The only way would be to deserialize the protobuf manually and use string methods?

Thanks

[rex-schilasky]

Hi David,

that is a very good question. This use case is not covered by the current protobuf service implementation. This looks like this

Server Side:

      /**
       * @brief Create service.
       *
       * @param service_  Google protobuf service instance.
       *
       * @return  True if successful.
      **/
      bool Create(std::shared_ptr<T> service_)
      {
        if (service_ == nullptr) return false;
        m_service = service_;

        const google::protobuf::ServiceDescriptor* service_descriptor = service_->GetDescriptor();
        Create(service_descriptor->full_name());       << the protobuf service name is used to name the server

Client Side:

    /**
     * @brief Google Protobuf Client wrapper class.
    **/
    template <typename T>
    class CServiceClient : public eCAL::CServiceClient
    {
    public:
      /**
       * @brief Constructor.
       *
      **/
      CServiceClient()
      {
        Create(T::descriptor()->full_name());      << the protobuf service name is used to name the client
      }

I will open an issue and we will find a solution here.

Best, ReX

[davidtwomey]

Right I see! Yes perfect, let’s start from an issue and see how we can resolve

[rex-schilasky]

Set an issue now (#493). Let's discuss this there ...

[rex-schilasky]

Going to master currently ... See "ping" service samples how to use it.

24 hours service ;-)

Good luck, ReX

[davidtwomey]

Amazing - thanks Rex!

[davidtwomey]

Another question for you @rex-schilasky :)

I have created a very simple pub/sub scenario where I am sending proto messages (via shm on the same host).
However, if the publisher sends 2 (or more) messages sequentially with no sleep, the subscriber drops at least one of these messages.
This is even the case when the subscriber callback is empty!

(Ignoring for a sec the point that if you are sending n messages sequentially without sleep, why not batch them)
Are there any settings I can configure to ensure I do not (or at least minimise) dropped messages? Possibly:

• Change sub/pub buffer size?
• Change QoS?
• Switch to zero-copy?
• Have to revert to tcp?

Or is the ONLY solution to add a small sleep between publishing? 1ms works for example

Many thanks!

David

[rex-schilasky]

Hi David,

there is a global configuration setting (ecal.ini) where you can define a handshake with timeout between pub/sub connections for local communication. See #441.

Maybe we need to better document this option ;-).

Best, ReX

[davidtwomey]

Great, I tried with setting it to 100ms, that seems to resolve the issue!

What exactly does enforcing a handshake timeout do here?
Is lower always better? Can I, for example set this down to 1? Or are there other considerations here?

[rex-schilasky]

It couples the publisher with a defined maximum timeout to the connected subscribers memory reader.

Not to the callback processing time, only to time the subscriber needs to make a copy of the memory files content.

So higher values will not really decrease performance.

There are special cases, for example if you activate zero copy additional THEN the subscriber will work on the memory file content directly and so blocking the publisher until the callback function returns.

[davidtwomey]

Understood thanks!
Is there a specific reason the default value is set to 0?

[Kerstin-Keller]

The thing is, this setting can lead to unintended side effect, which are mainly relevant in 1:N scenarios (1 publisher, N subscribers).
If any of the N subscribers is blocked from receiving their message (which until the next release happens if arbitrary time is spend in a callback, even of another topic subscription), the sender is blocked from sending at a given frequency.

In big scenarios, a few framedrops for any subscribers is much more acceptable than the whole system experiencing delays.

[harlenli2022]

Hi @rex-schilasky,

Nice to contact you, we are planning to use eCAL in our Cockpit and ADAS system, and currently we had ported eCAL to arm linux successfully.
Since more and more systems are to use QNX hypervisor solution, so we have to support eCAL in QNX system also.
Currently I'm trying to build the QNX version, but seems some third party dependencies (Asio, etc) don't support QNX very well.

Has eCAL ported successfully before to QNX system? it would be great helpful for our estimation, hope we are not the first one to do this...

Thanks

[rex-schilasky]

Hi,

I'm very curious for what company you are planning to use eCAL, would be interesting for my developer team :-).

Regarding your question. Yes, together with MATHWORKS we ported and run eCAL on top of QNX. Please take a closer look to the ecal-toolbox repo for simulink first.

Best, ReX

[harlenli2022]

Hi,

I met issues on network configuration when using eCAL to communicate between two PCs in different sub-net (e.g. 192.168.1.x and 192.168.2.x).
Following the wiki page, I had added route 239.0.0.0 already, after then two PCs in same sub-net (192.168.1.xx) can exchange info via eCAL successfully, but in case of different sub-net, server/client (same to pub and sub) can't identify each other...

Could you support to confirm if special configuration needed for my issue?

Thanks in advance.

[harlenli2022]

@rex-schilasky,
what's more, I also built eCAL in MacOS, everything seems ok except ecal rec application can't be launched with below protobuf errors, have your team occurred same issue? thanks.

[libprotobuf ERROR /Users/HarlenLI/workspace/ecal/thirdparty/protobuf/src/google/protobuf/descriptor_database.cc:120] File already exists in database: ecal/pb/host.proto
[libprotobuf FATAL /Users/HarlenLI/workspace/ecal/thirdparty/protobuf/src/google/protobuf/descriptor.cc:1356] CHECK failed: GeneratedDatabase()->Add(encoded_file_descriptor, size):
libc++abi: terminating with uncaught exception of type google::protobuf::FatalException: CHECK failed: GeneratedDatabase()->Add(encoded_file_descriptor, size):

[Kerstin-Keller]

@FlorianReimold maybe you can answer on UDP settings?

As for protobuf, which commit did you build, did you build one of the tags or the current master?
Usually you get these type of protobuf errors when any proto datatype is compiled multiple times into the program. I am not aware that we are doing any such thing in eCAL Rec. If it occurs in Macos, I would think it occurs on Ubuntu as well. What protobuf version are you using in the build?

[FlorianReimold]

Hi Harlenli,

Can you create either create a new discussion or an issue with your question? In the past we had misconfigured github, so everybody created their questions as a comment to this discussion as they were not able to create discussions (So it is basically our fault that this discussion looks as it looks 😑). But now creating discussions should be possible for all users and a proper new GH Discussion will better help other users in the future.

[harlenli2022]

@FlorianReimold
Sorry for my mistake, I had created separate discussion instead, let's switch to the new discuss then.