Dnae adas/serialized ipm #973

DensoADAS · 2020-01-30T09:45:35Z

Description:
Extended rclcpp to allow serialized intra process communication. Therefore, we added a second connection to the intraprocess manager to the publisher and subscriber for serialized messages. To ease up the memory management, the rcl_serialized_message_t is wrapped in SerializedContainer with a delete to prevent memory loss. To prevent double free the ownership of the serialized message is passed to the publisher. The conversion between serialized and deserialized messages is done in the SubscriptionIntraProcess on demand. The changes also allow general purpose publisher and subscriber as needed e. g. for rosbag2.

Possible applications:
• direct recording of serialized data without serialization (with IPM)
• non-templated publisher of serialized messages (with IPM)

Features:
• added serialization and deserialization functions in intraprocess manager
• implemented publish of serialized messages (call by value and unique pointer) in Publisher and PublisherLifecycle
• claiming ownership of serialized messages by publish(rcl_serialized_message_t)
• create_publisher and create_subscription support now general message type of rcl_serialized_message_t by passing message_type (so non-templated version is possible)
• added unit tests: test_intra_process_communication

DensoADAS · 2020-01-30T09:46:49Z

updated from #819

DensoADAS · 2020-02-21T07:29:43Z

Has anyone reviewed the PR already?
@ivanpauno @wjwwood

wjwwood · 2020-02-21T22:56:54Z

No, we have not sorry. We'll get to it as soon as we can.

DensoADAS · 2020-04-16T03:08:11Z

@ivanpauno @wjwwood Just a small after-Easter question: Any updates? This PR is blocking the rosbag2 PR.

ivanpauno · 2020-04-16T16:08:23Z

@ivanpauno @wjwwood Just a small after-Easter question: Any updates? This PR is blocking the rosbag2 PR.

Sorry, but I don't think that I will be able to do a review before Foxy release.

Karsten1987 · 2020-04-16T17:59:02Z

@DensoADAS can you please rebase this branch? The connected PR for rosbag2 please as well.

Karsten1987

Uff, so that's quite a thing to review :)

I think I understood a fair amount of it to move forward on it.
My main concern would be that the publisher implementation gets too heavily overloaded which makes it quite hard to introspect. What about we introduce a GenericPublisher or SerializedPublisher similar to what we currently have in rosbag2? That would make it a bit easier to distinguish what type of messages are being sent. It also makes it a bit more straight forward to design an API which is tailored to it.

Karsten1987 · 2020-04-16T23:35:54Z

rclcpp/include/rclcpp/create_publisher.hpp

+  typename PublisherT = rclcpp::Publisher<MessageT, AllocatorT>,
+  typename NodeT>
+std::shared_ptr<PublisherT>
+create_publisher(


why do I need this overload if both methods are templated by MessageT? The call to const auto type_support = *rosidl_typesupport_cpp::get_message_type_support_handle<MessageT>(); could be done in the original function or even within the create_publisher_factory function, couldn't it?

this allows an instantion with type "rcl_serialized_message_t" for generic publishers without previously known message type (e.g. for rosbag2)

Example:
auto publisher = rclcpp::create_publisher<rcl_serialized_message_t>( node, "/topic", *rosidl_typesupport_cpp::get_message_type_support_handle<test_msgs::msg::Strings>(), rclcpp::QoS(10));

rclcpp/include/rclcpp/create_subscription.hpp

Karsten1987 · 2020-04-16T23:38:21Z

rclcpp/include/rclcpp/experimental/serialization.hpp

+
+namespace rclcpp
+{
+namespace experimental


why should that be in experimental?

is there a definition for "experimental"? Should we move it?

I forgot the rationale why we introduced "experimental". But I see the Serialization class being easily usable outside of the context of the intraprocess communication. Therefore, I'd propose putting it in the regular rclcpp namespace.

Karsten1987 · 2020-04-16T23:46:34Z

rclcpp/include/rclcpp/experimental/serialization.hpp

+public:
+  virtual ~SerializationBase() {}
+
+  virtual std::shared_ptr<rcl_serialized_message_t> serialize_message(const void * message) = 0;


I am not sure I like that a raw pointer (void *) is getting converted into a shared pointer. What does that mean in terms of ownership? Is the raw pointer no longer owned and directly converted into the shared pointer? What if the message was original allocated on the stack and passed in via &message?

yeah, I understand your concern. The Serialization class hides the rmw functions. It's difficult to avoid the void* without knowing the actual content (cannot be templated here). For the current PR we skip changing this

I think the problem here is not passing the message in via a void *. The problem I see here is mainly that it's getting converted to a shared pointer which has implicit assumptions about ownership.

what about changing the signature to something like this:

virtual void serialize_message(const void * ros_message, rcl_serialize_message_t * serialized_message)

input and output are clearly defined and the encapsulation into a shared pointer can be done outside of this funciton.
That also goes along well with the deserialize_message function

The message is not converted in this sense. The resulting shared pointer is a new object. (And I want to prevent objects without cleanup functions as much as possible, so I prioritized preventing memory loss)

I thought I had mentioned this somewhere else already, but what about using the SerializedMessage for this?

virtual void serialize_message(const void * ros_message, SerializedMessage & serialized_message); virtual void deserialize_message(const SerializedMessage & serialized_message, void * ros_message);

Karsten1987 · 2020-04-16T23:47:15Z

rclcpp/include/rclcpp/experimental/serialization.hpp

+  virtual void deserialize_message(
+    const rcl_serialized_message_t & serialized_message,
+    void * msg) = 0;


I think that the signature of serialize should be the same as deserialize

rclcpp/include/rclcpp/publisher_factory.hpp

rclcpp/include/rclcpp/subscription_factory.hpp

Karsten1987 · 2020-04-17T00:43:09Z

rclcpp/src/rclcpp/intra_process_manager.cpp

-IntraProcessManager::add_subscription(SubscriptionIntraProcessBase::SharedPtr subscription)
+IntraProcessManager::add_subscription(
+  SubscriptionIntraProcessBase::SharedPtr subscription,
+  const bool is_serialized)


we don't need that parameter here. It's part of the subscription base: https://github.com/ros2/rclcpp/blob/master/rclcpp/include/rclcpp/subscription_base.hpp#L203-L205

Karsten1987 · 2020-04-17T00:44:11Z

rclcpp/src/rclcpp/intra_process_manager.cpp

-IntraProcessManager::add_publisher(rclcpp::PublisherBase::SharedPtr publisher)
+IntraProcessManager::add_publisher(
+  rclcpp::PublisherBase::SharedPtr publisher,
+  const bool is_serialized)


I don't see an easy way to set a flag on the publisher base similar to the subscription base which indicates that it's a serialized publisher. Bummer.

That being said, what happens if I take the same publisher and publish first a serialized message and then a regular MsgT?

EDIT: I just browsed through the tests to see what's happening on the user side. What about we don't call a create_publisher function, but rather a create_serialized_publisher which implements the publisherbase interface (sets the is_serialized parameter) and only allows a call to publish with serialized messages.

Karsten1987 · 2020-04-17T00:46:49Z

rclcpp/src/rclcpp/intra_process_manager.cpp

+  // a publisher and a subscription with different content type can't communicate
+  if (sub_info.is_serialized != pub_info.is_serialized) {
+    return false;
+  }


Just when I thought I understood what's happening the intraprocess manager, I read this comment and I am confused again. ;-)
I thought the four different variants are so that a non-serialized subscription could obtain values published from a serialized message?

@DensoADAS can you comment on this?

I think the point here is that here always the serialized and non-serialized publishers are added to the interprocess manager.

Does this make sense though? Either the publisher is a serialized one or not, am I right?
That is, I think we can just only add one id plus the flag whether it's serialized or not.

DensoADAS · 2020-04-17T17:31:09Z

@Karsten1987 Thank you for your review. I've implemented most of your comments (some stuff like raw pointers need more time) We'd appreciate it if the PR could make it in foxy. Have a nice, healthy and relaxing weekend!

Karsten1987

another round of review

Karsten1987 · 2020-04-17T17:53:01Z

rclcpp/include/rclcpp/experimental/serialization.hpp

+public:
+  virtual ~SerializationBase() {}
+
+  virtual std::shared_ptr<rcl_serialized_message_t> serialize_message(const void * message) = 0;


I think the problem here is not passing the message in via a void *. The problem I see here is mainly that it's getting converted to a shared pointer which has implicit assumptions about ownership.

what about changing the signature to something like this:

virtual void serialize_message(const void * ros_message, rcl_serialize_message_t * serialized_message)

input and output are clearly defined and the encapsulation into a shared pointer can be done outside of this funciton.
That also goes along well with the deserialize_message function

rclcpp/include/rclcpp/experimental/serialized_message.hpp

Karsten1987 · 2020-04-17T18:01:08Z

rclcpp/include/rclcpp/experimental/serialization.hpp

+
+namespace rclcpp
+{
+namespace experimental


I forgot the rationale why we introduced "experimental". But I see the Serialization class being easily usable outside of the context of the intraprocess communication. Therefore, I'd propose putting it in the regular rclcpp namespace.

rclcpp/include/rclcpp/experimental/serialization.hpp

Karsten1987 · 2020-04-17T18:13:29Z

rclcpp/include/rclcpp/publisher.hpp

@@ -191,34 +192,42 @@ class Publisher : public PublisherBase
      get_subscription_count() > get_intra_process_subscription_count();

    if (inter_process_publish_needed) {
-      auto shared_msg = this->do_intra_process_publish_and_return_shared(std::move(msg));
+      auto shared_msg = this->do_intra_process_publish_and_return_shared(
+        std::move(


don't think we need that line break here.

Karsten1987 · 2020-04-17T18:15:43Z

rclcpp/src/rclcpp/intra_process_manager.cpp

+  // a publisher and a subscription with different content type can't communicate
+  if (sub_info.is_serialized != pub_info.is_serialized) {
+    return false;
+  }


@DensoADAS can you comment on this?

rclcpp/src/rclcpp/subscription_base.cpp

rclcpp/test/test_intra_process_communication.cpp

rclcpp_lifecycle/include/rclcpp_lifecycle/lifecycle_publisher.hpp

DensoADAS · 2020-04-17T19:41:34Z

@Karsten1987 another update

Karsten1987 · 2020-04-17T19:57:07Z

~~@DensoADAS branch has (still) conflicts~~

I took the liberty to rebase your branch.
However, when compiling locally on my OSX, I get quite a bunch of compiler errors:

In file included from /Users/karsten/workspace/ros2/ros2_master/src/ros2/rclcpp/rclcpp/src/rclcpp/time_source.cpp:27:
In file included from /Users/karsten/workspace/ros2/ros2_master/src/ros2/rclcpp/rclcpp/include/rclcpp/node.hpp:37:
In file included from /Users/karsten/workspace/ros2/ros2_master/src/ros2/rclcpp/rclcpp/include/rclcpp/callback_group.hpp:23:
In file included from /Users/karsten/workspace/ros2/ros2_master/src/ros2/rclcpp/rclcpp/include/rclcpp/client.hpp:32:
/Users/karsten/workspace/ros2/ros2_master/src/ros2/rclcpp/rclcpp/include/rclcpp/function_traits.hpp:52:52: error: no member named 'operator()' in 'rclcpp::SubscriptionOptionsWithAllocator<std::__1::allocator<void> >'
    typename function_traits<decltype( &FunctionT::operator())>::arguments>::type;
                                        ~~~~~~~~~~~^
/Users/karsten/workspace/ros2/ros2_master/src/ros2/rclcpp/rclcpp/include/rclcpp/function_traits.hpp:136:38: note: in instantiation of template class 'rclcpp::function_traits::function_traits<const rclcpp::SubscriptionOptionsWithAllocator<std::__1::allocator<void> > >' requested here
struct function_traits<FunctionT &>: function_traits<FunctionT>
                                     ^
/Users/karsten/workspace/ros2/ros2_master/src/ros2/rclcpp/rclcpp/include/rclcpp/subscription_traits.hpp:91:39: note: in instantiation of template class 'rclcpp::function_traits::function_traits<const rclcpp::SubscriptionOptionsWithAllocator<std::__1::allocator<void> > &>' requested here
    typename rclcpp::function_traits::function_traits<CallbackT>::template argument_type<0>>
                                      ^
/Users/karsten/workspace/ros2/ros2_master/src/ros2/rclcpp/rclcpp/include/rclcpp/create_subscription.hpp:43:41: note: in instantiation of template class 'rclcpp::subscription_traits::has_message_type<const rclcpp::SubscriptionOptionsWithAllocator<std::__1::allocator<void> > &, void, void, void>' requested here
  typename rclcpp::subscription_traits::has_message_type<CallbackT>::type,
                                        ^
/Users/karsten/workspace/ros2/ros2_master/src/ros2/rclcpp/rclcpp/include/rclcpp/create_subscription.hpp:51:1: note: in instantiation of default argument for 'create_subscription<rcl_interfaces::msg::ParameterEvent_<std::__1::allocator<void> >, const rclcpp::SubscriptionOptionsWithAllocator<std::__1::allocator<void> > &, std::__1::allocator<void> >' required here
create_subscription(
^~~~~~~~~~~~~~~~~~~~
/Users/karsten/workspace/ros2/ros2_master/src/ros2/rclcpp/rclcpp/include/rclcpp/parameter_client.hpp:154:12: note: while substituting deduced template arguments into function template 'create_subscription' [with MessageT = rcl_interfaces::msg::ParameterEvent_<std::__1::allocator<void> >, CallbackT = const rclcpp::SubscriptionOptionsWithAllocator<std::__1::allocator<void> > &, AllocatorT = (no value), CallbackMessageT = (no value), SubscriptionT = (no value), MessageMemoryStrategyT = (no value), NodeT = std::__1::shared_ptr<rclcpp::node_interfaces::NodeTopicsInterface> &]
    return rclcpp::create_subscription<rcl_interfaces::msg::ParameterEvent>(
           ^

…ge to rcl_serialized_message_t and vice versa Signed-off-by: Joshua Hampp <j.hampp@denso-adas.de>

… for addind a deleter to ease up memory handling further features: * copy constructor (allowing static memory allocation, e.g. if for static memory allocation in device driver) * destructor Signed-off-by: Joshua Hampp <j.hampp@denso-adas.de>

* added flag for serialized communication * check for flag in "can_communicate" Signed-off-by: Joshua Hampp <j.hampp@denso-adas.de>

* distinguish between content type and callback type * use "SerializedContainer" for serialized messages for memory deletion * added specialized methods for combinations of (un)serialized content/callback * automatically (de)serialize messages * allowed communication types are now (MessageT==CallbackMessageT || MessageT==SerializedContainer || CallbackMessageT==rcl_serialized_message_t) Signed-off-by: Joshua Hampp <j.hampp@denso-adas.de>

… serialized communication in publisher base and subscriber base (adapted waitables) Signed-off-by: Joshua Hampp <j.hampp@denso-adas.de>

* implemented second communication channel for serialized intra process messages extended "Publisher" for serialized messages: * pass message type by argument * added constructor for backwards compatibility (moved common code in separate methods "init_setup") * added allocator for serialized messages Signed-off-by: Joshua Hampp <j.hampp@denso-adas.de>

* implemented second communication channel for serialized intra process messages * extend publish to handle "rcl_serialized_message_t" * changed behaviour of serialized message publishing by taking ownership of message (for deletion) Signed-off-by: Joshua Hampp <j.hampp@denso-adas.de>

Signed-off-by: Joshua Hampp <j.hampp@denso-adas.de>

…ssage type * backwards compatible * allows creation of publisher with type "rcl_serialized_message_t" Signed-off-by: Joshua Hampp <j.hampp@denso-adas.de>

Co-Authored-By: Karsten Knese <Karsten1987@users.noreply.github.com> Signed-off-by: Joshua Hampp <j.hampp@denso-adas.de>

* initiazlize variables Co-Authored-By: Karsten Knese <Karsten1987@users.noreply.github.com> Signed-off-by: Joshua Hampp <j.hampp@denso-adas.de>

Signed-off-by: Joshua Hampp <j.hampp@denso-adas.de>

* create_publisher to create_generic_publisher * create_subscription to create_generic_subscription Signed-off-by: Joshua Hampp <j.hampp@denso-adas.de>

DensoADAS · 2020-04-19T04:13:56Z

@Karsten1987 We do not have an osx machine available. From your error log I assume there is a problem with overloading the "create_subscription" function and template deduction. Therefore I renamed the "generic" one (create_generic_subscription/create_generic_publisher). Could you test this? Thank you.

Karsten1987 · 2020-04-19T07:20:08Z

@DensoADAS not sure if you've seen this, but I've opened a PR towards your original branch which addresses this: DensoADAS#1
There is also a second follow-up one, where I tried to summarize the other issues I had with this: DensoADAS#2

Karsten1987 · 2020-04-19T21:47:39Z

So I've just played around with a bit more and I actually think we should re-architecture this PR, maybe break it up into a few smaller ones even.

1.) Introduce a C++ version for the serialization. That includes the rclcpp::SerializedMessage and the rclcpp::Serialization. Additionally, the callback signature should be changed/enhanced to take the C++ class. The need for wrapping this into a shared pointer should vanish with this as well as the destructor of the C++ instance can make sure that the buffer is cleanup correctly.
That work was started in here:
DensoADAS@d6bf983 & DensoADAS@dd65645

2.) Create a template specialization for the publisher, which is instantiated only for rclcpp::SerializedMessage. This should be less impactful on the rclcpp::Publisher, the intra process manager and waitset API, as all the differentiation in the current PR (i.e. std::is_same<T, rcl_serialized_message_t>) can go away. The main motivation for this is that I don't see this scale up if new functionality gets added to the publisher as again the functions have to be cross-checked with a serialized message.
That also removes the need to constantly add two ids to the intra-process manager. This specialization could then further take the typesupport as a argument in the constructor whereas the other publisher does not need to be changed at all.
Work started here: https://gist.github.com/Karsten1987/bae218680ab400292dabd4537e516f64
That specialization has further benefits that all ambiguous member functions can go. In this specialization, there is no need to support a publish to a ROS message or if so, only provide a const void * ros_msg function. Similarly, a borrow_loaned_message can be implemented differently for this specialization - or not at all.
The differentiation template to be used can be done solely in the rclcpp::create_publisher function.
Also, in this case, I'd recommend to deprecate the publish(const rcl_serialized_message_t & msg) function overload on the regular Publisher.

3.) The same thing could technically be done for the subscription class as well. If the user wants to retrieve the data in a serialized way, rclcpp::create_subscription returns a specialized version of it. That doesn't really matter as for now given that we anyway hide this already away, but I think it would ease the change needed for the intra process communication.

The changes in 2.) might have drawbacks as to two different publisher instances have to be used when trying to publish ROS messages vs serialized messages. However, I generally also don't really see a big use-case for this in the first place. I could see how one wants to use one or the other publisher - c.f. rosbags when not being able to depend on the message types during compile time, but never really the mixed version.

DensoADAS · 2020-04-21T13:11:14Z

@Karsten1987 I replaced this PR by 3 new ones

Signed-off-by: Noy <noy@xtend.me>

DensoADAS mentioned this pull request Jan 30, 2020

Dnae adas/serialized intra process communication #819

Closed

DensoADAS force-pushed the dnae_adas/serialized_ipm branch from 7b8b5b5 to 2cecb4a Compare January 30, 2020 09:47

DensoADAS requested a review from wjwwood February 3, 2020 07:06

DensoADAS mentioned this pull request Feb 13, 2020

Dnae adas/sequential writing ros2/rosbag2#88

Closed

ivanpauno assigned wjwwood Feb 13, 2020

Karsten1987 assigned Karsten1987 and unassigned wjwwood Apr 16, 2020

Karsten1987 reviewed Apr 17, 2020

View reviewed changes

DensoADAS force-pushed the dnae_adas/serialized_ipm branch 2 times, most recently from 9f6d560 to 02571ac Compare April 17, 2020 17:28

DensoADAS force-pushed the dnae_adas/serialized_ipm branch from 02571ac to d8b65bb Compare April 17, 2020 18:08

Karsten1987 reviewed Apr 17, 2020

View reviewed changes

Joshua Hampp added 9 commits April 17, 2020 13:20

added "SerializationBase" and "Serialization" to convert a ROS2 messa…

9b73845

…ge to rcl_serialized_message_t and vice versa Signed-off-by: Joshua Hampp <j.hampp@denso-adas.de>

extended "IntraProcessManager" for serialized messages:

f3aab4f

* added flag for serialized communication * check for flag in "can_communicate" Signed-off-by: Joshua Hampp <j.hampp@denso-adas.de>

added secon communication channel for intra process communication for…

a300c2f

… serialized communication in publisher base and subscriber base (adapted waitables) Signed-off-by: Joshua Hampp <j.hampp@denso-adas.de>

updated "LifcecylePublisher" for serialized messages

c8522b3

Signed-off-by: Joshua Hampp <j.hampp@denso-adas.de>

extended "create_publisher" and "create_publisher_factory" to pass me…

29fea92

…ssage type * backwards compatible * allows creation of publisher with type "rcl_serialized_message_t" Signed-off-by: Joshua Hampp <j.hampp@denso-adas.de>

DensoADAS and others added 19 commits April 17, 2020 13:20

Update rclcpp/include/rclcpp/experimental/subscription_intra_process.hpp

657d9a0

Co-Authored-By: Karsten Knese <Karsten1987@users.noreply.github.com> Signed-off-by: Joshua Hampp <j.hampp@denso-adas.de>

* default deleter

9b65b7c

* initiazlize variables Co-Authored-By: Karsten Knese <Karsten1987@users.noreply.github.com> Signed-off-by: Joshua Hampp <j.hampp@denso-adas.de>

changes due to review of PR

18181f5

Signed-off-by: Joshua Hampp <j.hampp@denso-adas.de>

uncrustify

4c7506f

Signed-off-by: Joshua Hampp <j.hampp@denso-adas.de>

added backwards compatibility for publishing serialized messages

2cefdca

Signed-off-by: Joshua Hampp <j.hampp@denso-adas.de>

initializing variables

2147026

Signed-off-by: Joshua Hampp <j.hampp@denso-adas.de>

fix include guards

9b436fa

Signed-off-by: Joshua Hampp <j.hampp@denso-adas.de>

enabled publishing of rcl_serialized_message_t

a2fc8bd

Signed-off-by: Joshua Hampp <j.hampp@denso-adas.de>

moved serialization and serialized message out of "experimental"

b482164

Signed-off-by: Joshua Hampp <j.hampp@denso-adas.de>

added original constructor to LifcyclePublisher

93dc826

Signed-off-by: Joshua Hampp <j.hampp@denso-adas.de>

fixed year in header

35fd490

Signed-off-by: Joshua Hampp <j.hampp@denso-adas.de>

added comment

5472d36

Signed-off-by: Joshua Hampp <j.hampp@denso-adas.de>

updated loop signature

ab73a62

Signed-off-by: Joshua Hampp <j.hampp@denso-adas.de>

fixed memory loss

4fa5f09

Signed-off-by: Joshua Hampp <j.hampp@denso-adas.de>

reuse constructor

ff7a81f

Signed-off-by: Joshua Hampp <j.hampp@denso-adas.de>

renamed variable

408343f

Signed-off-by: Joshua Hampp <j.hampp@denso-adas.de>

fixed check

511476c

Signed-off-by: Joshua Hampp <j.hampp@denso-adas.de>

line break

1f8226a

Signed-off-by: Joshua Hampp <j.hampp@denso-adas.de>

fixed include order

2411642

Signed-off-by: Joshua Hampp <j.hampp@denso-adas.de>

Karsten1987 force-pushed the dnae_adas/serialized_ipm branch from 0c53533 to 2411642 Compare April 17, 2020 20:41

Karsten1987 mentioned this pull request Apr 17, 2020

make it compile on osx DensoADAS/rclcpp#1

Open

renamed overloaded functions:

43a8446

* create_publisher to create_generic_publisher * create_subscription to create_generic_subscription Signed-off-by: Joshua Hampp <j.hampp@denso-adas.de>

ivanpauno mentioned this pull request Apr 21, 2020

Dnae adas/generic subscriber and publisher #1077

Open

wjwwood mentioned this pull request Jul 12, 2021

intra-process communication with serialized message #1710

Open

clalancette changed the base branch from master to rolling June 28, 2022 14:28

nnmm pushed a commit to ApexAI/rclcpp that referenced this pull request Jul 9, 2022

Add additional null check for timer argument (ros2#973)

8ae0c9d

Signed-off-by: Noy <noy@xtend.me>

Dnae adas/serialized ipm #973

Are you sure you want to change the base?

Dnae adas/serialized ipm #973

Conversation

DensoADAS commented Jan 30, 2020

DensoADAS commented Jan 30, 2020

DensoADAS commented Feb 21, 2020 • edited Loading

wjwwood commented Feb 21, 2020

DensoADAS commented Apr 16, 2020 • edited Loading

ivanpauno commented Apr 16, 2020

Karsten1987 commented Apr 16, 2020

Karsten1987 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kfabian Apr 18, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

DensoADAS commented Apr 17, 2020

Karsten1987 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

DensoADAS commented Apr 17, 2020

Karsten1987 commented Apr 17, 2020 • edited Loading

DensoADAS commented Apr 19, 2020

Karsten1987 commented Apr 19, 2020

Karsten1987 commented Apr 19, 2020

DensoADAS commented Apr 21, 2020

DensoADAS commented Feb 21, 2020 •

edited

Loading

DensoADAS commented Apr 16, 2020 •

edited

Loading

kfabian Apr 18, 2020 •

edited

Loading

Karsten1987 commented Apr 17, 2020 •

edited

Loading