Speed up Decode and Encode timing

[xRowe]

I see there is a old thread on this topic, check if this topic can be carried on.
#362

Is there any new solution to speed up decoding and encoding timing?

Online receving Messages by python-can and decoding: canmatrix v.s cantools

Example Test Code

def on_message_received(msg):
        start = time.perf_counter_ns()

        dbc_msg = _database.frame_by_id(msg.arbitration_id)
        
        # start = time.perf_counter_ns()
        decoded = dbc_msg.decode(msg.data)
        end = time.perf_counter_ns()
        
       timings.append(end-start)

canmatrix
decode
Mean +- std dev: 49.1 us +- 36.5 us (2207 values)
Mean +- std dev: 97.4 us +- 182.5 us (12201 values)
received_message + decode
Mean +- std dev: 147.1 us +- 157.3 us (12201 values)

cantools
decode
Mean +- std dev: 30.0 us +- 29.9 us (2200 values)
Mean +- std dev: 34.8 us +- 35.9 us (12201 values)
received_message + decode
Mean +- std dev: 64.9 us +- 57.7 us (12201 values)

After looking at cantools solution, after database is parsed, it will generate a _codec info which allowed to more quickly encode/decode a message, and generate two dicts which allow quick search message by name or id.

But canmatrix, everytime when receive a raw message, I need to found it in database by using for test in self.frames:, when call decode() method, it try to upack the data by bytes_to_bitstrings() and bitstring_to_signal_list(), which also using for loop.

I do not if bitsruct may help better, and refer to cantools

[ebroecker]

Hi @xRowe

thanks for doing the measurements!

It would be easy to implement some dictionary to allow quick search message by name or id.
Thus I'm gonna implement such a hash-table which I think is a good idea.

Bitstruct on the other hand is (or at least was) not able to handle multiplexed messages.
Multiplexer enables messages(=frames/pdus) to have different layouts depending on the content of some bits.

[xRowe]

Thanks for your reply.

There are commits along with implementing the DICTs already,

canmatrix/src/canmatrix/canmatrix.py

Lines 1994 to 2002 in e7ce2aa

	def get_frame_by_id(self, id: int
	) -> typing.Union[Frame, None]:
	"""Get Frame by id.
	:param str name: Frame id to search for
	:rtype: Frame or None
	"""
	return self.frames_dict_id[id]

canmatrix/src/canmatrix/canmatrix.py

Lines 2029 to 2036 in e7ce2aa

	def get_frame_by_name(self, name): # type: (str) -> typing.Union[Frame, None]
	"""Get Frame by name.
	:param str name: Frame name to search for
	:rtype: Frame or None
	"""
	return self.frames_dict_name[name]

And I create/add the data here, but I think the way is not good enough, and not cover all the cases.

canmatrix/src/canmatrix/canmatrix.py

Lines 2075 to 2089 in e7ce2aa

	def add_frame(self, frame): # type: (Frame) -> Frame
	"""Add the Frame to the Matrix.
	:param Frame frame: Frame to add
	:return: the inserted Frame
	"""
	self.frames.append(frame)
	self.frames_dict_name[frame.name] = frame
	if frame.header_id:
	self.frames_dict_id[frame.header_id] = frame
	elif frame.arbitration_id.id:
	self.frames_dict_id[frame.arbitration_id.id] = frame
	return self.frames[len(self.frames) - 1]

Though my project is running these code, I did not run the benchmark then.

If you have better idea, please improve it.

Thanks,
Rowe