byte order and floats, what am I doing wrong?

[haisamido]

byte order and floats, what am I doing wrong?

I have my own python code, below, that successfully returns the float for MNEMONIC, whose binary byte order is 3412.

mnemonic_ba = bytearray([ byte_arr[offset+36+1], byte_arr[offset+36+0], byte_arr[offset+36+3], byte_arr[offset+36+2] ])
MNEMONIC    = (struct.unpack('<f', mnemonic_ba))[0]

However, when I use ccsdspy, with the below definition for MNEMONIC with byte_order little or big

PacketField(name='MNEMONIC',    data_type='float', bit_length=32, byte_order='little'),

the values that are returned are wrong and are either extremely large or extremely small float values. However, all uint's that I am returning by ccsdspy are correctly returned. The issue I'm having is only with floats.

[ddasilva]

Maybe I'm missing something, but I haven't encountered 3412 before. The little/big endian orders are 4321 and 1234... is there a name for this 3412? Does it happen a lot in your use cases?

What you could do in CCSDSPy is parse two uint fields, one for bytes 34 and one for bytes 12, and then write a custom converter to combine them. Something like this:

from ccsdspy import FixedLength, converters

class CustomConverter(converters.Converter):
    def __init__(self):
        pass
    def convert(self, first, second):
        # combine them here

pkt = FixedLength([
    PacketField(name="first", data_type="uint", bit_length=16),
    PacketField(name="second", data_type="uint", bit_length=16),
])
pkt.add_converted_field(
    ("first", "second"),
    "MyField_Converted",
    CustomConverter()
)

result = pkt.load("MyCCSDS.bin")

print(result["MyField_Converted"])

I haven't tested this code to implement # combine them here, but I think you could do something like this:

    def convert(self, first, second):
        combined = np.array([first, second], dtype=np.uint16).T.flatten()
        combined.dtype = np.float32
        return combined

[haisamido]

3412 is very common where I work. It's the dominant byte order. I think permitting the option of providing numeric representation of byte order would be very useful.

…

On Fri, Feb 9, 2024, 22:45 Daniel da Silva ***@***.***> wrote: Maybe I'm missing something, but I haven't encountered 3412 before. The little/big endian orders are 4321 and 1234... is there a name for this 3412? Does it happen a lot in your use cases? What you could do in CCSDSPy is parse two uint fields, one for bytes 34 and one for bytes 12, and then write a custom converter <https://docs.ccsdspy.org/en/latest/user-guide/converters.html> to combine them. Something like this: from ccsdspy import FixedLength, converters class CustomConverter(converters.Converter): def __init__(self): pass def convert(self, first, second): # combine them here pkt = FixedLength([ PacketField(name="first", data_type="uint", bit_length=16), PacketField(name="second", data_type="uint", bit_length=16), ])pkt.add_converted_field( ("first", "second"), "MyField_Converted", CustomConverter() ) result = pkt.load("MyCCSDS.bin") print(result["MyField_Converted"]) I haven't tested this code to implement # combine them here, but I think you could do something like this: def convert(self, first, second): combined = np.array([first, second], dtype=np.uint16).T.flatten() combined.dtype = np.float32 return combined — Reply to this email directly, view it on GitHub <#110 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AADJ6H23OMQGKDALUM62FVLYS3UMZAVCNFSM6AAAAABDCBX7A2VHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4DIMRVGE2DO> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[haisamido]

Thank you. The exception is gone with the new patch; however, the results still don't look correct. The float values that are being returned for the below fields

PacketField(name='FOOBAR1', data_type='float', bit_length=32, byte_order='3412'),
PacketField(name='FOOBAR2', data_type='float', bit_length=32, byte_order='3412'),
PacketField(name='FOOBAR3', data_type='float', bit_length=32, byte_order='3412'),
PacketField(name='FOOBAR4', data_type='float', bit_length=32, byte_order='3412'),

are summarized below

#name, expected float, returned float
FOOBAR1,-59.253719, -2205008400.0
FOOBAR2,65.153366, -3.8839992e-35
FOOBAR3,-37.579803,-3.955661e-19
FOOBAR4,64.456299,-4.9850678e-05

BTW the below returns the expected uint value

PacketField(name='FOOBARCNT', data_type='uint',  bit_length=16, byte_order='21')

of 64960 but byte_order='21' should really be byte_order='12' to return the correct value. So it seems things are reversed? While byte_order='big' returns the correct and expected value of 64960

[haisamido]

@ddasilva when I used byte_order='2134' the float values are much closer to correct ones, but not really within precision. However, byte_order='3412' should be the one that's returning the correct values.

#name, expected float, returned float
FOOBAR1,-59.253719, -59.45216
FOOBAR2,65.153366, 65.262314
FOOBAR3,-37.579803,-37.679996
FOOBAR4,64.456299,64.31428

[haisamido]

With byte_order='2143' the returned results are correct, and with the desired precision, while byte_order='3412' should be the one that's returning the correct results. We're getting closer :-) That's the power of the integer representation of byte_order.

[ddasilva]

Great! I changed two lines (basically added byte_order.reverse() to the library and test data)... let me know if it is better now.

[haisamido]

worked like a charm! Thank you @ddasilva :-) When do you think this branch will be part of main?

[ddasilva]

Great! are you okay working off that branch for now? I will find some time to write the proper documentation and add some additional tests later this week.

Can you also try with your 6 byte floats and let me know what happens? 6 byte floats are a little more dicey because it's not a natural length for floats (it's not clear to me what you expect to happen).

[haisamido]

I don't have actual data for 6 byte floats.

[ddasilva]

Sounds good, thanks!