xtce_converter doesn't support byte order list #268

Closed
severn opened this Issue Mar 21, 2016 · 13 comments

Comments

Projects
None yet
2 participants
@severn

severn commented Mar 21, 2016

To be fair I haven't done a true in depth look at this, just a quick run through of a single packet xtce file I know is correct from a mission. The xtce_converter doesn't support several elements, which is fine overall -- but the byte order list element is pretty important to us as although we are principally big-endian, this is not always the case for all mnemonics in our database however (and to follow through on it here -- I have to double check whether it is the case in this one packet in which case I can remove the element which we probably generate by default regardless). As the program runs, it produces a fair number of issues along those lines before dying as follows (I'm guess it's the poly exponent which can be 1.0 in straight XTCE 1.1 -- and wonder if you have a variant as this is being changed in xtce 1.2). Anyway thanks for reading...

 Ignoring Unknown: <ByteOrderList>
  Ignoring Unknown: <Byte>
  Ignoring Unknown: <Byte>
/root/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/cosmos-3.8.0/lib/cosmos/packets/packet_config.rb:584:in `Integer': invalid value for Integer(): "0.0" (ArgumentError)
    from /root/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/cosmos-3.8.0/lib/cosmos/packets/packet_config.rb:584:in `block in xtce_process_element'
    from /root/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/cosmos-3.8.0/lib/cosmos/packets/packet_config.rb:840:in `xtce_recurse_element'
    from /root/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/cosmos-3.8.0/lib/cosmos/packets/packet_config.rb:842:in `block in xtce_recurse_element'
    from /root/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/nokogiri-1.6.7.2/lib/nokogiri/xml/node_set.rb:187:in `block in each'
    from /root/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/nokogiri-1.6.7.2/lib/nokogiri/xml/node_set.rb:186:in `upto'
    from /root/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/nokogiri-1.6.7.2/lib/nokogiri/xml/node_set.rb:186:in `each'
    from /root/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/cosmos-3.8.0/lib/cosmos/packets/packet_config.rb:841:in `xtce_recurse_element'
    from /root/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/cosmos-3.8.0/lib/cosmos/packets/packet_config.rb:582:in `xtce_process_element'
    from /root/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/cosmos-3.8.0/lib/cosmos/packets/packet_config.rb:217:in `block (2 levels) in process_xtce'
    from /root/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/cosmos-3.8.0/lib/cosmos/packets/packet_config.rb:840:in `xtce_recurse_element'
    from /root/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/cosmos-3.8.0/lib/cosmos/packets/packet_config.rb:842:in `block in xtce_recurse_element'
    from /root/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/nokogiri-1.6.7.2/lib/nokogiri/xml/node_set.rb:187:in `block in each'
    from /root/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/nokogiri-1.6.7.2/lib/nokogiri/xml/node_set.rb:186:in `upto'
    from /root/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/nokogiri-1.6.7.2/lib/nokogiri/xml/node_set.rb:186:in `each'
    from /root/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/cosmos-3.8.0/lib/cosmos/packets/packet_config.rb:841:in `xtce_recurse_element'
    from /root/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/cosmos-3.8.0/lib/cosmos/packets/packet_config.rb:842:in `block in xtce_recurse_element'
    from /root/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/nokogiri-1.6.7.2/lib/nokogiri/xml/node_set.rb:187:in `block in each'
    from /root/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/nokogiri-1.6.7.2/lib/nokogiri/xml/node_set.rb:186:in `upto'
    from /root/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/nokogiri-1.6.7.2/lib/nokogiri/xml/node_set.rb:186:in `each'
    from /root/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/cosmos-3.8.0/lib/cosmos/packets/packet_config.rb:841:in `xtce_recurse_element'
    from /root/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/cosmos-3.8.0/lib/cosmos/packets/packet_config.rb:842:in `block in xtce_recurse_element'
    from /root/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/nokogiri-1.6.7.2/lib/nokogiri/xml/node_set.rb:187:in `block in each'
    from /root/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/nokogiri-1.6.7.2/lib/nokogiri/xml/node_set.rb:186:in `upto'
    from /root/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/nokogiri-1.6.7.2/lib/nokogiri/xml/node_set.rb:186:in `each'
    from /root/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/cosmos-3.8.0/lib/cosmos/packets/packet_config.rb:841:in `xtce_recurse_element'
    from /root/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/cosmos-3.8.0/lib/cosmos/packets/packet_config.rb:842:in `block in xtce_recurse_element'
    from /root/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/nokogiri-1.6.7.2/lib/nokogiri/xml/node_set.rb:187:in `block in each'
    from /root/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/nokogiri-1.6.7.2/lib/nokogiri/xml/node_set.rb:186:in `upto'
    from /root/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/nokogiri-1.6.7.2/lib/nokogiri/xml/node_set.rb:186:in `each'
    from /root/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/cosmos-3.8.0/lib/cosmos/packets/packet_config.rb:841:in `xtce_recurse_element'
    from /root/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/cosmos-3.8.0/lib/cosmos/packets/packet_config.rb:842:in `block in xtce_recurse_element'
    from /root/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/nokogiri-1.6.7.2/lib/nokogiri/xml/node_set.rb:187:in `block in each'
    from /root/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/nokogiri-1.6.7.2/lib/nokogiri/xml/node_set.rb:186:in `upto'
    from /root/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/nokogiri-1.6.7.2/lib/nokogiri/xml/node_set.rb:186:in `each'
    from /root/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/cosmos-3.8.0/lib/cosmos/packets/packet_config.rb:841:in `xtce_recurse_element'
    from /root/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/cosmos-3.8.0/lib/cosmos/packets/packet_config.rb:216:in `block in process_xtce'
    from /root/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/nokogiri-1.6.7.2/lib/nokogiri/xml/node_set.rb:187:in `block in each'
    from /root/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/nokogiri-1.6.7.2/lib/nokogiri/xml/node_set.rb:186:in `upto'
    from /root/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/nokogiri-1.6.7.2/lib/nokogiri/xml/node_set.rb:186:in `each'
    from /root/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/cosmos-3.8.0/lib/cosmos/packets/packet_config.rb:215:in `process_xtce'
    from /root/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/cosmos-3.8.0/bin/xtce_converter:71:in `<top (required)>'
    from /root/.rbenv/versions/2.2.2/bin/xtce_converter:23:in `load'
    from /root/.rbenv/versions/2.2.2/bin/xtce_converter:23:in `<main>'
@ryanatball

This comment has been minimized.

Show comment
Hide comment
@ryanatball

ryanatball Mar 24, 2016

Member

Your crash is an easy fix, I was requiring something to be an Integer that is specified as a Float. Already fixed. I'll look into supporting ByteOrderList next. Any other required elements?

Member

ryanatball commented Mar 24, 2016

Your crash is an easy fix, I was requiring something to be an Integer that is specified as a Float. Already fixed. I'll look into supporting ByteOrderList next. Any other required elements?

ryanatball added a commit that referenced this issue Mar 24, 2016

@ryanatball

This comment has been minimized.

Show comment
Hide comment
@ryanatball

ryanatball Mar 24, 2016

Member

ByteOrderList is now supported in the handle_xtce_byte_order_list branch. I'll wait for your response before merging.

Member

ryanatball commented Mar 24, 2016

ByteOrderList is now supported in the handle_xtce_byte_order_list branch. I'll wait for your response before merging.

@severn

This comment has been minimized.

Show comment
Hide comment
@severn

severn Mar 26, 2016

I have 4 large xtce file to run through it ultimately. I've paired it down to 1 representative but simple-ish packet each. But I can simplify these further still. For example I can remove limits and I can flatten or unroll things like arrays. I can "rename" aggregates -- dotted named mnemonics also as a temporary measure. Ultimately I may want them but for now many items can be worked around.

LocationInContaierInBit is another that comes to my mind. Our translation software calculates and hardcodes the packet entry offsets. Generally though they are all actually "packed" although not necessarily on byte boundaries if their actual telemetered length is < 8 bits. Anyway I could probably strip the location in container bits out on a known good xtce telemetry description, etc... that's likely to be case at least.

severn commented Mar 26, 2016

I have 4 large xtce file to run through it ultimately. I've paired it down to 1 representative but simple-ish packet each. But I can simplify these further still. For example I can remove limits and I can flatten or unroll things like arrays. I can "rename" aggregates -- dotted named mnemonics also as a temporary measure. Ultimately I may want them but for now many items can be worked around.

LocationInContaierInBit is another that comes to my mind. Our translation software calculates and hardcodes the packet entry offsets. Generally though they are all actually "packed" although not necessarily on byte boundaries if their actual telemetered length is < 8 bits. Anyway I could probably strip the location in container bits out on a known good xtce telemetry description, etc... that's likely to be case at least.

@severn

This comment has been minimized.

Show comment
Hide comment
@severn

severn Apr 7, 2016

It's taken me a bit to get back to this. I clearly don't quite know what I'm doing with Ruby but I managed by hand copying paket_config.rb from the branch into the installed Ruby subtree for cosmos -- that I see that the new ByteOrderList is being exercised. I can't really verify that it works but there is less complaining in the unsupported element messages from the converter.

Unfortunately I have lots of other elements it does not like as well:

-EnumerationAlarm & related
-AncillaryData (which is fair to not support)
-ArrayParameterType
-AggregrateParameterType

Those are the top items. But I can make these go away for an initial cut, so I'll do that first...

(note: again this 1 packet from a 1000 packet database, xtce file)

severn commented Apr 7, 2016

It's taken me a bit to get back to this. I clearly don't quite know what I'm doing with Ruby but I managed by hand copying paket_config.rb from the branch into the installed Ruby subtree for cosmos -- that I see that the new ByteOrderList is being exercised. I can't really verify that it works but there is less complaining in the unsupported element messages from the converter.

Unfortunately I have lots of other elements it does not like as well:

-EnumerationAlarm & related
-AncillaryData (which is fair to not support)
-ArrayParameterType
-AggregrateParameterType

Those are the top items. But I can make these go away for an initial cut, so I'll do that first...

(note: again this 1 packet from a 1000 packet database, xtce file)

@ryanatball ryanatball added the feature label Apr 11, 2016

@severn

This comment has been minimized.

Show comment
Hide comment
@severn

severn Apr 18, 2016

I removed the EnumerationAlarmLIst because this is not supported. Now for my 1 packet scenario I'm down to (I believe) the absolute addressing (LocationInContainerInBits) in container parameter entry.

Does the cosmos target DB support addressing its packet/container definition? (I'm poking around for documentation)

severn commented Apr 18, 2016

I removed the EnumerationAlarmLIst because this is not supported. Now for my 1 packet scenario I'm down to (I believe) the absolute addressing (LocationInContainerInBits) in container parameter entry.

Does the cosmos target DB support addressing its packet/container definition? (I'm poking around for documentation)

@severn

This comment has been minimized.

Show comment
Hide comment
@severn

severn Apr 18, 2016

I suppose this is a new issues... let me do that.

severn commented Apr 18, 2016

I suppose this is a new issues... let me do that.

@severn

This comment has been minimized.

Show comment
Hide comment
@severn

severn May 1, 2016

Ryan what I get after finally have a chance to look into the COSMOS telemetry database description language is that byte order is only supported on a per packet basis...

Whereas I was really after it being supported on a per mnemonic basis...

Am I missing the obvious?

What's your feeling on difficulty of adding that support: change to ITEM syntax of telemetry (& command but that can wait) to add a byte order list option, change to internal "decom" code to support it... (I assume this exists but I have no idea where it'd be in the code tree)...

Anything else?

severn commented May 1, 2016

Ryan what I get after finally have a chance to look into the COSMOS telemetry database description language is that byte order is only supported on a per packet basis...

Whereas I was really after it being supported on a per mnemonic basis...

Am I missing the obvious?

What's your feeling on difficulty of adding that support: change to ITEM syntax of telemetry (& command but that can wait) to add a byte order list option, change to internal "decom" code to support it... (I assume this exists but I have no idea where it'd be in the code tree)...

Anything else?

@ryanatball

This comment has been minimized.

Show comment
Hide comment
@ryanatball

ryanatball May 2, 2016

Member

Bit endianness is supported per item in the COSMOS config files, though it appears the web documentation is out of date. Each of the ITEM, PARAMETER, etc keywords takes another optional parameter after description of endianness. So just add BIG_ENDIAN or LITTLE_ENDIAN after the description and you can change the endianness of the specific item.

Member

ryanatball commented May 2, 2016

Bit endianness is supported per item in the COSMOS config files, though it appears the web documentation is out of date. Each of the ITEM, PARAMETER, etc keywords takes another optional parameter after description of endianness. So just add BIG_ENDIAN or LITTLE_ENDIAN after the description and you can change the endianness of the specific item.

@severn

This comment has been minimized.

Show comment
Hide comment
@severn

severn May 2, 2016

Thanks for getting back to me. Someone in between posting my posting this earlier and your comment suggested the same to me based on a command definition they had or had seen. I haven't had time to try it.

I believe this would of course cover %90+ of our byte orderings. Most of the time we strive where we have control for big endian. However I am aware that sometimes we have other byte orders. Little endian would be likely.

Even so I believe this is not guaranteed and historically we've had to support other strange orders -- on a per mnemonic basis.

Given that: both of our in house tool chains support giving the byte order basically as a comma delimited list against the byte length of the mnemonic.

For example a 16 bit mnemonic might be byte order 1,2 or 2,1 and so forth.

Any feeling on where in the code base one might add such? Level of difficulty (for me to try?)

severn commented May 2, 2016

Thanks for getting back to me. Someone in between posting my posting this earlier and your comment suggested the same to me based on a command definition they had or had seen. I haven't had time to try it.

I believe this would of course cover %90+ of our byte orderings. Most of the time we strive where we have control for big endian. However I am aware that sometimes we have other byte orders. Little endian would be likely.

Even so I believe this is not guaranteed and historically we've had to support other strange orders -- on a per mnemonic basis.

Given that: both of our in house tool chains support giving the byte order basically as a comma delimited list against the byte length of the mnemonic.

For example a 16 bit mnemonic might be byte order 1,2 or 2,1 and so forth.

Any feeling on where in the code base one might add such? Level of difficulty (for me to try?)

@ryanatball

This comment has been minimized.

Show comment
Hide comment
@ryanatball

ryanatball May 2, 2016

Member

The only other endianness I've ever seen is sometimes when using 1553 you'll see the two 16-bit fields that are each byte swapped individually, but together make up a 32-bit word. (If that makes sense).

Binary Accessor is what does the reading/writing of values and what handles endianness. So that would be the first place to make the change.

lib/cosmos/packets/binary_accessor.rb.

Most of the code is actually written in C at:

ext/cosmos/ext/structure/structure.c

I would rate the change as pretty hard and not worth it. For weird endianness cases you can always write a few lines of Ruby in a conversion to handle the weirdness.

Member

ryanatball commented May 2, 2016

The only other endianness I've ever seen is sometimes when using 1553 you'll see the two 16-bit fields that are each byte swapped individually, but together make up a 32-bit word. (If that makes sense).

Binary Accessor is what does the reading/writing of values and what handles endianness. So that would be the first place to make the change.

lib/cosmos/packets/binary_accessor.rb.

Most of the code is actually written in C at:

ext/cosmos/ext/structure/structure.c

I would rate the change as pretty hard and not worth it. For weird endianness cases you can always write a few lines of Ruby in a conversion to handle the weirdness.

@ryanatball

This comment has been minimized.

Show comment
Hide comment
@ryanatball

ryanatball May 3, 2016

Member

I've just checked in support for LocationInContainerInBits and Arrays into the handle_xtce_byte_order_list branch. If you could do a pull on that branch and try out the changes it would be appreciated.

Member

ryanatball commented May 3, 2016

I've just checked in support for LocationInContainerInBits and Arrays into the handle_xtce_byte_order_list branch. If you could do a pull on that branch and try out the changes it would be appreciated.

@ryanatball

This comment has been minimized.

Show comment
Hide comment
@ryanatball

ryanatball May 5, 2016

Member

Closed by pull request #291

Member

ryanatball commented May 5, 2016

Closed by pull request #291

@ryanatball ryanatball closed this May 5, 2016

@severn

This comment has been minimized.

Show comment
Hide comment
@severn

severn May 5, 2016

I will very soon give it a go, sorry for the delay!

severn commented May 5, 2016

I will very soon give it a go, sorry for the delay!

@ryanatball ryanatball modified the milestone: v3.8.1 May 12, 2016

@ryanatball ryanatball added maintenance and removed feature labels May 12, 2016

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment