Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

xtce_converter doesn't support byte order list #268

Closed
severn opened this issue Mar 21, 2016 · 13 comments
Closed

xtce_converter doesn't support byte order list #268

severn opened this issue Mar 21, 2016 · 13 comments
Labels
Milestone

Comments

@severn
Copy link

@severn severn commented Mar 21, 2016

To be fair I haven't done a true in depth look at this, just a quick run through of a single packet xtce file I know is correct from a mission. The xtce_converter doesn't support several elements, which is fine overall -- but the byte order list element is pretty important to us as although we are principally big-endian, this is not always the case for all mnemonics in our database however (and to follow through on it here -- I have to double check whether it is the case in this one packet in which case I can remove the element which we probably generate by default regardless). As the program runs, it produces a fair number of issues along those lines before dying as follows (I'm guess it's the poly exponent which can be 1.0 in straight XTCE 1.1 -- and wonder if you have a variant as this is being changed in xtce 1.2). Anyway thanks for reading...

 Ignoring Unknown: <ByteOrderList>
  Ignoring Unknown: <Byte>
  Ignoring Unknown: <Byte>
/root/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/cosmos-3.8.0/lib/cosmos/packets/packet_config.rb:584:in `Integer': invalid value for Integer(): "0.0" (ArgumentError)
    from /root/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/cosmos-3.8.0/lib/cosmos/packets/packet_config.rb:584:in `block in xtce_process_element'
    from /root/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/cosmos-3.8.0/lib/cosmos/packets/packet_config.rb:840:in `xtce_recurse_element'
    from /root/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/cosmos-3.8.0/lib/cosmos/packets/packet_config.rb:842:in `block in xtce_recurse_element'
    from /root/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/nokogiri-1.6.7.2/lib/nokogiri/xml/node_set.rb:187:in `block in each'
    from /root/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/nokogiri-1.6.7.2/lib/nokogiri/xml/node_set.rb:186:in `upto'
    from /root/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/nokogiri-1.6.7.2/lib/nokogiri/xml/node_set.rb:186:in `each'
    from /root/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/cosmos-3.8.0/lib/cosmos/packets/packet_config.rb:841:in `xtce_recurse_element'
    from /root/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/cosmos-3.8.0/lib/cosmos/packets/packet_config.rb:582:in `xtce_process_element'
    from /root/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/cosmos-3.8.0/lib/cosmos/packets/packet_config.rb:217:in `block (2 levels) in process_xtce'
    from /root/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/cosmos-3.8.0/lib/cosmos/packets/packet_config.rb:840:in `xtce_recurse_element'
    from /root/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/cosmos-3.8.0/lib/cosmos/packets/packet_config.rb:842:in `block in xtce_recurse_element'
    from /root/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/nokogiri-1.6.7.2/lib/nokogiri/xml/node_set.rb:187:in `block in each'
    from /root/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/nokogiri-1.6.7.2/lib/nokogiri/xml/node_set.rb:186:in `upto'
    from /root/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/nokogiri-1.6.7.2/lib/nokogiri/xml/node_set.rb:186:in `each'
    from /root/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/cosmos-3.8.0/lib/cosmos/packets/packet_config.rb:841:in `xtce_recurse_element'
    from /root/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/cosmos-3.8.0/lib/cosmos/packets/packet_config.rb:842:in `block in xtce_recurse_element'
    from /root/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/nokogiri-1.6.7.2/lib/nokogiri/xml/node_set.rb:187:in `block in each'
    from /root/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/nokogiri-1.6.7.2/lib/nokogiri/xml/node_set.rb:186:in `upto'
    from /root/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/nokogiri-1.6.7.2/lib/nokogiri/xml/node_set.rb:186:in `each'
    from /root/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/cosmos-3.8.0/lib/cosmos/packets/packet_config.rb:841:in `xtce_recurse_element'
    from /root/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/cosmos-3.8.0/lib/cosmos/packets/packet_config.rb:842:in `block in xtce_recurse_element'
    from /root/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/nokogiri-1.6.7.2/lib/nokogiri/xml/node_set.rb:187:in `block in each'
    from /root/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/nokogiri-1.6.7.2/lib/nokogiri/xml/node_set.rb:186:in `upto'
    from /root/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/nokogiri-1.6.7.2/lib/nokogiri/xml/node_set.rb:186:in `each'
    from /root/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/cosmos-3.8.0/lib/cosmos/packets/packet_config.rb:841:in `xtce_recurse_element'
    from /root/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/cosmos-3.8.0/lib/cosmos/packets/packet_config.rb:842:in `block in xtce_recurse_element'
    from /root/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/nokogiri-1.6.7.2/lib/nokogiri/xml/node_set.rb:187:in `block in each'
    from /root/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/nokogiri-1.6.7.2/lib/nokogiri/xml/node_set.rb:186:in `upto'
    from /root/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/nokogiri-1.6.7.2/lib/nokogiri/xml/node_set.rb:186:in `each'
    from /root/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/cosmos-3.8.0/lib/cosmos/packets/packet_config.rb:841:in `xtce_recurse_element'
    from /root/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/cosmos-3.8.0/lib/cosmos/packets/packet_config.rb:842:in `block in xtce_recurse_element'
    from /root/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/nokogiri-1.6.7.2/lib/nokogiri/xml/node_set.rb:187:in `block in each'
    from /root/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/nokogiri-1.6.7.2/lib/nokogiri/xml/node_set.rb:186:in `upto'
    from /root/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/nokogiri-1.6.7.2/lib/nokogiri/xml/node_set.rb:186:in `each'
    from /root/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/cosmos-3.8.0/lib/cosmos/packets/packet_config.rb:841:in `xtce_recurse_element'
    from /root/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/cosmos-3.8.0/lib/cosmos/packets/packet_config.rb:216:in `block in process_xtce'
    from /root/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/nokogiri-1.6.7.2/lib/nokogiri/xml/node_set.rb:187:in `block in each'
    from /root/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/nokogiri-1.6.7.2/lib/nokogiri/xml/node_set.rb:186:in `upto'
    from /root/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/nokogiri-1.6.7.2/lib/nokogiri/xml/node_set.rb:186:in `each'
    from /root/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/cosmos-3.8.0/lib/cosmos/packets/packet_config.rb:215:in `process_xtce'
    from /root/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/cosmos-3.8.0/bin/xtce_converter:71:in `<top (required)>'
    from /root/.rbenv/versions/2.2.2/bin/xtce_converter:23:in `load'
    from /root/.rbenv/versions/2.2.2/bin/xtce_converter:23:in `<main>'
@ryanatball
Copy link
Member

@ryanatball ryanatball commented Mar 24, 2016

Your crash is an easy fix, I was requiring something to be an Integer that is specified as a Float. Already fixed. I'll look into supporting ByteOrderList next. Any other required elements?

ryanatball added a commit that referenced this issue Mar 24, 2016
@ryanatball
Copy link
Member

@ryanatball ryanatball commented Mar 24, 2016

ByteOrderList is now supported in the handle_xtce_byte_order_list branch. I'll wait for your response before merging.

@severn
Copy link
Author

@severn severn commented Mar 26, 2016

I have 4 large xtce file to run through it ultimately. I've paired it down to 1 representative but simple-ish packet each. But I can simplify these further still. For example I can remove limits and I can flatten or unroll things like arrays. I can "rename" aggregates -- dotted named mnemonics also as a temporary measure. Ultimately I may want them but for now many items can be worked around.

LocationInContaierInBit is another that comes to my mind. Our translation software calculates and hardcodes the packet entry offsets. Generally though they are all actually "packed" although not necessarily on byte boundaries if their actual telemetered length is < 8 bits. Anyway I could probably strip the location in container bits out on a known good xtce telemetry description, etc... that's likely to be case at least.

@severn
Copy link
Author

@severn severn commented Apr 7, 2016

It's taken me a bit to get back to this. I clearly don't quite know what I'm doing with Ruby but I managed by hand copying paket_config.rb from the branch into the installed Ruby subtree for cosmos -- that I see that the new ByteOrderList is being exercised. I can't really verify that it works but there is less complaining in the unsupported element messages from the converter.

Unfortunately I have lots of other elements it does not like as well:

-EnumerationAlarm & related
-AncillaryData (which is fair to not support)
-ArrayParameterType
-AggregrateParameterType

Those are the top items. But I can make these go away for an initial cut, so I'll do that first...

(note: again this 1 packet from a 1000 packet database, xtce file)

@ryanatball ryanatball added the feature label Apr 11, 2016
@severn
Copy link
Author

@severn severn commented Apr 18, 2016

I removed the EnumerationAlarmLIst because this is not supported. Now for my 1 packet scenario I'm down to (I believe) the absolute addressing (LocationInContainerInBits) in container parameter entry.

Does the cosmos target DB support addressing its packet/container definition? (I'm poking around for documentation)

@severn
Copy link
Author

@severn severn commented Apr 18, 2016

I suppose this is a new issues... let me do that.

@severn
Copy link
Author

@severn severn commented May 1, 2016

Ryan what I get after finally have a chance to look into the COSMOS telemetry database description language is that byte order is only supported on a per packet basis...

Whereas I was really after it being supported on a per mnemonic basis...

Am I missing the obvious?

What's your feeling on difficulty of adding that support: change to ITEM syntax of telemetry (& command but that can wait) to add a byte order list option, change to internal "decom" code to support it... (I assume this exists but I have no idea where it'd be in the code tree)...

Anything else?

@ryanatball
Copy link
Member

@ryanatball ryanatball commented May 2, 2016

Bit endianness is supported per item in the COSMOS config files, though it appears the web documentation is out of date. Each of the ITEM, PARAMETER, etc keywords takes another optional parameter after description of endianness. So just add BIG_ENDIAN or LITTLE_ENDIAN after the description and you can change the endianness of the specific item.

@severn
Copy link
Author

@severn severn commented May 2, 2016

Thanks for getting back to me. Someone in between posting my posting this earlier and your comment suggested the same to me based on a command definition they had or had seen. I haven't had time to try it.

I believe this would of course cover %90+ of our byte orderings. Most of the time we strive where we have control for big endian. However I am aware that sometimes we have other byte orders. Little endian would be likely.

Even so I believe this is not guaranteed and historically we've had to support other strange orders -- on a per mnemonic basis.

Given that: both of our in house tool chains support giving the byte order basically as a comma delimited list against the byte length of the mnemonic.

For example a 16 bit mnemonic might be byte order 1,2 or 2,1 and so forth.

Any feeling on where in the code base one might add such? Level of difficulty (for me to try?)

@ryanatball
Copy link
Member

@ryanatball ryanatball commented May 2, 2016

The only other endianness I've ever seen is sometimes when using 1553 you'll see the two 16-bit fields that are each byte swapped individually, but together make up a 32-bit word. (If that makes sense).

Binary Accessor is what does the reading/writing of values and what handles endianness. So that would be the first place to make the change.

lib/cosmos/packets/binary_accessor.rb.

Most of the code is actually written in C at:

ext/cosmos/ext/structure/structure.c

I would rate the change as pretty hard and not worth it. For weird endianness cases you can always write a few lines of Ruby in a conversion to handle the weirdness.

@ryanatball
Copy link
Member

@ryanatball ryanatball commented May 3, 2016

I've just checked in support for LocationInContainerInBits and Arrays into the handle_xtce_byte_order_list branch. If you could do a pull on that branch and try out the changes it would be appreciated.

@ryanatball
Copy link
Member

@ryanatball ryanatball commented May 5, 2016

Closed by pull request #291

@ryanatball ryanatball closed this May 5, 2016
@severn
Copy link
Author

@severn severn commented May 5, 2016

I will very soon give it a go, sorry for the delay!

@ryanatball ryanatball modified the milestone: v3.8.1 May 12, 2016
@ryanatball ryanatball added maintenance and removed feature labels May 12, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
2 participants
You can’t perform that action at this time.