Python: Numba based SBP parsing [ESD-1154] #656
Closed
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
silverjam
changed the title
silverjam
changed the title
Contributor
|
|
Contributor
|
Changes for Python 2.7. Not sure how legit these are. After this Python 3.5 still failing |
Contributor
|
Changes for py3 |
pmiettinen
reviewed
Mar 25, 2019
Contributor
|
Floating point parsing produces different results than the original I have a log from one TTFF run and fix reported in the json file from sbp2json.py serial_link.py Latter matches with Haskell |
Contributor
|
Solution for above: |
Contributor
|
String handling doesn't cope with the settings NULL delimiter |
Contributor
|
|
Contributor
|
Possible solution for the settings NULL delimiter issue: |
Contributor
|
At the end of every Example: |
Contributor
This doesn't work with |
Contributor
|
Something like this instead: |
Contributor
|
Regarding the newline on the payload value. https://buildmedia.readthedocs.org/media/pdf/pybase64/latest/pybase64.pdf
To mitigate it and convert to str object at the same time: |
Contributor
|
Some failure happening and empty JSON object is printed in case of few messages sbp2json.py Expected Note also |
silverjam
force-pushed
the
silverjam/numba-jit2
branch
from
8efa20d to
bbae725
Compare
pmiettinen
force-pushed
the
silverjam/numba-jit2
branch
8 times, most recently
from
da9e562 to
601e04e
Compare
Contributor
|
In case of CRC failure, few following messages are lost compared to non-jit sbp2json. |
pmiettinen
force-pushed
the
silverjam/numba-jit2
branch
from
76b960d to
1112801
Compare
pmiettinen
force-pushed
the
silverjam/numba-jit2
branch
from
3666963 to
60ee7f4
Compare
silverjam
changed the title
silverjam
commented
May 7, 2019
| } | ||
| """ | ||
|
|
||
| module_name = "parse_float_c" |
Contributor
Author
There was a problem hiding this comment.
Echoing comments on #667, it looks like we need to pick one of the options from here: https://cffi.readthedocs.io/en/latest/cdef.html and implement it so that we don't have to manually invoke this script prior to using sbp.jit.
| while len(data) < size: | ||
| d = self._read(size - len(data)) | ||
| if self._broken: | ||
| if not d or self._broken: |
Contributor
Author
There was a problem hiding this comment.
Should this be an optional behavior? Do we know this will affect the console and it's command line utilities?
Contributor
|
#667 currently contains exactly the same commits as this PR so in that sense this can be closed. |
Contributor
Author
|
Closing in favor of #667 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is paired with the following
piksi_toolsPR: swift-nav/piksi_tools#985Design notes
Numba based parsing for SBP classes. Adds various
numbaacceleratedget_*functions used to parse SBP data.Parsing floats
Floats are parsed using a module generated by another library,
cffi-- https://cffi.readthedocs.io/en/latest/ -- this is used to generate a Python native module that can parse and re-assemble float and double values from a stream of bytes. This was done becausenumbawas unable to generate code that could do the necessary bit banging.To generate this module, run the following:
This will generate
parse_float_c.so(andparse_float_c.c) which will be loadable by Python.Next steps
The next step from here would be to make the various
MsgWhatever.parse_membersmethods able to be JIT generated. This would require some more clever use ofTuple(( ... ))for the return values of the.parse_membermethod.Testing results
Without JIT’d
unpack/parse(with JIT’dcrcroutine though):With JIT’d
crc/unpack/parse:Haskell
sbp2jsonfor reference: