Add Info Regarding Byte Alignment or Packing Mismatches #847
Comments
|
Yeah this issue has come up more and more often. I even made a blog post about it: https://tmrh20.blogspot.com/2020/09/transferring-data-between-systems-using.html?m=1 I agree it should be in common issues but not sure what to add to the examples yet.
… On Jun 28, 2022, at 2:37 PM, Jeffrey Rocchio ***@***.***> wrote:
I have discovered, the hard way, that the ATTiny84/Arduino (tiny84) platform and the Raspberry Pi (RPi) don't pack and unpack their data structures quite the same. What this means is that data sent out from one side may not be correct after receipt and loading into a matching C++ struct on the other side. To many folks this is probably a thing that is so obvious it doesn't bear mentioning. For some of us tho it is a surprise.
My Recommendation:
Add a topic to the 'Common Issues' doc that points this out so that less experienced users will be alerted to this.
Also, in the demo code add a comment that points folks to the topic in Common Issues. For example, in manualAcknowledgements.cpp add a comment where struct PayloadStruct is defined that points this out.
Rationale for Adding Comments in the Demo Code:
After I had gettingStarted working just fine between the tiny84 and the RPi I moved on to manualAcknowledgements. I got that working just fine. And I noted that on both ends both programs used a matching struct to send and receive the dummy data of that demo - a string and one uint8_t. My my use case I needed to send a few long int and two floats. So I added those to both the send and receive side .cpp files. But after doing this, on the RPi, receive, side, I was getting widly incorrect values for my added variables. But they were consistently wrong, so I figured it wasn't bad wiring, radio interference, or the like. I did figure it had to be how the RPi was treating the incoming bytes. But since the examples used matching 'structs', and on the receive side just passed the struct directly to the radio object, as in: PayloadStruct received; radio.read(&received, sizeof(received)); I assumed that this was the proper way to receive in the data. So that then sent me off on all sorts of theories that chewed up a lot of time (like, e.g, possible endian differences). If there had been a warning comment next to the 'struct' definition in the .cpp code then I would have known immediately what was going on.
FYI, My Solution:
In my search I did see any number of ideas concerning getting data cleanly across a network layer between different platforms, beit endian variance or, as I encountered here, byte boundary alignment and/or packing/unpacking. In order to debug the problem I leveraged the richness of the RPi platform to be able to display a lot of stuff on my SSH terminal window. What solved it for me was working out how to get and display the raw byte values in HEX so that I could see, step-by-step, what was going on. That was a whole C++ learning lab just there. But having done that it then became relatively trivial to just receive the incoming data on the RPi side into a uint8_t[], and then write a function that "manually" loads the matching transmit struct from the raw bytes. As in: pStruct->testFloat1 = *(float *)&pBytes[offset]; Of course, if you had an endian mismatch you'd have to do some additional work. And being a novice here, somebody may tell me this is a bad idea for reasons I don't yet understand.
Attached Files For Reference:
SSH terminal output that first showed the problem - CapDataReceive_02b_Output.txt
RPi Side Code For Above - CapDataReceive_02b.cpp
SSH terminal output with issue fixed - CapDataReceive_02c_Output.txt
RPi Side Code for Above - CapDataReceive_02c.cpp
CapDataReceive_02b_Output.txt
CapDataReceive_02c_Output.txt
CapDataReceive_02c_cpp.pdf
CapDataReceive_02b_cpp.pdf
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you are subscribed to this thread.
|
|
What you're fighting is called memory misalignment. It can be solved by making sure data fits into 32-bit wide boundaries. For example, your test data sends an 11-byte string in the first segment of the payload, but it would be better if this was the last segment. If you look at the raw hex data, you may have noticed that everything after the first c-string is offset by 1 byte. I love the idea of adding this to the docs and examples' comments. But, it must be done in a way that beginners can understand. Otherwise, they're likely to end up "wasting" precious development time. I'll ask for your review in the PR that aims to solve this request. For what it's worth, it sounds like your time debugging this wasn't a complete loss: You learned a lot. Python is great for debugging because it is such a high level language that you don't have to pay with pointers (arrays), and you can play with memory contents in realtime (using an interactive REPL - aka "python prompt"). |
|
Using same-size data types with A better choice, if you have the resources, would be to use CBOR or some similar format that has a well-defined way of encoding and decoding. |
I have run across a reference to this a few times over the past few days. Just did a quick look at it and can see that it is a good answer to this general problem - tho not sure as yet of the added complexity and resource usage. I also see there are several Arduino libraries, some of which are forks of Intel's TinyCBOR. Is there an obvious one to use for the ATTiny84/Arduino do you think? |
True. And this is the pleasure of hobby projects - no deadlines to get in the way of going down rabbit holes for the fun of it. lol
Python is everywhere it seems. It appears inevitable I'll take it up at some point. I have worked in probably a dozen different languages over my career (work and hobbies) so I know I am pretty good at picking up new languages. For the moment tho I am enjoying this revisit back to C, one of my first languages. And the 'modern' C++ is interesting as I have done OO in other languages; tho to be sure I haven't really exercised the OO capabilities of C++ in this project. |
The ATTiny's are probably too memory constrained for that, a "packed" struct would be cheaper. |
Gotcha; and thanks for the tip. |
* Docs: Add info on byte alignment/packing mismatches #847 --------- Co-authored-by: Brendan <2bndy5@gmail.com>
|
Resolved in #945 |
I have discovered, the hard way, that the ATTiny84/Arduino (tiny84) platform and the Raspberry Pi (RPi) don't pack and unpack their data structures quite the same. What this means is that data sent out from one side may not be correct after receipt and loading into a matching C++ struct on the other side. To many folks this is probably a thing that is so obvious it doesn't bear mentioning. For some of us tho it is a surprise.
My Recommendation:
struct PayloadStructis defined that points this out.Rationale for Adding Comments in the Demo Code:
After I had
gettingStartedworking just fine between the tiny84 and the RPi I moved on tomanualAcknowledgements. I got that working just fine. And I noted that on both ends both programs used a matching struct to send and receive the dummy data of that demo - a string and oneuint8_t. My my use case I needed to send a few long int and two floats. So I added those to both the send and receive side .cpp files. But after doing this, on the RPi, receive, side, I was getting widly incorrect values for my added variables. But they were consistently wrong, so I figured it wasn't bad wiring, radio interference, or the like. I did figure it had to be how the RPi was treating the incoming bytes. But since the examples used matching 'structs', and on the receive side just passed the struct directly to theradioobject, as in:PayloadStruct received; radio.read(&received, sizeof(received));I assumed that this was the proper way to receive in the data. So that then sent me off on all sorts of theories that chewed up a lot of time (like, e.g, possible endian differences). If there had been a warning comment next to the 'struct' definition in the .cpp code then I would have known immediately what was going on.FYI, My Solution:
In my search I did see any number of ideas concerning getting data cleanly across a network layer between different platforms, beit endian variance or, as I encountered here, byte boundary alignment and/or packing/unpacking. In order to debug the problem I leveraged the richness of the RPi platform to be able to display a lot of stuff on my SSH terminal window. What solved it for me was working out how to get and display the raw byte values in HEX so that I could see, step-by-step, what was going on. That was a whole C++ learning lab just there. But having done that it then became relatively trivial to just receive the incoming data on the RPi side into a
uint8_t[], and then write a function that "manually" loads the matching transmit struct from the raw bytes. As in:pStruct->testFloat1 = *(float *)&pBytes[offset];Of course, if you had an endian mismatch you'd have to do some additional work. And being a novice here, somebody may tell me this is a bad idea for reasons I don't yet understand.Attached Files For Reference:
SSH terminal output that first showed the problem - CapDataReceive_02b_Output.txt
RPi Side Code For Above - CapDataReceive_02b.cpp
SSH terminal output with issue fixed - CapDataReceive_02c_Output.txt
RPi Side Code for Above - CapDataReceive_02c.cpp
CapDataReceive_02b_Output.txt
CapDataReceive_02c_Output.txt
CapDataReceive_02c_cpp.pdf
CapDataReceive_02b_cpp.pdf
The text was updated successfully, but these errors were encountered: