Refine `ustruct` into `struct` in shared-bindings and shared-module.

[tannewt]

Again, good for docs and structure.

[tannewt]

Two examples of commits that do a similar thing are: a0058e6
778e975

Most of the work is restructuring the code and verifying the API is a strict subset of CPython's struct. Do NOT copy documentation from CPython.

If you have any questions, please visit the #circuitpython channel on Discord.

[mrmcwethy]

I was just comparing Circuitpython ustruct with Python 3.5.2 struct and ran the follow tests. In Python 3.5.2, the @llll pack format and the lLlL format produce 8-byte integers while iLiL format produces 4-byte integers. Circuitpython produces 4-byte integer in all cases. I think that the native size of long on my Ubuntu machine must be 8-byte integers therefore this difference should be allowed.

Mike

struct.pack("@llll",1,2,3,4)
b'\x01\x00\x00\x00\x00\x00\x00\x00\x02\x00\x00\x00\x00\x00\x00\x00\x03\x00\x00\x00\x00\x00\x00\x00\x04\x00\x00\x00\x00\x00\x00\x00'
struct.pack("lLlL",1,2,3,4)
b'\x01\x00\x00\x00\x00\x00\x00\x00\x02\x00\x00\x00\x00\x00\x00\x00\x03\x00\x00\x00\x00\x00\x00\x00\x04\x00\x00\x00\x00\x00\x00\x00'
struct.pack("<lLlL",1,2,3,4)
b'\x01\x00\x00\x00\x02\x00\x00\x00\x03\x00\x00\x00\x04\x00\x00\x00'
struct.pack(">lLlL",1,2,3,4)
b'\x00\x00\x00\x01\x00\x00\x00\x02\x00\x00\x00\x03\x00\x00\x00\x04'

[tannewt]

How do the docs define what the result should be? Is it platform dependent?

[mrmcwethy]

The Python 3.6 docu (section 7.1.2.2.) says "When a format string that starts with @ character or no special char, the native size is the used. When using native size, the size of the packed value is platform-dependent.". If the format string starts with '<', '>', '=','!' the number of bytes is specified in a table (i.e. not platform specific).

[mrmcwethy]

I have another difference in behavior Python 3.5.2 struct and CircuitPython 2.0 ustruct. See below the same code is on both systems. I was being a little tricky and and set the offset to 1 instead of 0. The Python 3.5.2 case placed the short int right at the offset=1, while CircuitPython placed it at offset=2! I am not sure why, but I am guessing the CircuitPython is forcing packing alignment from the beginning of the buffer,while Python 3.5.2 does the alignment starting at offset=1

Python 3.5.2

b=bytearray(b'\xff\xff\xff\xff')
struct.pack_into('h',b,1,1)
b
bytearray(b'\xff\x01\x00\xff')

CircuitPython 2.x

b=bytearray(b'\xff\xff\xff\xff')
struct.pack_into('h',b,1,1)
b
bytearray(b'\xff\xff\x01\x00')

[dhalbert]

Is this relevant? micropython#3314 ("ustruct unpack_from with nonzero offset doesn't match CPython (esp8266)"). Our code is the same as MicroPython.

[mrmcwethy]

Good question. Is this why Micropython called the class ustruct or was it something else. The good news is that Micropython ustruct.packinto() and CPython struct.pack_into() do have the same signature. And they function very much the same. Also since the native formats are platform dependent, maybe this edge case can be different.

[tannewt]

@mrmcwethy Sounds like we'd just need to rename it.

I'm ok with us varying where CPython platforms also vary.

Thanks for looking into this!

[mrmcwethy]

I believe fixing this issue consists of the following tasks:

1. #define MICROPY_PY_STRUCT (0) for atmel-samd and esp8266 ports
2. implement struct module in shared-bindings and/or shared-module
3. repair the bundle library .py code that uses import ustruct
4. change docu where needed

[mrmcwethy]

what branch should i make my changes for this issue . I still plan on making my own branch, but I wondered what should be the source branch.
Note that i have made the changes to to shared-bindings and shared-module. The atmel side of things build with error, but the esp8266 has many unknown compile errors. I am assuming that i do not have the appropriate sources for esp8266 to build.