Join GitHub today
GitHub is home to over 40 million developers working together to host and review code, manage projects, and build software together.
Sign upWhat would you like to be added to Construct? #824
Comments
This comment has been minimized.
This comment has been minimized.
|
@yannayl |
This comment has been minimized.
This comment has been minimized.
|
I'm still looking for a way to differentiate exceptions raised by I have multiple |
This comment has been minimized.
This comment has been minimized.
|
I dont think this is really necessary to be available to everyone. But the solution is simple, just copy paste the source of Checksum into your script and change the constructor to take an exception type. Simple. class Checksum(Construct):
def __init__(self, checksumfield, hashfunc, bytesfunc, exception):
super(Checksum, self).__init__()
self.checksumfield = checksumfield
self.hashfunc = hashfunc
self.bytesfunc = bytesfunc
self.flagbuildnone = True
self.exception = exception
def _parse(self, stream, context, path):
hash1 = self.checksumfield._parsereport(stream, context, path)
hash2 = self.hashfunc(self.bytesfunc(context))
if hash1 != hash2:
raise self.exception("wrong checksum, read %r, computed %r" % (
hash1 if not isinstance(hash1,bytestringtype) else binascii.hexlify(hash1),
hash2 if not isinstance(hash2,bytestringtype) else binascii.hexlify(hash2), ))
return hash1
def _build(self, obj, stream, context, path):
hash2 = self.hashfunc(self.bytesfunc(context))
self.checksumfield._build(hash2, stream, context, path)
return hash2
def _sizeof(self, context, path):
return self.checksumfield._sizeof(context, path) |
This comment has been minimized.
This comment has been minimized.
|
This approach was kinda already written in the docs, just not so directly. See for yourself: |
This comment has been minimized.
This comment has been minimized.
|
I just found your old Issue that you had for this: #731 |
This comment has been minimized.
This comment has been minimized.
|
I guess we can also post here #826 which is showing path in exceptin messages. Already done. |
This comment has been minimized.
This comment has been minimized.
|
MESSAGE TO EVERYONE: I am about to remove Embedded and EmbeddedSwitch from the core library. I know its in use here and there but please understand, the semantics have changed so much that using this as is now is pointless. |
This comment has been minimized.
This comment has been minimized.
|
@arekbulski Some code examples to migrate Embedded and EmbeddedSwitch to 2.10 would be useful. |
This comment has been minimized.
This comment has been minimized.
|
The alternative to embedding is simply nesting. Nesting is already covered in the docs through and through. Nesting is the recommended way of doing things anyway. |
This comment has been minimized.
This comment has been minimized.
|
I updated the docs accordingly: |
This comment has been minimized.
This comment has been minimized.
|
@arekbulski GreedyByte is no longer able to build from bytearray in python 3.
|
This comment has been minimized.
This comment has been minimized.
|
|
This comment has been minimized.
This comment has been minimized.
|
Sorry for being late to the party. I moved to different projects, I still use construct regularly but I am very satisfied with the current feature set. Thanks, btw :) |
This comment has been minimized.
This comment has been minimized.
|
TBH I am also very satisfied with the current feature set. :) Aside of 1-2 open tickets to be fixed in foreseeable time, there is little for me to do here. Makes me feel kinda useless. |
This comment has been minimized.
This comment has been minimized.
|
There is always a place for a talented programmer. |
This comment has been minimized.
This comment has been minimized.
|
Do you mean adding something like build but in a way that all values are random and not provided to the build function? I dont think I will be adding something like that. |
This comment has been minimized.
This comment has been minimized.
|
Yes, something along these lines more or less. In a fuzzing course I took 3 years ago in BlackHat they gave us a library for fuzzing that was heavily inspired by construct. So there was a need back then. Not sure if there is now. |
This comment has been minimized.
This comment has been minimized.
|
It would be very helpful if there were examples of using construct to parse and manipulate complex yet common protocols such as IPv4, ICMP, TCP, etc... |
This comment has been minimized.
This comment has been minimized.
|
The deprecated_gallery code is there for you to bring it up to 2.10... |
This comment has been minimized.
This comment has been minimized.
|
@yannayl I was looking over FuzzIon source and found this naughty boy: def LEBitsInteger(n):
return ct.ByteSwapped(ct.BitsInteger(n)) |
This comment has been minimized.
This comment has been minimized.
|
@guywhataguy Sorry for being condescending. Sometimes I just get too much of it, people coming and just demanding code. The gallery is there, but someone familiar with the actual data formats should rewrite the code for 2.10. Mostly removing embeddings and such. Networking packets are tricky, especially DNS packets. I never got that right. |
This comment has been minimized.
This comment has been minimized.
|
@arekbulski haha yes, you remember we talked about it? |
This comment has been minimized.
This comment has been minimized.
|
Yes, the insane idea of applying little endianness to non-multiples of 8 bits. Jeez luize. |
This comment has been minimized.
This comment has been minimized.
|
Bitwise endianness is a thing 🤷 |
This comment has been minimized.
This comment has been minimized.
|
@arekbulski as much as I feel for GHidra users, I see no reason to support python 2 any further. Of course it requires updating a major so nobody cries too much. |
This comment has been minimized.
This comment has been minimized.
|
Yes, if you want Python 2 support then use 2.9.*. |
This comment has been minimized.
This comment has been minimized.
|
Now I understand why you were hinting at type annotations. Yeah, python 3.5 or 3.6 support those. The thing is, I do not like those. It adds yet-another-one syntax and complicates things beyond need. |
This comment has been minimized.
This comment has been minimized.
|
AFAIK, Python 2.7.18 will be released on April 2020, and it was claimed that it would be the last Python 2.7.x version. Nevertheless, a lot of packages are dropping support for Python 2 these days. |
This comment has been minimized.
This comment has been minimized.
|
I think I will also change the Container class to derive from OrderedDict and not implement all of this stuff from scratch. Should make less lines of code nicely. |
This comment has been minimized.
This comment has been minimized.
|
Yes, type hinting complicates things a little but it plays really nicely with static analysis tools which are very helpful (e.g. IDE suggestions). Ghidra is the NSA's reverse engineering tool that became open source last year. Its written in Java and can be scripted with Jython. That's the only legit use of python2 I'm aware of. |
This comment has been minimized.
This comment has been minimized.
I've found this way: ExprAdapter(
RawCopy(Prefixed(
lengthfield=Int16ub,
subcon=Union(
0,
"integer" / Int32sb,
"bytes" / Bytes(4),
),
includelength=True,
)),
lambda obj, ctx: Container(size=obj.length, **obj.value),
lambda obj, ctx: {"value": obj},
)Indeed, that puts the size of the object in the container on parsing, and that should work even when the Anyway, I still think this behavior can be included as a |
This comment has been minimized.
This comment has been minimized.
|
Its too custom made to be included in the mainstream version of the library. Copy paste Prefixed and modify it to your needs. |
This comment has been minimized.
This comment has been minimized.
|
Its official, I just dropped 2.7 and 3.5 from the supported runtimes. |
This comment has been minimized.
This comment has been minimized.
|
Readthedocs just broken. Assisstance required: |
This comment has been minimized.
This comment has been minimized.
|
@yannayl @arekbulski from construct import *
from typing import Type
def attruct(cls: Type):
return Struct(**cls.__annotations__)
@attruct
class Format:
signature: Const(b"BMP")
width: Int8ub
height: Int8ub
pixels: Array(this.width * this.height, Byte)
@attruct
class OtherFormat:
@attruct
class SomeFmt:
signature: Const(b"BMP")
width: Int8ub
height: Int8ub
pixels: Array(this.width * this.height, Byte)
format: SomeFmtand PyCharm seems to like them too. Figured some actual examples may help the discussion. |
This comment has been minimized.
This comment has been minimized.
|
This leaves me on the fence. On one hand, I like the examples, on the other hand, there is already way too many ways of writing structs. That includes + operator, Struct(a=Byte), Struct("a"/Byte). I suggest to everyone to use the last syntax. |
This comment has been minimized.
This comment has been minimized.
|
Those annotations already have a ticket: #814 (comment) |
This comment has been minimized.
This comment has been minimized.
|
Given how little code is required to make it work, I'm happy as long as we keep the |
This comment has been minimized.
This comment has been minimized.
|
Fine, I was thinking of removing it but sure okay. |
This comment has been minimized.
This comment has been minimized.
|
Well, my decorator from the previous example could easily be replaced with: def attruct(cls: Type):
return Struct(*(key / value for key, value in cls.__annotations__.items()))So it doesn't really matter. Though I personally prefer the |
This comment has been minimized.
This comment has been minimized.
|
I think I will switch to the |
This comment has been minimized.
This comment has been minimized.
|
I get it guys. Syntax stays. |
This comment has been minimized.
This comment has been minimized.
|
@arekbulski it seems like the RTD build is still using Python2.7. |
This comment has been minimized.
This comment has been minimized.
|
Could you PR it? I dont know how to write it myself. |
This comment has been minimized.
This comment has been minimized.
|
There is this already: https://github.com/construct/construct/blob/master/docs/conf.py |
This comment has been minimized.
This comment has been minimized.
|
Created #835 |
This comment has been minimized.
This comment has been minimized.
|
IT WORKS!!! IT WORKS!!!! I owe you a beer, man. |
This comment has been minimized.
This comment has been minimized.
|
I guess it's high-time to drop all the extra Python2 code. |
This comment has been minimized.
This comment has been minimized.
|
@arekbulski would you accept PRs that remove legacy code / simplify internals? (Not at all sure that I'll submit any, but I might) |
This comment has been minimized.
This comment has been minimized.
|
Sure, I would consider such PRs. You can also tell me in advance what you want to remove so I can tell you yay or nay on the spot. |
This comment has been minimized.
This comment has been minimized.
|
I also like when people submit new test cases, to extend the test suite. Thats something worthy. |
This comment has been minimized.
This comment has been minimized.
|
I am taking a short leave since there is little for me to do. |
This comment has been minimized.
This comment has been minimized.
|
@jpsnyder You are welcome to share with us, your ideas for changes. |
This comment has been minimized.
This comment has been minimized.
|
@yannayl You gonna enjoy watching this... |
@DanseMacabre
I am opening this ticket for a chat...