TryteString.as_bytes() does not work as expected #62
Comments
Hey @th0br0. This is actually expected behaviour — the tryte sequence >>> from iota.types import int_from_trits, TryteString
>>> int_from_trits(TryteString('ZZ').as_trits())
-28
>>> bytearray([-28])
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ValueError: byte must be in range(0, 256)
byte must be in range(0, 256) In order to convert this sequence into a byte string, you must instruct PyOTA to handle invalid sequences differently (this is very similar to how >>> from iota import TryteString
>>> TryteString(b'ABABZZDD99').as_bytes('strict')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Users/phx/Documents/pyota/iota/types.py", line 498, in as_bytes
return decode(self._trytes, 'trytes', errors)
File "/Users/phx/Documents/pyota/iota/codecs.py", line 165, in decode
'input': input,
iota.codecs.TrytesDecodeError: 'trytes' codec can't decode trytes ZZ at position 4-5: ordinal not in range(255)
'trytes' codec can't decode trytes ZZ at position 4-5: ordinal not in range(255)
>>> TryteString(b'ABABZZDD99').as_bytes('replace')
b'77?p\x00'
>>> TryteString(b'ABABZZDD99').as_bytes('ignore')
b'77p\x00' |
I beg to differ. I'm not well-versed enough in Python3 codecs to pinpoint exactly where things need to change, but: value = 0
RADIX = 3
chunk = [-1,1,1,1,-1]
for c in chunk:
value = value * RADIX + c the resulting value is signed ( Decomposition can happen either via a lookup table computed at library load, e.g. https://github.com/iotaledger/iota.lib.java/blob/master/src/main/java/jota/utils/Converter.java#L37, Note that you need to store the number of encoded trits for decomposition of multiple bytes to work. This is what both the Java & Rust libraries do, I don't think that the JS library currently implements byte conversion. The current Python library can't even do RADIX = 3
MAX_TRIT_VALUE = 1
MIN_TRIT_VALUE = -1
def increment(trits):
for i in range(5):
trits[i] += 1
if trits[i] > MAX_TRIT_VALUE:
trits[i] = MIN_TRIT_VALUE
else:
break
return trits
if __name__ == '__main__':
trits = [0] * 5
for i in range(243):
value = 0
# to_bytes
for t in trits[::-1]:
value = value * RADIX + t
print("%s=%d"% (trits, value & 0xFF))
# to_trits
# there's 243 different value combinations
# this is just easier than having to worry about signedness when converting back...
value += 121
chunk = []
for _ in range(5):
rem = value % RADIX
chunk.append(int(rem))
value = (value - rem) / RADIX
chunk = [x - 1 for x in chunk]
assert(trits == chunk)
trits = increment(trits) |
Thanks for the thorough explanation! I see now what I was missing before.
I like the idea of implementing a proper trytes -> bytes conversion process (and I'm excited to finally see some maths showing how it works!); maybe we can rename Given everything that's going on with PyOTA right now, I think this will have to wait until v2.1 (v2.0 has the new hash algorithm and multisig — there's already enough backwards-incompatible changes for one release!). In the meantime, I think you can get the result you're looking for by first calling |
Hey @th0br0 I'm starting to work on this now. Would you be willing to help me come up with some test cases for this functionality? There's a few scenarios that I'm interested in:
@alon-e @paulhandy Would love to get your thoughts on this as well. What are some unit tests we can use for trits <-> bytes conversion? |
The previous name will still work, but it is deprecated and will be removed in PyOTA v2.1.
Hi all, I'm currently working on Python reading rocksdb data, and it is relative to this topic. ProblemI think
the name
In Python, For examle, For How to fixed it
TryteString internal representation as
|
Hey @mlouielu thanks for the info and the suggestions! I see what you're getting at with Let's figure out the trits <-> bytes conversion first, and then we can move on to trytes <-> characters. The most important thing we need right now to fix this issue is test cases. Could you help with #62 (comment) ? |
As for issue iotaledger#62, this is the test cases for converting TryteString to bytes, and bytes to TryteString.
Thanks @mlouielu for the test cases!! I will look at them this weekend, and I'll let you know if I have any questions. |
Hi @todofixthis Your latest commit regarding trytes was naming only. Is that correct? Or did you change some of the behaviour as well? There is a lot of concurrent work on the byte <-> tryte conversion. Both in #62 and #90 by you and @th0br0, and @mlouielu in the rocksdb. I work on encrypting IOTA seeds (somewhat similar to BIP38), and @dashengsun (with feedback from @paulhand) work on Unicode to trytes in the iota.lib.js and @pRizz has published pRizz/Tryte-UTF8-JSON-Codec, @tuupola has tuupola/trytes (for PHP) and @knarz has knarz/trytes (for Rust). Maybe we should coordinate a little, share the good ideas, and pin point difficulties before they get published and confuse the rest of the community? Because tryte <-> byte conversion is not trivial. Square pegs and round holes We probably should not pollute issue #62 with this discussion. Please suggest a better place. If we lack better options, I just now created vbakke/trytes where we can discuss this freely. |
Hey @vbakke About 4 months ago, I started work on this issue. I thought we'd have it wrapped up in a couple of weeks, so I made some changes to the codebase to make room for the new codec and mark the existing codec as deprecated. However, at this point, I'm still not sure that we have a bytes <-> trytes conversion process that can handle all edge cases (e.g., bytes in range |
Hi @todofixthis , I've been spending most of my spare time during xmas looking at this as well. I just now published my latest thoughts on the issue, as well as working example code (JavaScript, though). I think it covers all corner cases, and includes Unicode text to tryte strings, including a BOM mark to indicate the encoding, if used. Feel free to have a look at my latest write up of the Readme.md. I don't think there is a universal two-way byte <->tryte conversion. |
Thanks @vbakke! I'll have a look at your notes later this week. Quick glance shows that they are very thorough; looking forward to going through them! |
FYI I just added an online version of encoding native strings, and native trytes at https://vbakke.github.io/trytes/ |
The previous name will still work, but it is deprecated and will be removed in PyOTA v2.1.
Expected result (or similar):
Actual result:
The text was updated successfully, but these errors were encountered: