Encoding error on Pico-8 glyphs (or possibly: AttributeError: 'bytes' object has no attribute 'hex') #9

Closed
dagondev opened this Issue Dec 25, 2016 · 5 comments

Comments

Projects
None yet
2 participants
@dagondev

Wrote this at bbs, but not sure if you are looking there anymore. (http://www.lexaloffle.com/bbs/?pid=34133#p34133)

Dear dddaaannn.

I am trying to use your picotool to minify my game. Currently I have:
65302/65536 char count and 8136/8192 token count.
When I am trying to use luamin on my cart I get:

C:\Python34>python p8tool luamin saveyourself.p8
saveyourself.p8 -> saveyourself_fmt.p8
warning: token count 29022 exceeds the Pico-8 limit of 32768Traceback (most rece
nt call last):
  File "p8tool", line 8, in <module>
    sys.exit(tool.main(sys.argv[1:]))
  File "C:\Python34\pico8\tool.py", line 512, in main
    return args.func(args)
  File "C:\Python34\pico8\tool.py", line 349, in do_luamin
    return process_game_files(args.filename, luamin, args=args)
  File "C:\Python34\pico8\tool.py", line 210, in process_game_files
    procfunc(g, out_fname, args=args)
  File "C:\Python34\pico8\tool.py", line 242, in luamin
    lua_writer_cls=lua.LuaMinifyTokenWriter)
  File "C:\Python34\pico8\game\game.py", line 746, in to_file
    self.to_p8_file(outfh, *args, **kwargs)
  File "C:\Python34\pico8\game\game.py", line 659, in to_p8_file
    for l in self.gfx.to_lines():
  File "C:\Python34\pico8\gfx\gfx.py", line 95, in to_lines
    yield bytes(newdata).hex() + '\n'
AttributeError: 'bytes' object has no attribute 'hex'

I have ton of text packed into loooong strings if that helps.
Please help me, as I can't even export my game to html at this point.

Thanks in advance!

@dagondev

This comment has been minimized.

Show comment
Hide comment
@dagondev

dagondev Dec 26, 2016

In Python 3.3 and 3.4 I get error as specified above and it is related to the fact, in P3.3 way of obtaining hex changed: http://stackoverflow.com/questions/12917278/how-to-use-the-hex-encoding-in-python-3-2-or-higher

In Python 3.2 I get:
c:\Python32>python.exe p8tool luamin saveyourself.p8 saveyourself.p8 -> saveyourself_fmt.p8 Traceback (most recent call last): File "p8tool", line 8, in <module> sys.exit(tool.main(sys.argv[1:])) File "c:\Python32\pico8\tool.py", line 512, in main return args.func(args) File "c:\Python32\pico8\tool.py", line 349, in do_luamin return process_game_files(args.filename, luamin, args=args) File "c:\Python32\pico8\tool.py", line 210, in process_game_files procfunc(g, out_fname, args=args) File "c:\Python32\pico8\tool.py", line 242, in luamin lua_writer_cls=lua.LuaMinifyTokenWriter) File "c:\Python32\pico8\game\game.py", line 746, in to_file self.to_p8_file(outfh, *args, **kwargs) File "c:\Python32\pico8\game\game.py", line 634, in to_p8_file if transformed_lua.get_char_count() > PICO8_LUA_CHAR_LIMIT: File "c:\Python32\pico8\lua\lua.py", line 55, in get_char_count return sum(len(l) for l in self.to_lines()) File "c:\Python32\pico8\lua\lua.py", line 55, in <genexpr> return sum(len(l) for l in self.to_lines()) File "c:\Python32\pico8\lua\lua.py", line 153, in to_lines for line in writer.to_lines(): File "c:\Python32\pico8\lua\lua.py", line 287, in to_lines strs.clear() AttributeError: 'list' object has no attribute 'clear'
Clear was added in P3.3...

In P3.1 there isn't argparse module, so it is completely broken.

In Python 3.3 and 3.4 I get error as specified above and it is related to the fact, in P3.3 way of obtaining hex changed: http://stackoverflow.com/questions/12917278/how-to-use-the-hex-encoding-in-python-3-2-or-higher

In Python 3.2 I get:
c:\Python32>python.exe p8tool luamin saveyourself.p8 saveyourself.p8 -> saveyourself_fmt.p8 Traceback (most recent call last): File "p8tool", line 8, in <module> sys.exit(tool.main(sys.argv[1:])) File "c:\Python32\pico8\tool.py", line 512, in main return args.func(args) File "c:\Python32\pico8\tool.py", line 349, in do_luamin return process_game_files(args.filename, luamin, args=args) File "c:\Python32\pico8\tool.py", line 210, in process_game_files procfunc(g, out_fname, args=args) File "c:\Python32\pico8\tool.py", line 242, in luamin lua_writer_cls=lua.LuaMinifyTokenWriter) File "c:\Python32\pico8\game\game.py", line 746, in to_file self.to_p8_file(outfh, *args, **kwargs) File "c:\Python32\pico8\game\game.py", line 634, in to_p8_file if transformed_lua.get_char_count() > PICO8_LUA_CHAR_LIMIT: File "c:\Python32\pico8\lua\lua.py", line 55, in get_char_count return sum(len(l) for l in self.to_lines()) File "c:\Python32\pico8\lua\lua.py", line 55, in <genexpr> return sum(len(l) for l in self.to_lines()) File "c:\Python32\pico8\lua\lua.py", line 153, in to_lines for line in writer.to_lines(): File "c:\Python32\pico8\lua\lua.py", line 287, in to_lines strs.clear() AttributeError: 'list' object has no attribute 'clear'
Clear was added in P3.3...

In P3.1 there isn't argparse module, so it is completely broken.

@dansanderson

This comment has been minimized.

Show comment
Hide comment
@dansanderson

dansanderson Dec 27, 2016

Owner

Thanks for the report! I'm currently unable to reproduce the error with your cart because picotool doesn't yet support Pico-8 glyphs, which Pico-8 encodes as arbitrary high bytes. picotool tries to read the code region as utf-8 because I wanted the code APIs to use text strings and there were a few older carts created with external editors that had utf-8 high char sequences in comments (which Pico-8 ignores). Pico-8 doesn't actually use utf-8 in general, so this is incorrect. The correct solution is probably for picotool to deal entirely with bytestrings with regard to the Lua code region. This will be a significant upgrade.

For what it's worth, I'd expect picotool to work in Python 3.4+. Based on what other Python 3 projects appear to be doing, it looks like it'd be unrealistic to try and support older versions. I'm willing to take on more robust support for older versions if that's a common practice, though I'd imagine we'd need to run all tests in all supported versions of Python. b'...'.hex() is supported by Python 3.6, so I'm not sure yet if there's a fix needed there specifically or if the error is somewhere else.

screen shot 2016-12-27 at 2 14 09 pm

Owner

dansanderson commented Dec 27, 2016

Thanks for the report! I'm currently unable to reproduce the error with your cart because picotool doesn't yet support Pico-8 glyphs, which Pico-8 encodes as arbitrary high bytes. picotool tries to read the code region as utf-8 because I wanted the code APIs to use text strings and there were a few older carts created with external editors that had utf-8 high char sequences in comments (which Pico-8 ignores). Pico-8 doesn't actually use utf-8 in general, so this is incorrect. The correct solution is probably for picotool to deal entirely with bytestrings with regard to the Lua code region. This will be a significant upgrade.

For what it's worth, I'd expect picotool to work in Python 3.4+. Based on what other Python 3 projects appear to be doing, it looks like it'd be unrealistic to try and support older versions. I'm willing to take on more robust support for older versions if that's a common practice, though I'd imagine we'd need to run all tests in all supported versions of Python. b'...'.hex() is supported by Python 3.6, so I'm not sure yet if there's a fix needed there specifically or if the error is somewhere else.

screen shot 2016-12-27 at 2 14 09 pm

@dansanderson

This comment has been minimized.

Show comment
Hide comment
@dansanderson

dansanderson Dec 27, 2016

Owner

(I also need to add support for the __label__ region, which is new in 0.10. :) )

Owner

dansanderson commented Dec 27, 2016

(I also need to add support for the __label__ region, which is new in 0.10. :) )

@dansanderson dansanderson changed the title from AttributeError: 'bytes' object has no attribute 'hex' to Encoding error on Pico-8 glyphs (or possibly: AttributeError: 'bytes' object has no attribute 'hex') Dec 27, 2016

@dagondev

This comment has been minimized.

Show comment
Hide comment
@dagondev

dagondev Dec 29, 2016

I have no problem with using newest Python, I am just assumed proper version is 3.3 as it was coresponding with time you released repo (as you didn't provide which subversion use).

Regardless my error: I forgot to state this, but while obviously my cart isn't parsable if you take it straight from pico8 as your tool tells about wrong header - converting it to the UTF8 (without BOM if I remember correctly) makes it available, as I assumed it was necessary. Weird thing was that converting it to the UTF-8 with BOM worked on Notepad++, but on Sublime Text 3 I needed to convert without BOM o.o or was it other way around - I don't remember, sorry!

I have no problem with using newest Python, I am just assumed proper version is 3.3 as it was coresponding with time you released repo (as you didn't provide which subversion use).

Regardless my error: I forgot to state this, but while obviously my cart isn't parsable if you take it straight from pico8 as your tool tells about wrong header - converting it to the UTF8 (without BOM if I remember correctly) makes it available, as I assumed it was necessary. Weird thing was that converting it to the UTF-8 with BOM worked on Notepad++, but on Sublime Text 3 I needed to convert without BOM o.o or was it other way around - I don't remember, sorry!

@juanitogan juanitogan referenced this issue Feb 21, 2017

Closed

Python34 #12

@dansanderson

This comment has been minimized.

Show comment
Hide comment
@dansanderson

dansanderson Feb 22, 2017

Owner

I have converted all of picotool to treat text in Pico-8 carts as bytestrings instead of text strings to account for Pico-8's custom "encoding" (which is just lower ASCII with custom high characters for glyphs). I probably broke it all to hell but the tests pass. :)

picotool also now has label support. In theory, it should now be able to process the cart in the original bug report.

Owner

dansanderson commented Feb 22, 2017

I have converted all of picotool to treat text in Pico-8 carts as bytestrings instead of text strings to account for Pico-8's custom "encoding" (which is just lower ASCII with custom high characters for glyphs). I probably broke it all to hell but the tests pass. :)

picotool also now has label support. In theory, it should now be able to process the cart in the original bug report.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment