Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Already on GitHub? Sign in to your account

Writer converts everything to str #48

Open
dbrgn opened this Issue Mar 30, 2012 · 11 comments

Comments

Projects
None yet
10 participants

dbrgn commented Mar 30, 2012

The Writer class converts all strings to str. This causes Problems with unicode strings.

64     def __call__(self, s, newline=True, stream=STDOUT):
65 
66         if newline:
67             s = tsplit(s, NEWLINES)
68             s = map(str, s)
69             indent = ''.join(self.shared['indent_strings'])
70 
71             s = (str('\n' + indent)).join(s)
72 
73         _str = ''.join((
74             ''.join(self.shared['indent_strings']),
75             str(s),
76             '\n' if newline else ''
77         ))
78         stream(_str)

While I can print a unicode string

>>> print u'äöü'
äöü

...I cannot do that when using puts.

>>> puts(u'äöü')
Traceback (most recent call last):
  File "test.py", line 7, in <module>
    puts(u'äöü')
  File "/home/danilo/Projects/clint/clint/textui/core.py", line 83, in puts
    Writer()(s, newline, stream=stream)
  File "/home/danilo/Projects/clint/clint/textui/core.py", line 68, in __call__
    s = map(str, s)
UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-2: ordinal not in range(128)

This breaks puts-ing any non-ascii characters, which is a big problem for me.

This is probably the reason for #47.

myusuf3 commented Apr 24, 2012

👍

dbrgn commented Jun 5, 2012

Hey @kennethreitz, this is what I talked to you about this evening :) Would this still work if using unicode strings internally?

hdemers commented Jan 10, 2013

Any updates on this issue? I'm also having problems, specifically in formatters.py line 85 where the same map with str is done:

_row = map(str, _row)

@ghost ghost assigned jpiper Jan 9, 2014

@jpiper jpiper referenced this issue Jan 10, 2014

Closed

columns + unicode #47

Collaborator

jpiper commented Jan 10, 2014

This is a more widespread issue in clint with the handling of strings - particularly with regard to unicode strings. Some methods work on unicode strings and others fail . We need to go through with a fine toothcomb and try and work out a suitable way to fix it.

Contributor

gazpachoking commented Mar 1, 2014

This is a more widespread issue in clint with the handling of strings - particularly with regard to unicode strings. Some methods work on unicode strings and others fail . We need to go through with a fine toothcomb and try and work out a suitable way to fix it.

Yeah, I was just thinking about the best solution for this. Had several issues with implicit decoding using ascii, as well as other problems with explicit decoding using utf8 (which was the wrong encoding for the bytes I was giving.) I think the nicest solution would be to store all strings as strings internally, and only encode to bytes when outputting to the system. It would also be nice to be able to select a replace error handler, so that unicode strings can be outputted in consoles that do not support an encoding which can represent all the needed characters (windows, I'm looking at you.) I can take a crack at it, if we think that storing everything as strings internally is the right answer.

flexd commented May 23, 2014

Any progress on this? :-)

Running the unicode through colored works just fine (if that's an okay work around for anyone right now): puts(colored.red(u'äöü'))

lsaffre commented Jun 17, 2015

Yes, gazpachoking I agree that all strings should internally stored as unicode strings, and decoding should occur only when they are being output to a stream. It is a very annoying thing. Why is there no progress on this issue? Is there some workaround other than coloring the text?

Miserlou commented Jan 4, 2016

+1, this is a pretty shitty bug, ran into this while porting an app from Py2 to Py3. Blahh..

puts() should have safe unicode handling, and I suggest that there should also be a colored.default() that uses the default terminal font.

pevik commented Sep 16, 2016

@philadams puts(colored.red(u'äöü')) is still not working on python2 windows with non-unicode encoding :-(.

@pevik - it seems like clint is abandoned, I switched to using the 'click' project instead, which has this issue fixed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment