git-cola crash in openSUSE Tumbleweed

[guoyunhe]

git-cola 3.4

Operating System: openSUSE Tumbleweed 20190716
KDE Plasma Version: 5.16.2
KDE Frameworks Version: 5.59.0
Qt Version: 5.13.0
Kernel Version: 5.1.16-1-default
OS Type: 64-bit
Processors: 8 × Intel® Core™ i7-8550U CPU @ 1.80GHz
Memory: 31.2 GiB

> git-cola
Traceback (most recent call last):
  File "/usr/bin/git-cola", line 54, in <module>
    sys.exit(main())
  File "/usr/share/git-cola/lib/cola/main.py", line 27, in main
    return args.func(args)
  File "/usr/share/git-cola/lib/cola/main.py", line 299, in cmd_cola
    view = MainView(context, settings=args.settings)
  File "/usr/share/git-cola/lib/cola/widgets/main.py", line 93, in __init__
    N_('Actions'), self, widget=action.ActionButtons(context, self))
  File "/usr/share/git-cola/lib/cola/widgets/action.py", line 61, in __init__
    self.unstage_button = tooltip_button(N_('Unstage'), layout)
  File "/usr/share/git-cola/lib/cola/i18n.py", line 38, in N_
    return gettext(s)
  File "/usr/share/git-cola/lib/cola/i18n.py", line 22, in gettext
    if txt[-6:-4] == '@@':  # handle @@verb / @@noun
UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-1: ordinal not in range(128)

[guoyunhe]

My system language is Simplified Chinese and this error is an encoding error of gettext. So I guess it is problem of po file https://github.com/git-cola/git-cola/blob/master/po/zh_CN.po

[davvid]

hmm can you clone a copy of the repo down and edit cola/i18n.py at that line -- right before the if txt.... expression, please temporarily add these two lines to see what it reports:

print('txt: %s' % type(txt))
print('literal: %s' % type('@@'))

that can help hone in on a workaround. This is odd because that bit of code has been stable for a very long time, so there might be a subtle issue going on.

I'm assuming you're running python2. If it's possible to get a python3 PyQt5 installed then that might be an interesting experiment to see if it's well-behaved there.

A candidate solution is to add:

txt = core.decode(txt)

right before the line with the error and it should make the errors go away. The only reason that's weird is because, on python2, gettext.ugettext should already be returning a unicode string, and thus txt should not need to be decoded. The call is a no-op if the it's already unicode so it's safe to add so please let me know if that works.

It's surprising. Is your encoding not utf-8, btw? I'm assuming it is zh_CN.utf8 which should work fine.

[davvid]

I went ahead and optimistically added the decode() call -- let me know if that helps in your environment.

[guoyunhe]

My system has Python 2 as default.

Just tested with master:

python3 bin/git-cola # works
python2 bin/git-cola # crashes

[guoyunhe]

It is strange that I cannot find any description about ugettext() function on the internet. I guess it behaves differently in Python 2.

[davvid]

Yeah, back in the python2 days the regular gettext() method on the translation objects (the _translation variable in the i18n module) would return byte strings, so to get unicode strings a new method called ugettext was provided by the python stdlib.

In python3 that changed -- gettext() was changed to return unicode strings, and ugettext() was removed.

Our code tries to get python2 to behave roughly like python3, and we use unicode strings everywhere internally. When touching parts of the outside, we encode to utf-8 as needed, and use compat wrappers to paper over the 2 vs. 3 differences where one version might need encoding (or decoding), and the other doesn't.

The frozenset error is super curious -- it's behaving like it has a unicode string and wants to encode it, and the ascii encoder is not sufficient. The reason it's curious is because the arguments to the from_theme() call should all be regular ascii strings, so there should be no problem encoding to ascii internally by frozenset there. I don't think any translated strings pass through the memoize decorator, but I could be mistaken.

Anyways, let me know if you have any other clues. i18n and translations are a very important aspect so I appreciate the help.

[guoyunhe]

I searched 'utf-8' throughout the code base and found many files have specified 'utf-8' in some way:

cola/widgets/about.py:# encoding: utf-8
cola/compat.py:ENCODING = 'utf-8'
cola/core.py:ENCODING = 'utf-8'
cola/core.py:# Some files are not in UTF-8; some other aren't in any codification.
cola/core.py:    """encode(unencoded_string) returns a string encoded in utf-8
cola/core.py:    # default encoding (utf-8) when it gets unicode strings.
cola/qtutils.py:    e.g. terminator, prefer utf-8, so allow cola.dragencoding
cola/git.py:        :param _encoding: default encoding, defaults to None (utf-8).
extras/sphinxtogithub/sphinxtogithub.py:    app.add_config_value("sphinx_to_github_encoding", 'utf-8', '')
extras/sphinxtogithub/sphinxtogithub.py:        dest="encoding", default="utf-8",
extras/build_util.py:        result = string.encode('utf-8')
qtpy/QtDesigner.py:# -*- coding: utf-8 -*-
qtpy/_patch/qheaderview.py:# -*- coding: utf-8 -*-
qtpy/QtCore.py:# -*- coding: utf-8 -*-
qtpy/QtGui.py:# -*- coding: utf-8 -*-
qtpy/QtHelp.py:# -*- coding: utf-8 -*-
qtpy/QtLocation.py:# -*- coding: utf-8 -*-
qtpy/QtMultimediaWidgets.py:# -*- coding: utf-8 -*-
qtpy/QtNetwork.py:# -*- coding: utf-8 -*-
qtpy/QtOpenGL.py:# -*- coding: utf-8 -*-
qtpy/QtPrintSupport.py:# -*- coding: utf-8 -*-
qtpy/QtQml.py:# -*- coding: utf-8 -*-
qtpy/QtQuick.py:# -*- coding: utf-8 -*-
qtpy/QtQuickWidgets.py:# -*- coding: utf-8 -*-
qtpy/QtSql.py:# -*- coding: utf-8 -*-
qtpy/QtSvg.py:# -*- coding: utf-8 -*-
qtpy/QtTest.py:# -*- coding: utf-8 -*-
qtpy/QtWebChannel.py:# -*- coding: utf-8 -*-
qtpy/QtWebEngineWidgets.py:# -*- coding: utf-8 -*-
qtpy/QtWebSockets.py:# -*- coding: utf-8 -*-
qtpy/QtWidgets.py:# -*- coding: utf-8 -*-
qtpy/QtXmlPatterns.py:# -*- coding: utf-8 -*-
qtpy/QtCharts.py:# -*- coding: utf-8 -*-
qtpy/__init__.py:# -*- coding: utf-8 -*-
qtpy/compat.py:# -*- coding: utf-8 -*-
qtpy/py3compat.py:# -*- coding: utf-8 -*-
qtpy/py3compat.py:        return bytes(obj, 'utf-8' if encoding is None else encoding)
share/doc/git-cola/conf.py:# -*- coding: utf-8 -*-
share/doc/git-cola/conf.py:source_encoding = 'utf-8'
test/compat_test.py:# encoding: utf-8
test/core_test.py:# encoding: utf-8
test/core_test.py:        filename = helper.fixture('cyrillic-utf-8.txt')
test/core_test.py:        self.assertEqual(actual.encoding, 'utf-8')
test/i18n_test.py:        i18n.install('en_US.UTF-8')
test/i18n_test.py:        i18n.install('en_US.UTF-8')
test/i18n_test.py:        i18n.install('en_US.UTF-8')
setup.py:# distutils when installing from utf-8 paths.
setup.py:    sys.setdefaultencoding('utf-8')

Maybe this has something to do with the Unicode.