Add python 3 support

[ghost]

One day, like that, I decide I'll port one of the tools on my system to Python 3, so I did.

The result is at here. I would have pushed it to gitorious, but the website has just been bought by gitlab and it seemed like a bad place to push.

All tests except test_IT_keep_entities_1 pass on both python 2.7 and python 3.3. The test_IT_keep_entities_1 was already failing before my changes, I couldn't figure why.

I've been very conservative with my changes and I didn't go beyond the strict necessary to make things work. As with most porting, the trickiest change is the unicode/str/bytes thing. I went with unicode-everything.

At two places, you'll see a if not PY3: encode to utf8, that's because strangely, libxml requires bytes when under python2 and refuses bytes when under python3...

Does this look like something you'd be willing to merge?

[shaunix]

Definitely interested in this, thanks. And this is the right place to push. I'm in the process of moving itstool from gitorious to github. Website's not updated yet. Does libxml2 work with Python 3 now?

[ghost]

Yes, libxml2 runs on python 3. I ran itstool tests with both Python 2.7 and 3.3. Apparently, support for python 3 was added 2 years ago

[Keruspe]

Any updates about this?

[jjardon]

This is interesting as itstool is one of the few modules that still depend on python2 in GNOME

Maybe other option is to use python-lxml instead the python-libxml2: http://lxml.de/ ?

[Keruspe]

I don't think libxml2 is the problem here, the python bindings of libxml2 work just fine for python 3.4 and 3.5 at least

[jjardon]

My system doesnt have python2 at all, so I built libxml2 against python3; itstool fails to build because a missing libxml2 python module

[ghost]

@jjardon oh, lucky you! I'm trying to get there as well, hence this PR.

I'm pretty sure that libxml2 supports python3 because with this PR, I could run itstool under py3 (I'm on Gentoo). Maybe it's only that python3 bindings of libxml2 is not available on your system? Then it would be a distro problem.

[juergbi]

The configure check in itstool is broken, it assumes the 'python' binary is what you want to use, instead of $PYTHON. The 'python' binary/symlink typically points to python2 and doesn't exist on pure python3 systems. Fixing this doesn't appear to be sufficient, though. itstool fails here trying to build evolution.

Traceback (most recent call last):
File "/usr/bin/itstool", line 1516, in
doc.apply_its_rules(not(opts.nobuiltins), params=params)
File "/usr/bin/itstool", line 724, in apply_its_rules
self.apply_its_file(os.path.join(itsdir, dfile), params=params)
File "/usr/bin/itstool", line 754, in apply_its_file
if not nss.has_key(nsdef.name):
AttributeError: 'dict' object has no attribute 'has_key'

[ghost]

@juergbi are you sure you're using a version of itstool with this PR applied? If you look at https://github.com/hsoft/itstool/blob/c2e1824671747030234dab895400699cbc1d99be/itstool.in#L772 , in apply_its_file(), you'll see that has_key(), the source of the crash here, is not used.

[juergbi]

No, I tested with the wrong version. Sorry about that. The configure issue still needs to be fixed, though.

[Keruspe]

If this PR fixes at least most of the python3 compat while keeping python2 working, maybe it should be in a new release so that we can gather some testing and feedback?

[Keruspe]

I've made some testing, with this patch applied, it still works just as before with python2.

When testing with python3, I hit a problem though.

Error: Could not merge translations:
must be str or None, not bytes

[Keruspe]

I've tracked it down to the parent.addContent call in append_credits

841-            if val is not None:
842-                val = val.encode('utf-8')
843:                parent.addContent(val)

Any idea @hsoft @shaunix ?

[ghost]

@Keruspe I'm not well set up to test itstool right now. I'm guessing that wrapping the val = val.encode('utf-8') line in a if not PY3: would do the trick. Could you try that?

[Keruspe]

@hsoft applied that, and now I'm getting another error, unicode related:

Traceback (most recent call last):
  File "/usr/bin/itstool", line 1553, in <module>
    fout.write(doc._doc.serialize('utf-8'))
UnicodeEncodeError: 'ascii' codec can't encode character '\xe8' in position 312: ordinal not in range(128)

[ghost]

I had problems reproducing the crash because I don't use itstool directly so I don't really know how to actually use it, but I could finally reproduce it by:

1. Cloning evince (https://github.com/GNOME/evince)
2. Go to help/C
3. Run python /path/to/itstool/itstool.in -i /path/to/itstool/its/mallard.its -m ../fr/fr.mo textselection.page

Strangely though, when applying my first fix, I didn't have the same bug as yours, but a whole different one: my target file (textselection.page) became empty!

So, I fooled around and ended up fixing 3 bugs, in the commit reference above. See commit message for details.

@Keruspe does these fix solve your problem as well?

[ghost]

Oh, yes, one thing: ../fr/fr.mo doesn't actually exists in evince's repo, I msgfmt'd it manually first.

[Keruspe]

@hsoft nice, I confirm it works like a charm now! (confirmed on all the packages that previously failed here)

[Keruspe]

@shaunix Any chance you'd give it a go to merge and make a release?

[shaunix]

So I finally decided to look into merging this, but first I need to get through shaving the yak of figuring out why test_IT_keep_entities_1 is failing.

[Keruspe]

@shaunix any luck with your tests?

[shaunix]

The test failure is due to this libxml2 bug:
https://bugzilla.gnome.org/show_bug.cgi?id=762110

[Keruspe]

@shaunix so it fails with python2 too, right?

If the only blocker is a test filaing and the same test is also already failing on master, is it really a blocker?

[Keruspe]

Any news regarding this?

[shaunix]

Finally got that ridiculous test failure fixed. Now maybe we can get on with Python 3 before Python 4 becomes a thing.