Weekly Report VI (July 2~July 8)

Weekly Report VI

Done this week

The mainly job done this week is to migrate dpkt.py, decorators.py, crc32c.py and sctp.py to Python 3. Please check more details on migrate_py3 branch, https://github.com/kbandla/dpkt/compare/migrate_py3. Specifically, the migration job contains 4 aspects, which are listed as follows.

2to3 initial conversion

Use 2to3 to get an initial result which is compatible for Python 3. 2to3 can automatically solve the following migration issues (however there are still some issues needed to be fixed manually, which will be elaborated in Point 2 ~ 4).

• Python 2 has two integer types int and long. These have been unified in Python 3, so there is now only one type, int. Here is an example.

Python 2

tmp = ~crc & 0xffffffffL

Python 2 & 3

tmp = ~crc & 0xffffffff

• The Python 2 print statement is in Python 3 a function.

Python 2

print '%s : time = %f kstones = %f' % (function.__name__, time, kstones)

Python 2 & 3

print('%s : time = %f kstones = %f' % (function.__name__, time, kstones))

• In Python 3 the syntax to catch exceptions have changed.

Python 2

except struct.error, e:

Python 2 & 3

except struct.error as e:

import and dict related syntax.

The import and dict are both changed in Python 3. Note that the 2to3 automatic update cannot provide compatible code for both Python 2 and 3. Thus we should update the code based on different cases. Here are examples.

Python 2

from StringIO import StringIO

Python 2 & 3

try:
    from StringIO import StringIO
except ImportError:
    from io import StringIO

Metaclass related issues

We use metaclass in dpkt.py module. Unfortunately the syntax of metaclass in Python 2 and 3 are not the same.

Python 2

class Packet(object):
    __metaclass__ = _MetaPacket

Python 3

class Packet(object, metaclass=_MetaPacket):

I look up some materials, seeking for a proper solution for this problem, the following links could be a good reference for this issue.

http://python-future.org/compatible_idioms.html#metaclasses

http://www.zopatista.com/python/2014/03/14/cross-python-metaclasses/

Then the easily and maybe the most proper way to solve this problem is to use six module. The modified code looks like

Python 2 & 3

with_metaclass(_MetaPacket, object)

Thus the problem is can we use six as an additional dependency?

Bytes and strings

This is the trickiest problem I met this week, and I spent most of time looking for a proper solution for this. There are also some discussions on this problem.

http://python3porting.com/problems.html#bytes-strings-and-unicode

http://stackoverflow.com/questions/5471158/typeerror-str-does-not-support-the-buffer-interface

http://python-future.org/compatible_idioms.html#strings-and-bytes

In Python 2, we use str objects to hold binary data and ASCII text. While In Python 3, instead of str and unicode objects, we use bytes objects for binary data and str objects for all kinds of text data, Unicode or not.

To solve the problem, we need to manually update all methods that deal with binary data from str to bytes, such as pack() and unpack(). Please see the detailed implementation in the current commit, https://github.com/kbandla/dpkt/compare/migrate_py3.

General Plan