JUnitReporter fails with Python 3.x

[jenisys]

When you use behave --junit ... with Python 3.x you get an exception related to XML/Unicode.

$ behave --junit ...
...
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.3/bin/behave", line 9, in <module>
    load_entry_point('behave==1.2.2', 'console_scripts', 'behave')()
...
  File ".../python3.3/site-packages/behave/reporter/junit.py", line 125, in feature
    tree.write(open(report_filename, 'w'), 'utf8')
  File ".../python3.3/xml/etree/ElementTree.py", line 828, in write
    serialize(write, self._root, qnames, namespaces)
  File ".../python3.3/contextlib.py", line 66, in __exit__
    self.gen.throw(type, value, traceback)
...
  File ".../python3.3/contextlib.py", line 200, in _exit_wrapper
    callback(*args, **kwds)
TypeError: must be str, not bytes

SEE ALSO:
Issue #80 comment.

[javornikolov]

I actually tried to somehow run that from source and I observed some other incompatibilities:

• exception handling: exception name in Python 3 should be separated with as (this is supposed to have been backported to Python 2.7)
• the class unicode has gone - and we're inheriting from it. (Maybe str should be used instead)
• print is now a function and should be invoked with ()
• ConfigParser should be replaced with configparser
• StringIO has gone (probably should be replaced by io.StringIO or io.BytesIO

[jenisys]

behave is currently distributed as Python2 source code.
When you try to use it under Python3 as source distribution (or python3 setup.py develop), you should run the 2to3 conversion tool to correct/patch the source files. This should fix all of the problems above. When you install behave for Python3 this happens automatically in the background while using python3 setup.py install.

To convert the source distribution run:

$ cd behave-1.2.2

# -- Show changes that will be made without performing any changes
$ 2to3 .
...

# -- Apply conversion fixes, maybe use also: -n (no-backup), -p (print-function fixes)
$ 2to3 -w
...

NOTE:
In certain cases you need to apply the 2to3 conversion several times.

SEE ALSO:

• http://docs.python.org/3/howto/index.html
• http://docs.python.org/3/howto/pyporting.html
• http://docs.pythonsprints.com/python3_porting/py-porting.html

[jenisys]

One additional note over which I stumbled during the last few days.
When you try to use tox to run the test, the current tox.ini will only work when you use either Python2 or Python3 (with converted Python3 sources). The reason for this is that tox runs per default in the toxinidir = BEHAVE_ROOT_DIR. Therefore, it will find the sources/tests there before finding the sdist-package that is installed in the virtual-environment. The general solution for this is to use the [testenv] changedir = {toxworkdir}/{envname} in tox.ini (or another directory). This is not yet done.

[aragaer]

I have managed to get JUnitReporter working in python3. For this i had to fix 2 problems:

1. File format has to be "wb" or encoding has to be "unicode". This is explicitly mentioned here: http://docs.python.org/3.3/library/xml.etree.elementtree.html#xml.etree.ElementTree.ElementTree.write
2. (at least in xml.etree v 1.3.0 that I have installed) - ElementTree.write doesn't pass encoding parameter to _serialize. However ElementTreeWithCDATA._serialize_xml is expecting that parameter.

[midopa]

I'm seeing this issue with Python 3.3:

Traceback (most recent call last):
  File "/Users/sampar/apps/behave/behave/__main__.py", line 133, in <module>
    sys.exit(main())
  File "/Users/sampar/apps/behave/behave/__main__.py", line 111, in main
    failed = runner.run()
  File "/Users/sampar/apps/behave/behave/runner.py", line 659, in run
    return self.run_with_paths()
  File "/Users/sampar/apps/behave/behave/runner.py", line 680, in run_with_paths
    return self.run_model()
  File "/Users/sampar/apps/behave/behave/runner.py", line 495, in run_model
    reporter.feature(feature)
  File "/Users/sampar/apps/behave/behave/reporter/junit.py", line 124, in feature
    tree.write(report_filename, 'UTF-8')
  File "/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/xml/etree/ElementTree.py", line 830, in write
    serialize(write, self._root, qnames, namespaces)
TypeError: _serialize_xml() missing 1 required positional argument: 'namespaces'

@aragaer When I step through the code, serialize_xml is not expecting encoding, it has four arguments:
def _serialize_xml(write, elem, qnames, namespaces)

Setting to the file handler to wb mode didn't work. Have you seen this error?

[aragaer]

This is the patch I'm manually applying to get JUnit reporter work:

--- env/lib/python3.3/site-packages/behave/reporter/junit.py    2013-09-06 02:27:27.000000000 +0400
+++ env/lib/python3.3/site-packages/behave/reporter/junit.py    2013-11-02 02:10:44.623216762 +0400
@@ -18,24 +18,22 @@


 class ElementTreeWithCDATA(ElementTree.ElementTree):
-    def _write(self, file, node, encoding, namespaces):
+    def _write(self, file, node, namespaces):
         """This method is for ElementTree <= 1.2.6"""

         if node.tag == '![CDATA[':
-            text = node.text.encode(encoding)
-            file.write("\n<![CDATA[%s]]>\n" % text)
+            file.write("\n<![CDATA[%s]]>\n" % node.text)
         else:
-            ElementTree.ElementTree._write(self, file, node, encoding,
-                                           namespaces)
+            ElementTree.ElementTree._write(self, file, node, namespaces)


 if hasattr(ElementTree, '_serialize'):
-    def _serialize_xml(write, elem, encoding, qnames, namespaces,
+    def _serialize_xml(write, elem, qnames, namespaces,
                        orig=ElementTree._serialize_xml):
         if elem.tag == '![CDATA[':
-            write("\n<%s%s]]>\n" % (elem.tag, elem.text.encode(encoding)))
+            write("\n<%s%s]]>\n" % (elem.tag, elem.text))
             return
-        return orig(write, elem, encoding, qnames, namespaces)
+        return orig(write, elem, qnames, namespaces)

     ElementTree._serialize_xml = ElementTree._serialize['xml'] = _serialize_xml

@@ -120,7 +118,7 @@

         tree = ElementTreeWithCDATA(suite)
         report_filename = os.path.join(self.config.junit_directory, filename)
-        tree.write(codecs.open(report_filename, 'w'), 'UTF-8')
+        tree.write(codecs.open(report_filename, 'w'), 'unicode')

     # -- MORE:
     @staticmethod

[midopa]

Thanks @aragaer ! What about submitting a pull request for this change?

[aragaer]

I doubt this patch will not break compatibility with Python 2.

[midopa]

Have any of you considered using lxml? I just tried it out to get proper line break printing for CDATA sections. It seems to work nicely without any of the class/method overloading stuff.

Ref: https://github.com/midopa/behave/commit/0e309fec4f7e6ede4582c4e7d8a09ad4b43809f6

[midopa]

Actually, scratch that. I just checked and the CDATA is alright with the current implementation. Chrome was just mangling all the line breaks. That said, lxml does make it a bit simpler by removing the need to have custom CDATA classes and serialization overloading and presumably it's faster?

[jenisys]

@midopa
I really like lxml. I am using it in another project extensively where I deal with processing and retrieving information from a number of XML files (mostly with lxml.objectify).
But I don't want to make behave dependent on lxml (as external Python package which needs to be compiled). The necessity in behave is less, just outputting some XML stuff (for JUnit…). Therefore, I currently do not want to introduce this dependency on lxml (by intention).

[vortec]

I second this request. Since behave works fine with Python 3, I'd greatly appreciate the availability to have it generate JUnit files as well.

[benoitroussel]

Same here with Python 3.4. When I set the junit and junit_directory options in behave.ini or from the command line, I get the following error:

/Library/Frameworks/Python.framework/Versions/3.4/bin/python3.4 "/Users/myusername/Documents/Programming/Selenium Programming/GMail Project/GMailTests.py"
Traceback (most recent call last):
  File "/Users/myusername/Documents/Programming/Selenium Programming/GMail Project/GMailTests.py", line 62, in <module>
    config = Configuration()
  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/behave/configuration.py", line 481, in __init__
    load_configuration(self.defaults, verbose=verbose)
  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/behave/configuration.py", line 394, in load_configuration
    defaults.update(read_configuration(filename))
  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/behave/configuration.py", line 348, in read_configuration
    result[dest] = cfg.get('behave', dest, use_raw_value)
TypeError: get() takes 3 positional arguments but 4 were given

As soon as I switch back to Python 2.7, everything works fine.
Is there any plan to address this in a new release anytime soon?
Is there a patch available for this error in the meantime?

[jenisys]

@benoitroussel
This problem is already fixed (in one of the last commits). Use the newest version (tip) of this GIT repository.

[benoitroussel]

Thanks a lot @jenisys !

[aragaer]

Traceback (most recent call last):
  File "/home/aragaer/Projects/zenclash/env/bin/behave", line 9, in <module>
    load_entry_point('behave==1.2.5a1', 'console_scripts', 'behave')()
  File "/home/aragaer/Projects/zenclash/env/lib/python3.4/site-packages/behave/__main__.py", line 109, in main
    failed = runner.run()
  File "/home/aragaer/Projects/zenclash/env/lib/python3.4/site-packages/behave/runner.py", line 665, in run
    return self.run_with_paths()
  File "/home/aragaer/Projects/zenclash/env/lib/python3.4/site-packages/behave/runner.py", line 686, in run_with_paths
    return self.run_model()
  File "/home/aragaer/Projects/zenclash/env/lib/python3.4/site-packages/behave/runner.py", line 495, in run_model
    reporter.feature(feature)
  File "/home/aragaer/Projects/zenclash/env/lib/python3.4/site-packages/behave/reporter/junit.py", line 122, in feature
    tree.write(codecs.open(report_filename, 'w'), 'UTF-8')
  File "/usr/lib/python3.4/xml/etree/ElementTree.py", line 778, in write
    short_empty_elements=short_empty_elements)
TypeError: _serialize_xml() got an unexpected keyword argument 'short_empty_elements'

$ pip freeze | grep behave
behave==1.2.5a1

[jenisys]

@aragaer
This seems to be more a problem of the Python library than behave.
I just looked over the "xml.etree.ElementTree.py" code and it looks OK after first inspection.
_serialize_xml() has a parameter with this name. Therefore, you should be able to use it as named argument.

[aragaer]

Python's _serialize_xml() does have that parameter. When _serialize_xml() is called, the parameter is given to it and it is expected that parameter will be accepted by that function. However this one doesn't have that parameter: https://github.com/behave/behave/blob/master/behave/reporter/junit.py#L33

[nikhil2712]

@jenisys
Is there any update on this.

[actionless]

@nikhil2712 it should be fixed with this commit:
c90ef5d

[jenisys]

Problem no longer observed with current tip of the repository (behave-1.2.5a1).

XML issues fixed by pull #257.
Other unicode issues may be fixed by improved Python3 support (related to: #268 changes).