pkgutil.get_data support

[xmo-odoo]

pkgutil.get_data currently only returns None when called from a module py.test matches, which is inconvenient when trying to load test data from a file in a test package.

This happens because AssertionRewritingHook sets itself as the module's loader, but it does not implement the (optional) PEP302 get_data loader API. As a result, pkgutil.get_data simply returns None when invoked on a test module while running py.test.

It would be nice if either

• AssertionRewritingHook didn't set itself as the module's loader, though I don't know what the side-effects would be if it set pkgutil.ImpLoader as the loader

• or AssertionRewritingHook implemented get_data, for which the trivial implementation seems like it should work given the rewriting hook correctly sets __file__ on the loaded module:

  def get_data(self, pathname):
      return open(pathname, "rb").read()
  

[nicoddemus]

Would care to open a PR, or perhaps give a use case to test this with?

[xmo-odoo]

Well our use case is the test for an import function with a "big" import (to ensure nothing blows up and get perf info), there's ~50k of test data so it's not really convenient to have it inlined in the test module. The test data lives in a json file next to the test module instead, and is opened via get_data() to try and avoid overly depending on living in an actual filesystem.

[nicoddemus]

I could not reproduce the problem... here's what I tried:

Created a simple text file foo_data.txt:

Hey

# contents of foo.py
def foo():
    return 1

# contents of test_foo.py
import pkgutil
def test_foo():
    assert pkgutil.get_data('foo', 'foo_data.txt') == 'Hey'

This test passes for me, both for master and for pytest-2.7:

============================= test session starts =============================
platform win32 -- Python 2.7.10 -- py-1.4.30 -- pytest-2.7.2
rootdir: X:\pytest, inifile: tox.ini
collected 1 items

test_foo.py .

========================== 1 passed in 0.01 seconds ===========================

If you could provide a minimum example with the problem, I would be happy to work on a fix.

[xmo-odoo]

Yeah the problem is the test module being used as base to get the test data (otherwise AssertionRewritingHook probably isn't involved)

Create a foo/__init__.py, foo/data.txt with Hey\n and in foo/test_foo.py

# contents of test_foo.py
import pkgutil
import unittest

class TestFoo(unittest.TestCase):
    def test_foo(self):
        assert pkgutil.get_data(self.__module__, 'data.txt') == 'Hey\n'

then from outside foo

> python -munittest foo.test_baz
.
----------------------------------------------------------------------
Ran 1 test in 0.000s

OK

> py.test foo
============================= test session starts =============================
platform darwin -- Python 2.7.10 -- py-1.4.30 -- pytest-2.7.2
rootdir: test/foo, inifile: 
collected 1 items 

foo/test_baz.py F

================================ FAILURES ================================
________________________________ TestFoo.test_foo _______________________________

self = <foo.test_baz.TestFoo testMethod=test_foo>

    def test_foo(self):
>       assert pkgutil.get_data(self.__module__, 'data.txt') == 'Hey\n'
E       AssertionError: assert None == 'Hey\n'
E        +  where None = <function get_data at 0x10a9e1488>('foo.test_baz', 'data.txt')
E        +    where <function get_data at 0x10a9e1488> = pkgutil.get_data
E        +    and   'foo.test_baz' = <foo.test_baz.TestFoo testMethod=test_foo>.__module__

foo/test_baz.py:7: AssertionError
=========================== 1 failed in 0.03 seconds ===========================

If I monkeypatch the AssertionRewritingHook with a (crappy) get_data in foo/__init__.py:

from _pytest.assertion.rewrite import AssertionRewritingHook

AssertionRewritingHook.get_data = lambda self, pathname: open(pathname, 'rb').read()

then the test passes:

platform darwin -- Python 2.7.10 -- py-1.4.30 -- pytest-2.7.2
rootdir: test/foo, inifile: 
collected 1 items 

foo/test_baz.py .

[nicoddemus]

Thanks, I will give it a try.