Adding integration tests from docs into starter project fails

[stevepiercy]

The integration tests as they appear in the docs on master branch do not pass in a project created with the starter scaffold. Here are steps to reproduce.

1. Create a workspace, virtual environment for Python 2.7, and install Pyramid 1.6b2.

2. Create a starter project MyProject according to the docs on master branch http://docs.pylonsproject.org/projects/pyramid/en/master/narr/project.html

   $ $VENV/bin/pcreate -s starter MyProject
   $ cd MyProject
   $ $VENV/bin/python setup.py develop
   $ $VENV/bin/python setup.py test -q
   

3. Grab the integration tests from the testing chapter, copy and paste into myproject/tests.py the following test code. http://docs.pylonsproject.org/projects/pyramid/en/master/narr/testing.html#creating-integration-tests

   class ViewIntegrationTests(unittest.TestCase):
       def setUp(self):
           """ This sets up the application registry with the
           registrations your application declares in its ``includeme``
           function.
           """
           self.config = testing.setUp()
           self.config.include('myproject')
   
       def tearDown(self):
           """ Clear out the application registry """
           testing.tearDown()
   
       def test_my_view(self):
           from myproject.views import my_view
           request = testing.DummyRequest()
           result = my_view(request)
           self.assertEqual(result.status, '200 OK')
           body = result.app_iter[0]
           self.assertTrue('Welcome to' in body)
           self.assertEqual(len(result.headerlist), 2)
           self.assertEqual(result.headerlist[0],
                            ('Content-Type', 'text/html; charset=UTF-8'))
           self.assertEqual(result.headerlist[1], ('Content-Length',
                                                   str(len(body))))
   

4. Run tests:

   $ $VENV/bin/python setup.py test -q
   ...
   .E
   ======================================================================
   ERROR: test_my_view (myproject.tests.ViewIntegrationTests)
   ----------------------------------------------------------------------
   Traceback (most recent call last):
     File "/Users/stevepiercy/projects/hack-on-pyramid/MyProject/myproject/tests.py", line 27, in setUp
       self.config.include('myproject')
     File "/Users/stevepiercy/projects/hack-on-pyramid/pyramid/config/__init__.py", line 776, in include
       "module %r has no attribute 'includeme'" % (module.__name__)
   ConfigurationError: module 'myproject' has no attribute 'includeme'
   
   ----------------------------------------------------------------------
   Ran 2 tests in 0.221s
   
   FAILED (errors=1)
   
   Process finished with exit code 1
   

I don't know what should go into an includeme function in the module myproject, or where it should be placed.

[digitalresistor]

Basically instead of having a main() that returns a WSGI App, you would have a includeme() that does everything but return a WSGI app.

Then your main() would call config.includeme() and return a WSGI app.

[stevepiercy]

Here's the content of the original myproject/__init__.py before my failing attempts:

from pyramid.config import Configurator

def main(global_config, **settings):
    """ This function returns a Pyramid WSGI application.
    """
    config = Configurator(settings=settings)
    config.include('pyramid_chameleon')
    config.add_static_view('static', 'static', cache_max_age=3600)
    config.add_route('home', '/')
    config.scan()
    return config.make_wsgi_app()

and after:

from pyramid.config import Configurator


def includeme(config):
    config.include('pyramid_chameleon')
    config.add_static_view('static', 'static', cache_max_age=3600)
    config.add_route('home', '/')
    config.scan()


def main(global_config, **settings):
    """ This function returns a Pyramid WSGI application.
    """
    config = Configurator(settings=settings)
    # config.include('pyramid_chameleon')
    # config.add_static_view('static', 'static', cache_max_age=3600)
    # config.add_route('home', '/')
    # config.scan()
    config.include('myproject')
    return config.make_wsgi_app()

The app runs, but the integration test still fails:

======================================================================
ERROR: test_my_view (myproject.tests.ViewIntegrationTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/stevepiercy/projects/hack-on-pyramid/MyProject/myproject/tests.py", line 37, in test_my_view
    self.assertEqual(result.status, '200 OK')
AttributeError: 'dict' object has no attribute 'status'

----------------------------------------------------------------------

The dict is simply {'project': 'MyProject'}. No surprise there based on views.py content:

from pyramid.view import view_config


@view_config(route_name='home', renderer='templates/mytemplate.pt')
def my_view(request):
    return {'project': 'MyProject'}

That means the integration test is not written to test what it says it should test. I'm not sure what to do from here.

[digitalresistor]

The documentation for an integration test is wrong. It's doing a unit test of the view only, sure all of your routes and everything are also set up, but the view machinery itself is not run so you don't get back anything but the raw data from the view function, not a valid Response() object.

@mmerickel or @mcdonc might be able to shed some more insight, but I do think that the Integration Tests as it currently stands are bad and should feel bad.

[mcdonc]

Integration (as opposed to unit or functional) tests are kind of the worst kinds of tests to write and read, because you need to understand the system at a very deep level to make sense of how the test is working. The testing chapter apparently tries to explain how to do it, and it appears to be wrong. I would probably not add integration tests to the scaffold, or at least I'd remove them where they already exist.

[stevepiercy]

@mcdonc I have no desire to add the integration test to the starter scaffold. Instead I am trying to apply the example in the docs to a project created from the starter scaffold.

So at this point, what should I do with the entire docs section Creating Integration Tests? I'd appreciate any suggestions, whether it is some edits or its entire removal.

(BTW, the functional test that was added to the docs around the same time as the integration test works.)

[stevepiercy]

I did a little more research on this one. I couldn't find a mention that anyone ever ran the integration test to ensure it actually works.

Git blame led me to an ambiguously named PR #1467. This merely moved the example code from inline in the docs to an external file, then used :literalinclude: to render the code inline.

So I went back to the original version, and traced it forward. There were various updates from its original version, but I didn't see any changes that would have broken the tests along the way. meh.

We don't want wrong or broken examples in the docs, but we also don't want to remove the entire section, so my proposal now is to do the following:

• Remove the example integration test code and any reference to it in the docs.
• Close this issue.
• Open a new issue "Add example integration test code to narr/testing.rst"

Any opinions for/against/do something differently?

[digitalresistor]

The reason for the integration tests is that everything is registered as if it were running for real. Which means that other than a fake Request object everything else is considered "live". This type of testing requires that you understand not just Pyramid, but also your entire application stack.

In the case of the example view, it doesn't do anything with the fact that all of the other stuff is registered and ready like a real request coming in.

I think it is a good idea to remove the section, and add it back in at a later point in time.

[stevepiercy]

Closing per #2180 and in favor of a clearly defined issue #2181.