Add setting to disable registered error views for debugging in WSGI s…

…etups (#563)

* - ignore VIM swap files [ci skip]

* - fix calls to publish_module by explicitly naming keyword arguments

* - Exceptions during publishing are now re-raised when in debug mode

* - add section for werkzeug and for debugging Zope applications

* - switching everything over to its own configuration setting

* - fix test

* - silence isort

* - simplify this patch

* - change already in master

* - documentation updates and a test polish

.gitignore

@@ -1,6 +1,7 @@
 *.egg-info
 *.py?
 *.profraw
+*.swp
 .Python
 .coverage
 .coverage.*

CHANGES.rst

@@ -57,6 +57,11 @@ Features
 Other changes
 +++++++++++++
 
+- Exceptions during publishing are now re-raised in a new exceptions debug
+  mode to allow WSGI middleware to handle/debug it. See the `documentation
+  <https://zope.readthedocs.io/en/latest/wsgi.html#werkzeug>`_ for examples.
+  (`#562 <https://github.com/zopefoundation/Zope/issues/562>`_)
+
 - Remove hardcoded list of factories that don't want an add dialog
   (`#540 <https://github.com/zopefoundation/Zope/issues/540>`_)
 

docs/wsgi.rst

@@ -30,8 +30,8 @@ To compose your pipeline in Python code:
    app = make_wsgi_app({}, '/path/to/zope.conf')
 
 
-WSGI tools and helpers for Zope
--------------------------------
+Building a Zope instance with WSGI support
+------------------------------------------
 Zope ships with several helper scripts to set up a default WSGI-enabled
 environment. The document :doc:`operation` walks you through using
 ``mkwsgiinstance`` for a default configuration that you can use in conjunction
@@ -68,8 +68,8 @@ starting point for changes. The `Python Logging Cookbook
 of topics for advanced configurations.
 
 
-WSGI server integrations
-------------------------
+Compatible WSGI servers
+-----------------------
 This section describes how to integrate specific WSGI servers into your Zope
 instance. These servers were chosen because they either have a `PasteDeploy`
 entry point or have one provided by shim software, which means they work with
@@ -210,6 +210,92 @@ section will pull in the correct dependencies:
    wsgi = ${buildout:directory}/etc/bjoern.ini
 
 
+werkzeug
+~~~~~~~~
+`werkzeug <https://palletsprojects.com/p/werkzeug/>`_ is a WSGI library that
+contains not just a WSGI server, but also a powerful debugger. It can
+easily integrate wth Zope using a shim package called `dataflake.wsgi.werkzeug 
+<https://dataflakewsgiwerkzeug.readthedocs.io/>`_. See the `Using this package`
+section for how to integrate `werkzeug` using Zope's own ``runwsgi`` script and
+how to create a suitable WSGI configuration.
+
+If you use ``plone.recipe.zope2instance``, the following section will pull in
+the correct dependencies, after you have created a WSGI configuration file:
+
+.. code-block:: ini
+
+   [zopeinstance]
+   recipe = plone.recipe.zope2instance
+   eggs =
+       dataflake.wsgi.werkzeug
+   zodb-temporary-storage = off
+   user = admin:password
+   http-address = 8080
+   wsgi = ${buildout:directory}/etc/werkzeug.ini
+
+
+Debugging Zope applications under WSGI
+--------------------------------------
+You can debug a WSGI-based Zope application the same way you have debugged
+ZServer-based installations in the past. In addition, you can now take
+advantage of WSGI middleware or debugging facilities built into the chosen
+WSGI server.
+
+When developing your application or debugging, which is the moment you want to
+use debugging tools, you can start your Zope instance in `exceptions debug
+mode`. This will disable all registered exception views including
+``standard_error_message`` so that exceptions are not masked or hidden.
+
+This is how you run Zope in exceptions debug mode using the built-in
+``runwsgi`` script:
+
+.. code-block:: console
+
+   $ bin/runwsgi -e etc/zope.ini
+
+If you built your environment using ``plone.recipe.zope2instance`` you will
+need to do a manual change to your Zope configuration file. Enable exceptions
+debug mode by adding the ``debug-exceptions on`` setting before starting your
+application. The example presumes the Zope instance was named ``zopeinstance``,
+your Zope configuration file will be at `parts/zopeinstance/etc/zope.conf`.
+
+.. code-block:: console
+
+   bin/zopeinstance fg
+
+With Zope set up to let WSGI handle exceptions, these are a few options for the
+WSGI pipeline:
+
+If you use ``waitress``, you can make it output exception tracebacks in the
+browser by configuring ``expose_tracebacks``. The keyword works in both
+standard and ``plone.recipe.zope2instance`` configurations:
+
+.. code-block:: ini
+
+   [server:main]
+   use = egg:waitress#main
+   host = 127.0.0.1
+   port = 8080
+   expose_tracebacks = True
+
+   ... or ...
+
+   [server:main]
+   paste.server_factory = plone.recipe.zope2instance:main
+   use = egg:plone.recipe.zope2instance#main
+   listen = 0.0.0.0:8080
+   threads = 2
+   expose_tracebacks = True
+
+``werkzeug`` includes a full-featured debugging tool. See the
+`dataflake.wsgi.werkzeug documentation
+<https://dataflakewsgiwerkzeug.readthedocs.io/en/latest/usage.html#using-the-werkzeug-debugger>`_
+for how to enable the debugger. Once you're up and running, the `werkzeug
+debugger documentation
+<https://werkzeug.palletsprojects.com/en/0.15.x/debug/#using-the-debugger>`_
+will show you how to use it.
+
+
 WSGI documentation links
 ------------------------
 - the WSGI standard is described in `PEP-3333

src/ZPublisher/WSGIPublisher.py

@@ -49,6 +49,7 @@
 else:
     _FILE_TYPES = (IOBase, file)  # NOQA
 
+_DEFAULT_DEBUG_EXCEPTIONS = False
 _DEFAULT_DEBUG_MODE = False
 _DEFAULT_REALM = None
 _MODULE_LOCK = allocate_lock()
@@ -73,6 +74,16 @@ def validate_user(request, user):
     newSecurityManager(request, user)
 
 
+def set_default_debug_exceptions(debug_exceptions):
+    global _DEFAULT_DEBUG_EXCEPTIONS
+    _DEFAULT_DEBUG_EXCEPTIONS = debug_exceptions
+
+
+def get_debug_exceptions():
+    global _DEFAULT_DEBUG_EXCEPTIONS
+    return _DEFAULT_DEBUG_EXCEPTIONS
+
+
 def set_default_debug_mode(debug_mode):
     global _DEFAULT_DEBUG_MODE
     _DEFAULT_DEBUG_MODE = debug_mode
@@ -191,7 +202,8 @@ def transaction_pubevents(request, response, tm=transaction.manager):
             if retry:
                 reraise(*exc_info)
 
-            if not (exc_view_created or isinstance(exc, Unauthorized)):
+            if not (exc_view_created or isinstance(exc, Unauthorized)) or \
+               getattr(response, 'debug_exceptions', False):
                 reraise(*exc_info)
         finally:
             # Avoid traceback / exception reference cycle.
@@ -206,6 +218,8 @@ def publish(request, module_info):
     request.processInputs()
     response = request.response
 
+    response.debug_exceptions = get_debug_exceptions()
+
     if debug_mode:
         response.debug_mode = debug_mode
 

src/ZPublisher/tests/test_WSGIPublisher.py

@@ -221,15 +221,20 @@ def _callFUT(self, environ, start_response,
         if _publish is not None:
             if _response_factory is not None:
                 if _request_factory is not None:
-                    return publish_module(environ, start_response, _publish,
-                                          _response_factory, _request_factory)
-                return publish_module(environ, start_response, _publish,
-                                      _response_factory)
+                    return publish_module(environ, start_response,
+                                          _publish=_publish,
+                                          _response_factory=_response_factory,
+                                          _request_factory=_request_factory)
+                return publish_module(environ, start_response,
+                                      _publish=_publish,
+                                      _response_factory=_response_factory)
             else:
                 if _request_factory is not None:
-                    return publish_module(environ, start_response, _publish,
+                    return publish_module(environ, start_response,
+                                          _publish=_publish,
                                           _request_factory=_request_factory)
-                return publish_module(environ, start_response, _publish)
+                return publish_module(environ, start_response,
+                                      _publish=_publish)
         return publish_module(environ, start_response)
 
     def _registerView(self, factory, name, provides=None):
@@ -631,6 +636,31 @@ def testHandleErrorsFalseBypassesExceptionResponse(self):
         with self.assertRaises(Unauthorized):
             self._callFUT(environ, start_response, _publish)
 
+    def testDebugExceptionsBypassesExceptionResponse(self):
+        from zExceptions import BadRequest
+
+        # Register an exception view for BadRequest
+        registerExceptionView(IException)
+        environ = self._makeEnviron()
+        start_response = DummyCallable()
+        _publish = DummyCallable()
+        _publish._raise = BadRequest('debugbypass')
+
+        # Responses will always have debug_exceptions set
+        def response_factory(stdout, stderr):
+            response = DummyResponse()
+            response.debug_exceptions = True
+            return response
+
+        try:
+            # With debug_exceptions, the exception view is not called.
+            with self.assertRaises(BadRequest):
+                self._callFUT(environ, start_response, _publish,
+                              _response_factory=response_factory)
+        finally:
+            # Clean up view registration
+            unregisterExceptionView(IException)
+
 
 class ExcViewCreatedTests(ZopeTestCase):
 

src/Zope2/Startup/run.py

@@ -62,6 +62,9 @@ def make_wsgi_app(global_config, zope_conf):
     if 'debug_mode' in global_config:
         if global_config['debug_mode'] in ('true', 'on', '1'):
             opts.configroot.debug_mode = True
+    if 'debug_exceptions' in global_config:
+        if global_config['debug_exceptions'] in ('true', 'on', '1'):
+            opts.configroot.debug_exceptions = True
     handleWSGIConfig(opts.configroot, opts.confighandlers)
     setConfiguration(opts.configroot)
     starter.setConfiguration(opts.configroot)

src/Zope2/Startup/serve.py

@@ -130,6 +130,12 @@ class ServeCommand(object):
         const=1,
         dest='debug',
         help="Enable debug mode.")
+    parser.add_option(
+        '-e', '--debug-exceptions',
+        action='store_const',
+        const=1,
+        dest='debug_exceptions',
+        help="Enable exceptions debug mode.")
 
     _scheme_re = re.compile(r'^[a-z][a-z]+:', re.I)
 
@@ -180,6 +186,8 @@ def run(self):
 
         if 'debug_mode' not in vars and self.options.debug:
             vars['debug_mode'] = 'true'
+        if 'debug_exceptions' not in vars and self.options.debug_exceptions:
+            vars['debug_exceptions'] = 'true'
         app = self.loadapp(app_spec, name=app_name, relative_to=base,
                            global_conf=vars)
 

src/Zope2/Startup/starter.py

@@ -74,6 +74,7 @@ def setupPublisher(self):
         import ZPublisher.HTTPRequest
         from ZPublisher import WSGIPublisher
         WSGIPublisher.set_default_debug_mode(self.cfg.debug_mode)
+        WSGIPublisher.set_default_debug_exceptions(self.cfg.debug_exceptions)
         WSGIPublisher.set_default_authentication_realm(
             self.cfg.http_realm)
         if self.cfg.trusted_proxies:

src/Zope2/Startup/wsgischema.xml

@@ -131,6 +131,22 @@
     <metadefault>off</metadefault>
   </key>
 
+  <key name="debug-exceptions" datatype="boolean" default="off">
+    <description>
+    This switch controls how exceptions are handled. If it is set to
+    "off" (the default), Zope's own exception handling is active.
+    Exception views or a standard_error_message are used to handle them.
+
+    If set to "on", exceptions are not handled by Zope and can propagate
+    into the WSGI pipeline, where they may be handled by debugging
+    middleware.
+
+    This setting should always be "off" in production. It is useful for
+    developers and while debugging site issues.
+    </description>
+    <metadefault>off</metadefault>
+  </key>
+
   <key name="locale" datatype="locale" handler="locale">
     <description>
      Locale name to be used. See your operating system documentation for locale

src/Zope2/utilities/skel/etc/wsgi.conf.in

@@ -81,6 +81,27 @@ instancehome $INSTANCE
 #    debug-mode on
 
 
+# Directive: debug-exceptions
+#
+# Description:
+#     This switch controls how exceptions are handled. If it is set to
+#     "off" (the default), Zope's own exception handling is active.
+#     Exception views or a standard_error_message are used to handle them.
+#
+#     If set to "on", exceptions are not handled by Zope and can propagate
+#     into the WSGI pipeline, where they may be handled by debugging
+#     middleware.
+#
+#     This setting should always be "off" in production. It is useful for
+#     developers and while debugging site issues.
+#
+# Default: off
+#
+# Example:
+#
+#    debug-exceptions on
+
+
 # Directive: http-realm
 #
 # Description: