Merge branch 'master' into re-345-appveyor

CHANGES.rst

@@ -44,9 +44,15 @@ Fixes
   Python 3.
   (`#577 <https://github.com/zopefoundation/Zope/pull/577>`_)
 
+- Prevent ``FindSupport.ZopeFind`` from throwing ``UnicodeDecodeErrors``
+  (`#594 <https://github.com/zopefoundation/Zope/issues/594>`_)
+
 Features
 ++++++++
 
+- Add a configuration flag to show bookmarkable URLs in the ZMI
+  (`#580 <https://github.com/zopefoundation/Zope/issues/580>`_)
+
 - Add a flag for suppressing object events during file import
   (`#42 <https://github.com/zopefoundation/Zope/issues/42>`_)
 
@@ -65,6 +71,9 @@ Features
 Other changes
 +++++++++++++
 
+- Changed the WSGI configuration template so those annoying waitress queue
+  messages only go into the event log, but not onto the console.
+
 - Change naming for the generated WSGI configurations to ``zope.conf`` and
   ``zope.ini`` to match existing documentation for Zope configurations.
   (`#571 <https://github.com/zopefoundation/Zope/issues/571>`_)
@@ -357,7 +366,7 @@ New features
   `#307 <https://github.com/zopefoundation/Zope/pull/307>`_)
 
 - Add zconsole module for running scripts and interactive mode.
-  See the `document Running Zope 
+  See the `document Running Zope
   <https://zope.readthedocs.io/en/latest/operation.html#debugging-zope>`_.
 
 - Add support for Python 3.7.

constraints.txt

@@ -1,15 +1,15 @@
-AccessControl==4.0b7
+AccessControl==4.0
 Acquisition==4.6
 AuthEncoding==4.1
 BTrees==4.5.1
 Chameleon==3.6.1
 DateTime==4.3
-DocumentTemplate==3.0b9
+DocumentTemplate==3.0
 ExtensionClass==4.4
 Missing==4.1
 MultiMapping==4.1
 PasteDeploy==2.0.1
-Persistence==3.0b4
+Persistence==3.0
 Products.BTreeFolder2==4.2
 Products.ZCatalog==4.4
 Record==3.5
@@ -20,16 +20,16 @@ WebTest==2.0.33
 ZConfig==3.4.0
 ZEO==5.2.1
 ZODB==5.5.1
-ZServer==4.0b3 ; python_version < '3.0'
+ZServer==4.0 ; python_version < '3.0'
 Zope2==4.0b1
 five.globalrequest==99.1
 five.localsitemanager==3.2.2
 funcsigs==1.0.2
 future==0.17.1
 ipaddress==1.0.22
-mock==2.0.0
+mock==3.0.5
 pbr==5.2.0
-persistent==4.4.3
+persistent==4.5.0
 pytz==2019.1
 roman==3.2
 shutilwhich==1.1.0

docs/wsgi.rst

@@ -68,16 +68,84 @@ starting point for changes. The `Python Logging Cookbook
 of topics for advanced configurations.
 
 
-Compatible WSGI servers
------------------------
-This section describes how to integrate specific WSGI servers into your Zope
-instance. These servers were chosen because they either have a `PasteDeploy`
-entry point or have one provided by shim software, which means they work with
-the default Zope scripts for starting/stopping the service.
-
-
-waitress (the default)
-~~~~~~~~~~~~~~~~~~~~~~
+Choosing WSGI server software
+-----------------------------
+The WSGI integration gives you a choice of WSGI server software to run your
+Zope application. This section lists several options that were selected
+because they either have a `PasteDeploy` entry point or have one provided by
+shim software, which means they work with the default Zope scripts for
+starting/stopping the service.
+
+
+Things to watch out for
+~~~~~~~~~~~~~~~~~~~~~~~
+The ZODB uses connection pooling where a working thread grabs a connection
+from the pool to serve content and then releases it when the work is done.
+The default size of this connection pool is 7. The advice from ``ZServer``
+days to choose a number of application threads that stays safely below that
+number of ZODB connections is still valid. ``ZServer`` used 4 threads by
+default, so if the WSGI server lets you configure the number of threads 4 is
+still a safe choice.
+
+Another recommendation from Zope 2 is still valid as well: If you have a choice
+between less Zope instances with a higher number of threads each, or more
+instances with less threads each, choose the latter. Create more separate Zope
+instances and set the WSGI server threads value to e.g. 2.
+
+.. warning::
+
+   If the WSGI server software lets you configure a number of worker processes,
+   like ``gunicorn`` does, do not configure more than a single worker.
+   Otherwise you will see issues due to concurrent ZODB access by more than
+   one process, which may corrupt your ZODB.
+
+
+Test criteria for recommendations
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+A simple contrived load test was done with the following parameters:
+
+- 100 concurrent clients accessing Zope
+- 100 seconds run time
+- the clients just fetch "/"
+- standard Zope 4 instances, one with ZEO and one without
+- Python 2.7.16 on macOS Mojave/10.14.4
+- standard WSGI server configurations, the only changes are to number of
+  threads and/or number of workers where available.
+
+This load test uncovered several issues:
+
+- ``cheroot`` (tested version: 6.5.5) was magnitudes slower than all others.
+  Unlike the others, it did not max out CPU. It is unclear where the slowdown
+  originates. Others reached 500-750 requests/second. ``cheroot`` only served
+  12 requests/second per configured thread.
+- ``gunicorn`` (tested version: 19.9.0) showed very strange behavior against
+  the non-ZEO Zope instance. It serves around 500 requests/second, but then
+  hangs and serves no requests for several seconds, before picking up again.
+- ``gunicorn`` (tested version: 19.9.0) does not like the ZEO instance at all.
+  No matter what configuration in terms of threads or workers was chosen
+  ``gunicorn`` just hung so badly that even CTRL-C would not kill it.
+  Switching to an asynchronous type of worker (tested with ``gevent``)
+  did not make a difference.
+- ``werkzeug`` (tested version: 0.15.2) does not let you specify the number
+  of threads, you only tell it to use threads or not. In threaded mode it
+  spawns too many threads and immedialy runs up agains the ZODB connection
+  pool limits, so with Zope only the unthreaded mode is suitable. Even in
+  unthreaded mode, the service speed was inconsistent. Just like ``gunicorn``
+  it had intermittent hangs before recovering.
+- ``bjoern`` (tested version: 3.0.0) is the clear speed winner with 740
+  requests/second against both the ZEO and non-ZEO Zope instance, even though
+  it is single-threaded.
+- ``waitress`` (tested version: 1.3.0) is the all-around best choice. It's
+  just 10-15% slower than ``bjoern``, but both the built-in WSGI tools as well
+  as ``plone.recipe.zope2instance`` use it as the default and make it very
+  convenient to use.
+
+
+Recommended WSGI servers
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+waitress (the default and recommended choice)
++++++++++++++++++++++++++++++++++++++++++++++
 If you create a Zope instance using the ``mkwsgiinstance`` script described
 above or the ``plone.recipe.zope2instance`` buildout recipe, you will
 automatically get a ``waitress``-based server. The default configurations set
@@ -103,8 +171,58 @@ list is `part of the waitress documentation
 <https://docs.pylonsproject.org/projects/waitress/en/stable/arguments.html>`_.
 
 
+bjoern (the fastest)
+++++++++++++++++++++
+The `bjoern WSGI server <https://github.com/jonashaag/bjoern>`_ can be
+integrated using a shim package called `dataflake.wsgi.bjoern
+<https://dataflakewsgibjoern.readthedocs.io/>`_. See the `Using this package`
+section for details on how to integrate `bjoern` using Zope's own
+``runwsgi`` script and how to create a suitable WSGI configuration.
+
+If you use ``plone.recipe.zope2instance``, the following
+section will pull in the correct dependencies:
+
+.. code-block:: ini
+
+   [zopeinstance]
+   recipe = plone.recipe.zope2instance
+   eggs =
+       dataflake.wsgi.bjoern
+   zodb-temporary-storage = off
+   user = admin:password
+   http-address = 8080
+   wsgi = ${buildout:directory}/etc/bjoern.ini
+
+
+Problematic WSGI servers
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+werkzeug
+++++++++
+`werkzeug <https://palletsprojects.com/p/werkzeug/>`_ is a WSGI library that
+contains not just a WSGI server, but also a powerful debugger. It can
+easily integrate wth Zope using a shim package called `dataflake.wsgi.werkzeug 
+<https://dataflakewsgiwerkzeug.readthedocs.io/>`_. See the `Using this package`
+section for how to integrate `werkzeug` using Zope's own ``runwsgi`` script and
+how to create a suitable WSGI configuration.
+
+If you use ``plone.recipe.zope2instance``, the following section will pull in
+the correct dependencies, after you have created a WSGI configuration file:
+
+.. code-block:: ini
+
+   [zopeinstance]
+   recipe = plone.recipe.zope2instance
+   eggs =
+       dataflake.wsgi.werkzeug
+   zodb-temporary-storage = off
+   user = admin:password
+   http-address = 8080
+   wsgi = ${buildout:directory}/etc/werkzeug.ini
+
+
 gunicorn
-~~~~~~~~
+++++++++
 The `gunicorn WSGI server <https://gunicorn.org/>`_ has a built-in
 `PasteDeploy` entry point and integrates easily. The following example buildout
 configuration section will create a ``bin/runwsgi`` script that uses
@@ -165,7 +283,7 @@ path to the path of the configuration file you created yourself:
 
 
 cheroot
-~~~~~~~
++++++++
 The `cheroot WSGI server <https://cheroot.cherrypy.org>`_ can be integrated
 using a shim package called `dataflake.wsgi.cheroot
 <https://dataflakewsgicheroot.readthedocs.io/>`_. See the `Using this package`
@@ -187,53 +305,6 @@ section will pull in the correct dependencies:
    wsgi = ${buildout:directory}/etc/cheroot.ini
 
 
-bjoern
-~~~~~~
-The `bjoern WSGI server <https://github.com/jonashaag/bjoern>`_ can be
-integrated using a shim package called `dataflake.wsgi.bjoern
-<https://dataflakewsgibjoern.readthedocs.io/>`_. See the `Using this package`
-section for details on how to integrate `bjoern` using Zope's own
-``runwsgi`` script and how to create a suitable WSGI configuration.
-
-If you use ``plone.recipe.zope2instance``, the following
-section will pull in the correct dependencies:
-
-.. code-block:: ini
-
-   [zopeinstance]
-   recipe = plone.recipe.zope2instance
-   eggs =
-       dataflake.wsgi.bjoern
-   zodb-temporary-storage = off
-   user = admin:password
-   http-address = 8080
-   wsgi = ${buildout:directory}/etc/bjoern.ini
-
-
-werkzeug
-~~~~~~~~
-`werkzeug <https://palletsprojects.com/p/werkzeug/>`_ is a WSGI library that
-contains not just a WSGI server, but also a powerful debugger. It can
-easily integrate wth Zope using a shim package called `dataflake.wsgi.werkzeug 
-<https://dataflakewsgiwerkzeug.readthedocs.io/>`_. See the `Using this package`
-section for how to integrate `werkzeug` using Zope's own ``runwsgi`` script and
-how to create a suitable WSGI configuration.
-
-If you use ``plone.recipe.zope2instance``, the following section will pull in
-the correct dependencies, after you have created a WSGI configuration file:
-
-.. code-block:: ini
-
-   [zopeinstance]
-   recipe = plone.recipe.zope2instance
-   eggs =
-       dataflake.wsgi.werkzeug
-   zodb-temporary-storage = off
-   user = admin:password
-   http-address = 8080
-   wsgi = ${buildout:directory}/etc/werkzeug.ini
-
-
 Debugging Zope applications under WSGI
 --------------------------------------
 You can debug a WSGI-based Zope application the same way you have debugged

docs/zdgbook/ObjectPublishing.rst

@@ -918,10 +918,43 @@ Record marshalling provides you with the ability to create complex
 forms. However, it is a good idea to keep your web interfaces as
 simple as possible.
 
-Please note, that records do not work with input fields of type radio as you
-might expect, as all radio fields with the same name are considered as one
-group - even if they are in different records. That means, activating one radio
-button will also deactivate all other radio buttons from the other records.
+.. note::
+
+  Records do not work with input fields of type radio as you might
+  expect, as all radio fields with the same name are considered as one
+  group - even if they are in different records. That means, activating
+  one radio button will also deactivate all other radio buttons from
+  the other records.
+
+.. attention::
+
+    When using records please note that there is a known issue when
+    you use a form, where checkboxes are used in the first "column".
+
+    As browsers leave out empty checkboxes when sending a request, the
+    **object publisher** may not be able to match checked checkboxes
+    with the correct record.
+
+    This behaviour cannot not be fixed.
+
+    If you want a checkbox as the first form field, you can work
+    around the problem by using a hidden input field.
+
+    **Code example with applied workaround**::
+
+      <form action="records_parse">
+          <p>
+          <input type="hidden" name="index.dummy:records" value="dummy" />
+          <input type="checkbox" name="index.enabled:records" value="1" checked="checked" />
+          <input type="text" name="index.name:records" value="index 1" />
+          <p>
+          <input type="hidden" name="index.dummy:records" value="dummy" />
+          <input type="checkbox" name="index.enabled:records" value="2" />
+          <input type="text" name="index.name:records" value="index 2" />
+          <p>
+          <input type="submit" name="submit" value="send" />
+      </form>
+
 
 Exceptions
 ----------
@@ -984,26 +1017,18 @@ When Zope receives a request it begins a transaction. Then it begins
 the process of traversal. Zope automatically commits the transaction
 after the published object is found and called. So normally each web
 request constitutes one transaction which Zope takes care of for you.
-See Chapter 4. for more information on transactions.
 
 If an unhandled exception is raised during the publishing process,
-Zope aborts the transaction. As detailed in Chapter
-4. Zope handles 'ConflictErrors' by re-trying the request up to three
-times. This is done with the 'zpublisher_exception_hook'.
-
-In addition, the error hook is used to return an error message to the
-user. In Zope the error hook creates error messages by calling the
-'raise_standardErrorMessage' method. This method is implemented by
-'SimpleItem.Item'. It acquires the 'standard_error_message' DTML
-object, and calls it with information about the exception.
-
-You will almost never need to override the
-'raise_standardErrorMessage' method in your own classes, since it is
-only needed to handle errors that are raised by other components. For
-most errors, you can simply catch the exceptions normally in your code
-and log error messages as needed. If you need to, you should be able
-to customize application error reporting by overriding the
-'standard_error_message' DTML object in your application.
+Zope aborts the transaction.
+When a **ConflictError** occurs, Zope retries the request up to three
+times by default. You can change that number in the **zope.conf** by
+adding a ``max_conflict_retries`` directive.
+
+.. note::
+
+  For further information on transactions please refer to chapter 6
+  `ZODB Persistent Components <https://zope.readthedocs.io/en/latest/zdgbook/ZODBPersistentComponents.html>`_.
+
 
 Manual Access to Request and Response
 -------------------------------------