Merge branch 'master' into xmlrpc

CHANGES.rst

@@ -8,60 +8,66 @@ https://zope.readthedocs.io/en/2.13/CHANGES.html
 For the change log of the alpha versions see
 https://github.com/zopefoundation/Zope/blob/4.0a6/CHANGES.rst
 
-4.0b11 (unreleased)
--------------------
+4.0.1 (unreleased)
+------------------
+
+- Nothing changed yet.
+
+
+4.0 (2019-05-10)
+----------------
 
 Fixes
 +++++
 
-- Make sure new object IDs don't clash with the views lookup mechanism
+- Make sure new object IDs don't clash with the views lookup mechanism.
   (`#591 <https://github.com/zopefoundation/Zope/issues/591>`_)
 
-- Be more careful when guessing at encoding for document template types
+- Be more careful when guessing at encoding for document template types.
 
-- Ensure a redirect path does not get URL-encoded twice
+- Ensure a redirect path does not get URL-encoded twice.
 
-- Prevent inability to log into the ZMI due to failing exception views
+- Prevent inability to log into the ZMI due to failing exception views.
 
-- Hardeded ``RESPONSE.redirect`` to deal with any unencoded or encoded input
+- Harden ``RESPONSE.redirect`` to deal with any unencoded or encoded input.
   (`#435 <https://github.com/zopefoundation/Zope/issues/435>`_)
 
-- Fix broken ``title_and_id`` behavior
+- Fix broken ``title_and_id`` behaviour.
   (`#574 <https://github.com/zopefoundation/Zope/issues/574>`_)
 
-- Fix broken ZMI DTML rendering for mixed unicode/bytes content
+- Fix broken ZMI DTML rendering for mixed unicode/bytes content.
   (`#271 <https://github.com/zopefoundation/Zope/issues/271>`_)
 
-- Fix wrong `Content-Length` set by ``App.ImageFile`` on 304 responses
+- Fix wrong `Content-Length` set by ``App.ImageFile`` on 304 responses.
   (`#513 <https://github.com/zopefoundation/Zope/issues/513>`_)
 
-- make the ZMI `Find` tab work for searching HTML tags
-  by adding support for `Tainted` strings in ``ZopeFind``
+- Make the ZMI `Find` tab work for searching HTML tags
+  by adding support for `Tainted` strings in ``ZopeFind``.
 
-- prevent ``mkwsgiinstance`` from blowing up parsing ``buildout.cfg``
+- Prevent ``mkwsgiinstance`` from blowing up parsing ``buildout.cfg``.
 
 - Fix ``ZPublisher.HTTPResponse.HTTPBaseResponse.isHTML`` for binary data on
   Python 3.
   (`#577 <https://github.com/zopefoundation/Zope/pull/577>`_)
 
-- Prevent ``FindSupport.ZopeFind`` from throwing ``UnicodeDecodeErrors``
+- Prevent ``FindSupport.ZopeFind`` from throwing ``UnicodeDecodeErrors``.
   (`#594 <https://github.com/zopefoundation/Zope/issues/594>`_)
 
 Features
 ++++++++
 
-- Add a configuration flag to show bookmarkable URLs in the ZMI
+- Add a configuration flag to show bookmarkable URLs in the ZMI.
   (`#580 <https://github.com/zopefoundation/Zope/issues/580>`_)
 
-- Add a flag for suppressing object events during file import
+- Add a flag for suppressing object events during file import.
   (`#42 <https://github.com/zopefoundation/Zope/issues/42>`_)
 
-- Add a Configuration details tab to the Control_Panel
+- Add a Configuration details tab to the Control_Panel.
 
-- Resurrect the Interfaces ZMI tab
+- Resurrect the Interfaces ZMI tab.
   (`#450 <https://github.com/zopefoundation/Zope/issues/450>`_)
 
-- Better default logging configuration for simple waitress WSGI setups
+- Better default logging configuration for simple waitress WSGI setups.
   (`#526 <https://github.com/zopefoundation/Zope/issues/526>`_)
 
 - Replace usage of ``urllib.parse.splitport`` and ``urllib.parse.splittype``
@@ -71,7 +77,11 @@ Features
 Other changes
 +++++++++++++
 
-- Changed the WSGI configuration template so those annoying waitress queue
+- Update ZODB migration documentation.
+
+- Expand the Zope 4 migration documentation.
+
+- Change the WSGI configuration template so those annoying waitress queue
   messages only go into the event log, but not onto the console.
 
 - Change naming for the generated WSGI configurations to ``zope.conf`` and
@@ -89,40 +99,39 @@ Other changes
   for examples.
   (`#562 <https://github.com/zopefoundation/Zope/issues/562>`_)
 
-- Remove hardcoded list of factories that don't want an add dialog
+- Remove hardcoded list of factories that don't want an add dialog.
   (`#540 <https://github.com/zopefoundation/Zope/issues/540>`_)
 
-- Increase link visibility in old ZMI forms
+- Increase link visibility in old ZMI forms.
   (`#530 <https://github.com/zopefoundation/Zope/issues/530>`_)
 
-- Always keep action buttons visible on the content list for large folders
+- Always keep action buttons visible on the content list for large folders.
   (`#537 <https://github.com/zopefoundation/Zope/issues/537>`_)
 
-- Make showing the ZMI modal add dialog configurable per product
+- Make showing the ZMI modal add dialog configurable per product.
   (`#535 <https://github.com/zopefoundation/Zope/issues/535>`_)
 
-- Added a few Zope 4 ZMI screenshots to the documentation
+- Added a few Zope 4 ZMI screenshots to the documentation.
   (`#378 <https://github.com/zopefoundation/Zope/issues/378>`_)
 
-- refreshed Sphinx configuration and switched to the ReadTheDocs theme
-
-- expanded the Zope 4 migration documentation
+- Refresh Sphinx configuration and switched to the ReadTheDocs theme.
 
-- renamed/moved the `Zope 2 Book` to `Zope Book`
+- Rename/move the `Zope 2 Book` to `Zope Book`.
   (`#443 <https://github.com/zopefoundation/Zope/issues/443>`_)
 
-- show item icons on ZMI `Find` tab results
+- Show item icons on ZMI `Find` tab results.
   (`#534 <https://github.com/zopefoundation/Zope/issues/534>`_)
 
-- full PEP-8 compliance
+- Full PEP-8 compliance.
 
-- fixed ZMI font rendering on macOS
+- Fix ZMI font rendering on macOS.
   (`#531 <https://github.com/zopefoundation/Zope/issues/531>`_)
 
-- provided method to get breadcrumb length to prevent ZMI errors
+- Provide a method to get breadcrumb length to prevent ZMI errors.
   (`#533 <https://github.com/zopefoundation/Zope/issues/533>`_)
 
-- add zodbupdate_rename_dict to move webdav.LockItem to OFS.LockItem
+- Add ``zodbupdate_rename_dict`` to move ``webdav.LockItem`` to
+  ``OFS.LockItem``.
   (`Products.CMFPlone#2800 <https://github.com/plone/Products.CMFPlone/issues/2800>`_)
 
 

buildout.cfg

@@ -19,7 +19,6 @@ parts =
     requirements
 sources-dir = develop
 auto-checkout =
-    Products.ZCatalog
 
 [testenv]
 PYTHONHASHSEED = random

constraints.txt

@@ -11,9 +11,9 @@ MultiMapping==4.1
 PasteDeploy==2.0.1
 Persistence==3.0
 Products.BTreeFolder2==4.2
-Products.ZCatalog==4.4
+Products.ZCatalog==5.0
 Record==3.5
-RestrictedPython==4.0b8
+RestrictedPython==4.0
 WSGIProxy2==0.4.6
 WebOb==1.8.5
 WebTest==2.0.33

docs/maintenance.rst

@@ -33,6 +33,7 @@ Steps for creating a new Zope release
 
   $ bin/checkversions versions-prod.cfg
   $ bin/checkversions versions.cfg
+  $ bin/buildout
 
 .. note::
 
@@ -49,6 +50,8 @@ Steps for creating a new Zope release
 
 - Garden the change log.
 
+- Check in the changes.
+
 - Run the tests::
 
   $ bin/tox

docs/zope4/migration/zodb.rst

@@ -2,6 +2,7 @@
 
 Migrating the ZODB
 ==================
+
 This document describes the process of migrating a ZODB created with Zope 2
 into a Zope 4 environment.
 
@@ -16,14 +17,18 @@ into a Zope 4 environment.
    :local:
 
 
-Migrating to Zope 4 under Python 2
-----------------------------------
-There are no specific ZODB-related migration steps to take when moving to a
-Python 2-based Zope 4 environment, but the warning shown above still applies.
+Python 2
+
+    There are no specific ZODB-related migration steps to take when moving to a
+    Python 2-based Zope 4 environment, but the warning shown above still applies.
+
+Python 3
+
+    Due to a string/bytes/unicode incompatibilities, additional steps are needed.
 
 
-Migrating to Zope 4 under Python 3
-----------------------------------
+Migrating the ZODB from Python 2 to 3
+-------------------------------------
 
 .. highlight:: python
 
@@ -44,13 +49,17 @@ purpose because ``str`` has a different meaning for the two
 Python versions: Under Python 2, a ``str`` is a container for
 characters with an arbitrary encoding (aka ``bytes​``). Python 3
 knows ``str`` as a text datatype which was called ``unicode``
-in Python 2. Trying to load a ``str`` object in Python 3
+in Python 2.
+
+Trying to load a ``str`` object in Python 3
 which actually contains binary data will fail. It has to be
 bytes, but ``bytes`` is an alias for ``str`` in Python 2.
 This means Python 2 replaces ``bytes`` with ``str``, making it
 impossible to give Python 3 the class it expects for binary data.
 A Python 2 ``str`` with any non-ascii characters will break, too.
 
+For more details, read the `Saltlab-Sprint notes from Harald Frisnegger <https://github.com/frisi/coredev52multipy/blob/3e440d6bd918adba3e6f2557f7281ce448a9c3cc/README.rst>`_
+
 
 The string solution
 ~~~~~~~~~~~~~~~~~~~
@@ -104,33 +113,45 @@ Migration example
 
 - Test that converted code works as expected
 
-- Prepare a Python 2 environment containing Zope 4 (latest), all relevant
-  applications for your ZODB as well as ``zodbupdate`` and ``zodb.py3migrate``
+- Prepare a Python 3 environment, containing:
+
+  - Zope 4 (latest),
+  - all relevant applications and addons for your ZODB,
+  - `zodbupdate <https://pypi.org/project/zodbupdate/>`_,
+  - `zodbverify <https://pypi.org/project/zodbverify/>`_,
+
+- Prepare a Zope configuration
 
   - Create a new Zope instance using ``mkwsgiinstance``
 
   - Update configuration in ``zope.ini`` and ``zope.conf`` to match
     previous Zope2 instance configuration.
 
-  - Run ``zodb-py3migrate-analyze Data.fs`` to determine if third party
-    products have serialized objects into the ZODB that would cause decoding
-    errors in Python 3.
-    Note: This functionality might be rolled into zodbupdate, see https://github.com/zopefoundation/zodbupdate/issues/10.
+- Migrate the database:
+
+  - Make sure no zope instance is running.
+
+  - Dry-run the migration with
+    ``zodbupdate --pack --convert-py3 --dry-run path/to/Data.fs``.
+    This may take a while.
+
+  - If no errors are shown, start the in-place migration of the ZODB
+    ``zodbupdate --pack --convert-py3 path/to/Data.fs``.
+    This may take a while.
 
-  - Run ``zodbupdate --pack --convert-py3 --file Data.fs`` to migrate the ZODB.
+- Check the migrated database:
 
-- Prepare a Python 3 environment with Zope 4 (latest) and all relevant
-  applications.
+  - Verify th ZODB by iterative loading every pickle using
+    ``zodbverify --zodbfile path/to/Data.fs``.
 
   - Start the Application using ``runwsgi etc/zope.ini``.
-
-  - ``Data.fs.index`` will be discarded at the first start, you can ignore
+    ``Data.fs.index`` will be discarded at the first start, you can ignore
     the error message telling that it cannot be read.
 
-  - Delete and recreate all ZCatalog/Index objects.
-
   - Verify that the Application works as expected.
 
+  - If there are problems with one of the ZCatalogs in the ZODB, do a clear and rebuild.
+
 In case of ``UnicodeDecodeError``
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
@@ -151,11 +172,4 @@ or `a code example in PythonScripts <https://github.com/zopefoundation/Products.
 Further reading
 ~~~~~~~~~~~~~~~
 
-This guide is adapted from several sources that contain further information
-and examples.
-
-* https://blog.gocept.com/2018/06/07/migrate-a-zope-zodb-data-fs-to-python-3/
-* https://github.com/frisi/coredev52multipy/tree/zodbupdate
-* https://github.com/plone/Products.CMFPlone/issues/2525
-* https://github.com/plone/documentation/pull/1022
-* https://github.com/zopefoundation/Zope/pull/285
+The Plone project documentation contains a section `Migrate a ZODB from Python 2.7 to Python 3 <https://github.com/plone/documentation/blob/5.2/manage/upgrading/version_specific_migration/upgrade_zodb_to_python3.rst>`_

requirements-full.txt

@@ -12,9 +12,9 @@ MultiMapping==4.1
 PasteDeploy==2.0.1
 Persistence==3.0
 Products.BTreeFolder2==4.2
-Products.ZCatalog==4.4
+Products.ZCatalog==5.0
 Record==3.5
-RestrictedPython==4.0b8
+RestrictedPython==4.0
 WSGIProxy2==0.4.6
 WebOb==1.8.5
 WebTest==2.0.33

setup.py

@@ -29,7 +29,7 @@ def _read_file(filename):
 README = _read_file('README.rst')
 CHANGES = _read_file('CHANGES.rst')
 
-version = '4.0b11.dev0'
+version = '4.0.1.dev0'
 
 
 setup(

tox.ini

@@ -36,7 +36,7 @@ commands =
     coverage combine
     coverage html -i
     coverage xml -i
-    coverage report -i --fail-under=80.5
+    coverage report -i --fail-under=81.5
 
 [testenv:isort-apply]
 basepython = python3.6

versions-prod.cfg

@@ -17,9 +17,9 @@ MultiMapping = 4.1
 PasteDeploy = 2.0.1
 Persistence = 3.0
 Products.BTreeFolder2 = 4.2
-Products.ZCatalog = 4.4
+Products.ZCatalog = 5.0
 Record = 3.5
-RestrictedPython = 4.0b8
+RestrictedPython = 4.0
 WSGIProxy2 = 0.4.6
 WebOb = 1.8.5
 WebTest = 2.0.33

versions.cfg

@@ -61,7 +61,7 @@ typing = 3.6.6
 urllib3 = 1.24.3
 virtualenv = 16.5.0
 webencodings = 0.5.1
-wheel = 0.33.2
+wheel = 0.33.3
 z3c.checkversions = 1.1
 zc.recipe.egg = 2.0.7
 zc.recipe.testrunner = 2.0.0