Fix migration documentation

[icemac]

The main change is not to convert the ZODB in Python 2 as this is not the preferred way and the second attempt in Python 3 will fail because the ZODB is already converted.

[dataflake]

Omitting the ZODB migration step under Zope 4/Python 2 will not work for an older ZODB from Zope 2.13, which is my use case. Zope 4/Python 3 will refuse to open that ZODB because the magic marker is too old. The migration step under Zope 4/Python 2 will update the magic marker.

[dataflake]

P.S.: And the migration under Python 3 does not fail, unless something in zodbupdate has changed radically in the last 2-3 weeks.

I think I ran through that procedure about 2 weeks ago and it worked as I documented it with a ZODB from Zope 2.13.26.

[icemac]

@dataflake That's strange. The removed zodbupdate call already creates a ZODB which is Python 3 only and converted to Python 3. Because the conversion is done in Python 2 land there have to be conversion tables for str objects containing the encoding.

Calling it again from Python 3 should lead to the error that the magic marker is not compatible as the ZODB is already converted. So this step is not necessary if the conversion is already done using Python 2. The documentation of zodbupdate suggests to prefer doing the conversion in Python 3 as it allows to use the --encoding parameter instead of the mapping tables in the code.

I personally converted a Data.fs from Python 3. It was created using Zope 2.13 and later used with Zope 4 on Python 2.

[dataflake]

Have you run through that procedure recently?

I just tested it in a local sandbox with an actual production ZODB from Zope 2.13 in a sandbox using Zope 4/Python 3.7 as per your shortened instructions and the zodbupdate call fails. Looking at the line that raises the FileStorageFormatError error it reads the magic number and under Python 3 expects it to be FS30. That ZODB is FS21, though. The same code under Python 2 will expect FS21, which is why the two-step upgrade works.

I don't see how this can work at all when you run just a single zodbupdate step under Python 3.

$ bin/zodbupdate -n -v -f /opt/stage/5Ms_buildout/var/filestorage/Data.fs --convert-py3 --encoding utf-8 --encoding-fallback latin1
create storage /opt/stage/5Ms_buildout/var/filestorage/Data.fs
Traceback (most recent call last):
  File "bin/zodbupdate", line 129, in <module>
    sys.exit(zodbupdate.main.main())
  File "/opt/zope/.eggs/zodbupdate-1.2-py3.7.egg/zodbupdate/main.py", line 197, in main
    storage = ZODB.FileStorage.FileStorage(args.file)
  File "/opt/zope/.eggs/ZODB-5.5.1-py3.7.egg/ZODB/FileStorage/FileStorage.py", line 306, in __init__
    read_only=read_only,
  File "/opt/zope/.eggs/ZODB-5.5.1-py3.7.egg/ZODB/FileStorage/FileStorage.py", line 1619, in read_index
    raise FileStorageFormatError(name)
ZODB.FileStorage.FileStorage.FileStorageFormatError: /opt/stage/5Ms_buildout/var/filestorage/Data.fs

Uncovered the real issue

[dataflake]

I have spent some time on this and tried to somehow monkey-patch the expected Data.fs magic number into the code at runtime if specific conditions are met (dry run, Python 3), but have not managed a way to do it.

So for the migration instructions, the only rough outline I can suggest is the following:

UNDER PYTHON 2:

• fix your code
• run zodbverify

UNDER PYTHON 3:

• run zodbupdate (drop the dry-run option)
• run zodbverify

[dataflake]

P.S.: As I couldn't find a way to beat the ZODB code into submission by monkeypatching it during the zodbupdate run, I am guessing the only way to make it all work is to change the ZODB code itself. The way the magic number is determined, imported and used in the FileStorage code would need to change so it is possible to influence it during runtime.

[dataflake]

@icemac I have updated your branch and removed the dry-run step on Python 3, with a note added why. I also added a step to remove the ZODB index file before running the conversion, I am seeing consistent tracebacks that don't affect the conversion, but they are annoying. This may be ready for merging now.

[dataflake]

Good to go I think

[icemac]

Thank you!