Django version 1.X is not compatible with Python 3.7

[brookst]

TL;DR Paperless needs to migrate to Django 2.X

I just just noticed my Paperless instance was throwing syntax errors. E.g.:

  File "/usr/lib/python3.6/site-packages/django/contrib/admin/widgets.py", line 152
    '%s=%s' % (k, v) for k, v in params.items(),
    ^
SyntaxError: Generator expression must be parenthesized

It's running via WSGI on a bare Arch Linux distribution. I noticed the update to Python 3.7 but didn't check all my services were running afterwards.

I removed all the Python 3.6 packages but reinstalling them for Python 3.7 reproduces the same error since Paperless requires exactly django==1.11.13.

It seems that Django 1.X does not support Python 3.7 at all. There's some discussion of fixes to 1.X here, but it seems like those fixes would have to be vendored in. The only real option seems to be migration to Django 2.X.

Does anyone know how big a task it'd be to do that migration? I might be willing to give it a go if no-one familiar with the code is in a position to do so.

[danielquinn]

Ooh, yeah that's a good point. I don't have a lot of time right now, so an upgrade won't likely happen until I return from PyCon UK next month. In the short term, I'd suggest that you what I've done for my other Python 3.6-based projects and install 3.6 from the AUR.

To be clear though, the intent was always to keep Paperless up to date with the latest Django version, it's just that 2.x was sufficiently different that upgrading is more work than usual so I've been putting it off.

If you feel like doing the work yourself though, I'm happy to field a pull request :-) In the short term though, installing that AUR version will mean that 3.6 will be installed in parallel to 3.7 on your system and then you can just create your virtualenv the way you usually do: pipenv --python 3.6.

[dadosch]

So, I have done some small modifications (see #390) which fixes some obvious errors, but I didn't deploy it fully neither did I any thorough tests. So feedback and testing is very appreciated.

[brookst]

I also took a stab at this and had to do a few more things than in #390. My changes are: master...brookst:django-upgrade

The extra changes are

• MIDDLEWARE_CLASSES in src/paperless/settings.py needs to be renamed to just MIDDLEWARE
• django.contrib.auth.middleware.SessionAuthenticationMiddleware is redundant and removed now
• The version of pluggy in requirements.txt needs to be 0.7.1 (0.6.0 is incompatible with pytest)
• In src/documents/migrations/0011_auto_20160303_1929.py the migration needs an atomic = False line, or else I get: django.db.utils.NotSupportedError: Renaming the 'documents_sender' table while in a transaction is not supported on SQLite because it would break referential integrity. I'm not entirely sure what this is doing, but the tests pass 🤷‍♂️
• I set the on_delete method for the Reminders to PROTECT, but I'm not sure exactly what happens in that case or what the intention would be there. I think that might need some bikeshedding

py.test --cov --rootdir=src src works on my machine as does src/manage.py test, and I can stand up a test instance ok.

@dadosch: Do you want to pull in those extra changes from my fork? I'm not sure how important the atomic and on_delete issues are, so maybe that needs some discussion first.

[dadosch]

Do you want to pull in those extra changes from my fork?

@brookst Yes, I just did. Also for me, tests are running fine. Here is some reading about on_delete

[brookst]

Ah, one more issue. I copied over my database and media to test out the reminder things and there was something obviously wrong. The admin interface has messed up html encoding and so displays many tags literally. I made a fix on top of the latest commit @dadosch included in #390. It's on my fork here: brookst@35ff657

With that, everything I can see renders correctly so hopefully this was only an issue in that src/documents/admin.py file.

I also tried deleting a document with a reminder attached and get a reasonable looking page saying:

Cannot delete document

Deleting the selected document would require deleting the following protected related objects:

• Reminder: Reminder object (1)

That seems better than silently dropping the reminders or breaking the linkage to the corresponding document.

[wmader]

@brookst I tested paperless as of your commit on my 220-german-documents data base, and everything renders properly. Before your commit, thumbnails showed html links as plain text.

[danielquinn]

Ok, I've just merged #390 and added a few changes myself to fix the API. From the looks of things, everything is working in Python 3.6 & 3.7 so I'm closing this ticket now. Thanks to all of you on this!