Refactor: Remove dependency on old Django admin internal templates (+ bump Django from 3.1 to >=3.2)

[Jonasmadsen]

Currently archivebox uses django: "django>=3.1.3,<3.2" as defined in setup.py
However according to the django website:
https://www.djangoproject.com/download/
3.1.14 has been unsupported for quite a while.
This has some security implications and should be updated to django 3.2 (LTS) or newer.

I tried just updating the dependency to "django>=3.2,<3.3" but the test fails.
Unfortunately, there are breaking changes so it is not a simple version bump.

If anyone has a good idea of the scope of changes needed that would be great.
Love the project!

[pirate]

The changes are unfortunately complicated because I use some internal Django template components to build the UI (specifically the table-view action buttons), and those components were completely rewritten after 3.1.

[caj-larsson]

@pirate I know you are working on something of rewrite of much of the internals, do you have any plans on the template side of things? I decided to see how hard this would be and it seems to be no exaggeration that is quite a bit of work because of the usage of admin template/widgets.

Is it time to rebuild the foundation of the UI, if so got any thoughts on the direction?
If not, have you done any research on what needs to be done or suggestions to make this easier?

[pirate]

My inclination is to bridge the gap temporarily by including the old template files from the previous Django version's source code manually. I can create some template overrides that point to the old Django version for just the files needed, and default to all the newer files for everything else.

Then we can go through and rewrite those components/ remove the dependency on unstable Django internals entirely.

I knew this would be a pain eventually when I depended on those unstable templates but tbh they saved me a ton of dev time early on so I don't entirely regret incurring that tech debt.

In regards to the security fixes, last time I checked (2022/05) the two Django 3.1 CVEs were not hit by archivebox code paths, which is why I didn't push an urgent fix, however there may be newer CVEs since then that I haven't checked yet.

[joepie91]

last time I checked (2022/05) the two Django 3.1 CVEs were not hit by archivebox code paths

For what it's worth, nixpkgs specifies a rather longer list of relevant CVEs:

• CVE-2021-45115
• CVE-2021-45116
• CVE-2021-45452
• CVE-2022-23833
• CVE-2022-22818
• CVE-2022-28347
• CVE-2022-28346

Not all of these explicitly mark Django 3.1 as being affected; as I understand it, the reason they are added is because they appeared when 3.1 was already out of support, so it is likely that they also exist in 3.1 but nobody has ever explicitly confirmed that.

[pirate]

For now I have manually verified all of these CVEs, luckily ArchiveBox is not affected by any as we don't use the vulnerable code paths. Upgrading Django is still high on my priority list, but I'm not worried about security concerns unless other CVEs are announced that we are vulnerable to (and I have subscribed to notifications for this with Dependabot).

I also recommend everyone check out our new dedicated Security section on Github: https://github.com/ArchiveBox/ArchiveBox/security

You can subscribe to advisories and keep an eye our for new CVE announcements there too.

I understand this is causing trouble for our friends who help repackage archivebox for nix, arch, and other platforms that only offer newer Django releases. Sorry! Fixing this dependency is the next major internal work on my todo list.

[pirate]

Huge thanks @jimwins for doing this! #1388