Arch Linux AUR bug fixes backport

[srevinsaju]

Summary

Few changes required for ArchiveBox Arch Linux AUR Packages.

1. 4223801 addresses a minor Django change which created a ValueError when trying to run archivebox init on Arch Linux,

  File "/usr/lib/python3.9/site-packages/django/db/models/options.py", line 246, in _get_default_pk_class
    raise ValueError(
ValueError: Primary key 'django.db.models.UUIDField' referred by core.apps.CoreConfig.default_auto_field must subclass AutoField.

2. The vendor and other subdirectories were not included in the setup.py install because only the main package (archivebox) was included, not the subdirectories within.

Feel free to close this PR, if it seems redundant, or let me know if I can improve it.

Related issues

#738, #728

Changes these areas

• Bugfixes
• Feature behavior
• Command line interface
• Configuration options
• Internal architecture
• Snapshot data layout on disk

[lgtm-com]

This pull request introduces 1 alert when merging 9b3585b into 4770ec8 - view on LGTM.com

new alerts:

• 1 for Module is imported with 'import' and 'import from'

[pirate]

What version of Django are you running with? It only supports 3.1.x, not 3.2 or 2.x/1.x, that could be causing the error.

I don't believe it's necessary to add find_packages(), as the vendor directory is included through the Manifest.in file, and should be bundled as part of archivebox directory without needing to add the subdirectories. Can you post a screenshot of the build output files/errors when you used setup.py to assemble the package for AUR?

[srevinsaju]

What version of Django are you running with? It only supports 3.1.x, not 3.2 or 2.x/1.x, that could be causing the error.

The latest. Arch Linux always has the latest of all python package. So, using an older version of django for an arch linux package is not directly possible, or is not considered a good practice. Alternatively, if 4223801 is not merged, AUR maintainer will have to maintain this patch separately to make archivebox work with the latest version of django.

I don't believe it's necessary to add find_packages(), as the vendor directory is included through the Manifest.in file, and should be bundled as part of archivebox directory without needing to add the subdirectories.

Reproducer

git clone https://github.com/ArchiveBox/ArchiveBox.git
cd ArchiveBox
pip3 install .

Can you post a screenshot of the build output files/errors when you used setup.py to assemble the package for AUR?

Well, actually.. this is the AUR maintained by another maintainer.

Looking at the README.md I noticed another small error

pacman install archivebox

is mentioned as the mode of installation for Arch Linux, but however pacman doesn't have an subcommand install.
I do not what exactly is intended.

pacman install archivebox
error: no operation specified (use -h for help)

Installing from the AUR requires an AUR helper, or manually building the arch linux package:
Can be done in two ways:

git clone https://aur.archlinux.org/archivebox.git
cd archivebox
makepkg -si

or

yay -S archivebox

I use an AUR helper, so I did this:

Cllick to see logs

% yay -S archivebox
:: There are 2 providers available for archivebox:
:: Repository AUR
    1) archivebox 2) archivebox-git 

Enter a number (default=1): 
:: Checking for conflicts...
:: Checking for inner conflicts...
 -> 
Package conflicts found:
 -> Installing archivebox will remove: archivebox-git (archivebox)
 -> Conflicting packages will have to be confirmed manually
[Aur:1]  archivebox-0.6.2-1

  1 archivebox                               (Build Files Exist)
==> Packages to cleanBuild?
==> [N]one [A]ll [Ab]ort [I]nstalled [No]tInstalled or (1 2 3, 1-3, ^4)
==> 
:: PKGBUILD up to date, Skipping (1/1): archivebox
  1 archivebox                               (Build Files Exist)
==> Diffs to show?
==> [N]one [A]ll [Ab]ort [I]nstalled [No]tInstalled or (1 2 3, 1-3, ^4)
==> 
:: (1/1) Parsing SRCINFO: archivebox
==> Making package: archivebox 0.6.2-1 (Sat 15 May 2021 10:40:23 PM +03)
==> Retrieving sources...
  -> Found archivebox-0.6.2.tar.gz
==> Validating source files with sha256sums...
    archivebox-0.6.2.tar.gz ... Passed
==> Making package: archivebox 0.6.2-1 (Sat 15 May 2021 10:40:23 PM +03)
==> Checking runtime dependencies...
==> Checking buildtime dependencies...
==> Retrieving sources...
  -> Found archivebox-0.6.2.tar.gz
==> Validating source files with sha256sums...
    archivebox-0.6.2.tar.gz ... Passed
==> Removing existing $srcdir/ directory...
==> Extracting sources...
  -> Extracting archivebox-0.6.2.tar.gz with bsdtar
==> Sources are ready.
==> Making package: archivebox 0.6.2-1 (Sat 15 May 2021 10:40:25 PM +03)
==> Checking runtime dependencies...
==> Checking buildtime dependencies...
==> WARNING: Using existing $srcdir/ tree
==> Sources are ready.
 -> archivebox-0.6.2-1 already made -- skipping build
loading packages...
resolving dependencies...
looking for conflicting packages...
:: archivebox and archivebox-git are in conflict. Remove archivebox-git? [y/N] y

Packages (2) archivebox-git-r2630.efebd8e-1 [removal]  archivebox-0.6.2-1

Total Installed Size:  2.47 MiB
Net Upgrade Size:      0.86 MiB

:: Proceed with installation? [Y/n] Y
(1/1) checking keys in keyring                                 [##################################] 100%
(1/1) checking package integrity                               [##################################] 100%
(1/1) loading package files                                    [##################################] 100%
(1/1) checking for file conflicts                              [##################################] 100%
(2/2) checking available disk space                            [##################################] 100%
:: Processing package changes...
(1/1) removing archivebox-git                                  [##################################] 100%
(1/1) installing archivebox                                    [##################################] 100%
:: Running post-transaction hooks...
(1/1) Arming ConditionNeedsUpdate...
% archivebox
Welcome to ArchiveBox v0.6.2!

To import an existing archive (from a previous version of ArchiveBox):
    1. cd into your data dir OUTPUT_DIR (usually ArchiveBox/output) and run:
    2. archivebox init

To start a new archive:
    1. Create an empty directory, then cd into it and run:
    2. archivebox init

For more information, see the documentation here:
    https://github.com/ArchiveBox/ArchiveBox/wiki
% archivebox  init
[i] [2021-05-15 19:40:41] ArchiveBox v0.6.2: archivebox init
    > /tmp

Traceback (most recent call last):
  File "/usr/bin/archivebox", line 33, in <module>
    sys.exit(load_entry_point('archivebox==0.6.2', 'console_scripts', 'archivebox')())
  File "/usr/lib/python3.9/site-packages/archivebox/cli/__init__.py", line 140, in main
    run_subcommand(
  File "/usr/lib/python3.9/site-packages/archivebox/cli/__init__.py", line 74, in run_subcommand
    setup_django(in_memory_db=subcommand in fake_db, check_db=cmd_requires_db and not init_pending)
  File "/usr/lib/python3.9/site-packages/archivebox/config.py", line 1136, in setup_django
    django.setup()
  File "/usr/lib/python3.9/site-packages/django/__init__.py", line 24, in setup
    apps.populate(settings.INSTALLED_APPS)
  File "/usr/lib/python3.9/site-packages/django/apps/registry.py", line 114, in populate
    app_config.import_models()
  File "/usr/lib/python3.9/site-packages/django/apps/config.py", line 301, in import_models
    self.models_module = import_module(models_module_name)
  File "/usr/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 855, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/usr/lib/python3.9/site-packages/archivebox/core/models.py", line 87, in <module>
    class Snapshot(models.Model):
  File "/usr/lib/python3.9/site-packages/django/db/models/base.py", line 161, in __new__
    new_class.add_to_class(obj_name, obj)
  File "/usr/lib/python3.9/site-packages/django/db/models/base.py", line 326, in add_to_class
    value.contribute_to_class(cls, name)
  File "/usr/lib/python3.9/site-packages/django/db/models/fields/related.py", line 1635, in contribute_to_class
    self.remote_field.through = create_many_to_many_intermediary_model(self, cls)
  File "/usr/lib/python3.9/site-packages/django/db/models/fields/related.py", line 1104, in create_many_to_many_intermediary_model
    return type(name, (models.Model,), {
  File "/usr/lib/python3.9/site-packages/django/db/models/base.py", line 320, in __new__
    new_class._prepare()
  File "/usr/lib/python3.9/site-packages/django/db/models/base.py", line 333, in _prepare
    opts._prepare(cls)
  File "/usr/lib/python3.9/site-packages/django/db/models/options.py", line 285, in _prepare
    pk_class = self._get_default_pk_class()
  File "/usr/lib/python3.9/site-packages/django/db/models/options.py", line 246, in _get_default_pk_class
    raise ValueError(
ValueError: Primary key 'django.db.models.UUIDField' referred by core.apps.CoreConfig.default_auto_field must subclass AutoField.
% 

[pirate]

Thanks @srevinsaju, it may be better to just wait for the next ArchiveBox release then, I plan to add Django 3.2 support in the next version. It may not be for a month or two while though, I hope that's ok.

[grawlinson]

Any update on Django 3.2+ support?

[pirate]

No progress on 3.2 support yet. I'm aware that it's a painful wart for packaging, but unfortunately it's lower priority than most of the other release tasks and dev time is very limited right now.

[grawlinson]

No worries, I get where you're coming from. :)

[CatalinFrancu]

If at all possible, I encourage you to give this PR some love. I use Arch Linux and holding back python-django to version 3.1.* worked well for a while. However, Python itself is now up to 3.10, while python-django 3.1 is built for Python 3.9. So the list of packages to hold back keeps growing.

I am aware of solutions like Docker or the AUR package python39, but I would be happier running ArchiveBox off of vanilla Arch.

Regardless, thanks for ArchiveBox!

[CrazyPython]

Please fix this. The Docker version requires dockerd and root and is inconvenient.

[pirate]

Closing this as stale. It's also not the fix I want to implement for the underlying issue, as we need to keep UUID fields as UUIDs.