Fix Docker image builds CI messing up :latest, :stable, and :dev tags

[stanthewizzard]

Starting log:

archivebox  | [i] [2024-04-24 08:20:42] ArchiveBox v0.7.2: archivebox server --quick-init 0.0.0.0:8000
archivebox  |     > /data
archivebox  | 
archivebox  | [+] Initializing a new ArchiveBox v0.7.2 collection...
archivebox  | ----------------------------------------------------------------------
archivebox  | 
archivebox  | [+] Building archive folder structure...
archivebox  |     + ./archive, ./sources, ./logs...
archivebox  |     + ./ArchiveBox.conf...
archivebox  | 
archivebox  | [+] Building main SQL index and running initial migrations...
archivebox  | Traceback (most recent call last):
archivebox  |   File "/usr/local/bin/archivebox", line 8, in <module>
archivebox  |     sys.exit(main())
archivebox  |              ^^^^^^
archivebox  |   File "/app/archivebox/cli/__init__.py", line 140, in main
archivebox  |     run_subcommand(
archivebox  |   File "/app/archivebox/cli/__init__.py", line 80, in run_subcommand
archivebox  |     module.main(args=subcommand_args, stdin=stdin, pwd=pwd)    # type: ignore
archivebox  |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
archivebox  |   File "/app/archivebox/cli/archivebox_server.py", line 64, in main
archivebox  |     server(
archivebox  |   File "/app/archivebox/util.py", line 116, in typechecked_function
archivebox  |     return func(*args, **kwargs)
archivebox  |            ^^^^^^^^^^^^^^^^^^^^^
archivebox  |   File "/app/archivebox/main.py", line 1308, in server
archivebox  |     run_subcommand('init', subcommand_args=['--quick'], stdin=None, pwd=out_dir)
archivebox  |   File "/app/archivebox/cli/__init__.py", line 80, in run_subcommand
archivebox  |     module.main(args=subcommand_args, stdin=stdin, pwd=pwd)    # type: ignore
archivebox  |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
archivebox  |   File "/app/archivebox/cli/archivebox_init.py", line 43, in main
archivebox  |     init(
archivebox  |   File "/app/archivebox/util.py", line 116, in typechecked_function
archivebox  |     return func(*args, **kwargs)
archivebox  |            ^^^^^^^^^^^^^^^^^^^^^
archivebox  |   File "/app/archivebox/main.py", line 362, in init
archivebox  |     for migration_line in apply_migrations(out_dir):
archivebox  |                           ^^^^^^^^^^^^^^^^^^^^^^^^^
archivebox  |   File "/app/archivebox/util.py", line 116, in typechecked_function
archivebox  |     return func(*args, **kwargs)
archivebox  |            ^^^^^^^^^^^^^^^^^^^^^
archivebox  |   File "/app/archivebox/index/sql.py", line 145, in apply_migrations
archivebox  |     call_command("makemigrations", interactive=False, stdout=null)
archivebox  |   File "/usr/local/lib/python3.11/site-packages/django/core/management/__init__.py", line 168, in call_command
archivebox  |     return command.execute(*args, **defaults)
archivebox  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
archivebox  |   File "/usr/local/lib/python3.11/site-packages/django/core/management/base.py", line 371, in execute
archivebox  |     output = self.handle(*args, **options)
archivebox  |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
archivebox  |   File "/usr/local/lib/python3.11/site-packages/django/core/management/base.py", line 85, in wrapped
archivebox  |     res = handle_func(*args, **kwargs)
archivebox  |           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
archivebox  |   File "/usr/local/lib/python3.11/site-packages/django/core/management/commands/makemigrations.py", line 87, in handle
archivebox  |     loader = MigrationLoader(None, ignore_no_migrations=True)
archivebox  |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
archivebox  |   File "/usr/local/lib/python3.11/site-packages/django/db/migrations/loader.py", line 53, in __init__
archivebox  |     self.build_graph()
archivebox  |   File "/usr/local/lib/python3.11/site-packages/django/db/migrations/loader.py", line 210, in build_graph
archivebox  |     self.load_disk()
archivebox  |   File "/usr/local/lib/python3.11/site-packages/django/db/migrations/loader.py", line 112, in load_disk
archivebox  |     migration_module = import_module(migration_path)
archivebox  |                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
archivebox  |   File "/usr/local/lib/python3.11/importlib/__init__.py", line 126, in import_module
archivebox  |     return _bootstrap._gcd_import(name[level:], package, level)
archivebox  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
archivebox  |   File "<frozen importlib._bootstrap>", line 1204, in _gcd_import
archivebox  |   File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
archivebox  |   File "<frozen importlib._bootstrap>", line 1147, in _find_and_load_unlocked
archivebox  |   File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
archivebox  |   File "<frozen importlib._bootstrap_external>", line 940, in exec_module
archivebox  |   File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
archivebox  |   File "/app/archivebox/api/migrations/0001_initial.py", line 9, in <module>
archivebox  |     class Migration(migrations.Migration):
archivebox  |   File "/app/archivebox/api/migrations/0001_initial.py", line 22, in Migration
archivebox  |     ('token', models.CharField(default=auth.models.hex_uuid, max_length=32, unique=True)),
archivebox  |                                        ^^^^
archivebox  | NameError: name 'auth' is not defined
archivebox exited with code 1

Describe the bug

docker not starting

Steps to reproduce

docker compose

services:
    archivebox:
      image: archivebox/archivebox:latest
      container_name: archivebox
      ports:
        - 8019:8000
      volumes:
        - ./data:/data
      environment:
        - ADMIN_USERNAME=admin            # create an admin user on first run with the given user/pass combo
        - ADMIN_PASSWORD=Cgy6J5Aiw0gHsobyrrHUU (it"s a dumy pass)
        - ALLOWED_HOSTS=*                   # restrict this to only accept incoming traffic via specific domain name
        - PUBLIC_INDEX=True                 # set to False to prevent anonymous users from viewing snapshot list
        - PUBLIC_SNAPSHOTS=True             # set to False to prevent anonymous users from viewing snapshot content

ArchiveBox version

docker latest image (24/04/2024)

Thanks for help

[Snyaify]

same error after recent update :(

[stanthewizzard]

same error after recent update :(

my archive default has the issue also

ConsequentlyI made a clean one with the errro too

[ddoroshev]

Looks like it caused by this recent commit (auth never imported).

Try image: archivebox/archivebox:stable (#1391, comment)

[stanthewizzard]

archivebox/archivebox:stable
It works like a charm

thanks
what's the diff between stable and latest please ?

[rcarmo]

Came here to report this as well. I would suggest running the migrations in CI to avoid pushing broken images to :latest, since for many projects that is effectively "stable".

[sclu1034]

what's the diff between stable and latest please ?

I'd like to expand this question to all the alias tags. Because the deeper I dig, the more confusing it gets:

Like rcarmo said, the vast majority of projects I use consider :latest to mean "stable", and point it to the newest release version. So far, ArchiveBox seemed to be doing the same, as the occasional mention of a :dev tag in issues suggested that :latest was not a development build.

But now I've learned that there is also :stable, and that it's different from :latest. So apparently :latest is not a development build, but also not a release build. But checking the list of tags, :latest also isn't used by its literal name, as there is currently a nightly that's newer than :latest.

:stable itself does seem relatively straightforward and true to its name, as it's aliased to the newest versioned tag.

And then there's also :main, apparently, which by its name suggests to be "the one to be used primarily". But it doesn't actually point to :stable, but to :latest, even though the linked issue above uses :main and :stable in the same context (but does not use :latest, suggesting that :main might only sometimes be aliased to that).

And finally, :dev suggests by its name that it's an in-development build. But the current one is a month old, while both :stable and :latest are newer.

Moreover, I'd generally expect installation instructions to suggest the stable, production-ready version. But the Wiki uses :latest for the manual Docker section, and the recommended docker-compose.yml even uses :dev.

---

Ideally, the tags would be migrated to something that aligns with common practices (either ':latest == latest stable' or ':latest == any newest image'), but at the very least, there should be a page documenting all these aliases. And the installation instructions should probably be consistent in what image they use.

[Cinj216]

Came here to report this as well. I would suggest running the migrations in CI to avoid pushing broken images to :latest, since for many projects that is effectively "stable".

THIS

[pirate]

There is CI to handle all the tags as you'd expect but I broke it a couple ways with a recent big push, sorry all!

There are three issues that caused this:

• I merged a big PR into main instead of dev yesterday by accident. It triggered a broken build getting tagged with :stable, :main, and :latest last night while I was asleep. (because main was the default branch and I didn't notice the PR wasn't based off dev, I've since fixed this)
• Github Actions builds for :dev were succeeding but failing to tag them properly with the :dev tag, so that tag ended up stale. (working on the fix now)
• I just added :stable recently because some users asked for it, it's not yet handled by CI at all I just manually tagged it to be equal to the latest stable version, hence it didn't break when the others broke (will be CI-handled in the future)

---

I'm working on fixing it right now. The final state should be:

• :dev should be 1:1 with the dev branch, aka unstable and likely to break once or twice a month during busy months
• :latest should be 1:1 with the :stable and the stable branch, wont be updated except for official mainline tagged version releases like 0.7, 0.7.1, etc.

[pirate]

Ok I just fixed all the tags manually, and I'm pushing the CI fixes next.

:latest == :stable == :0.7 == :0.7.2 == :sha-315c9f3 _{== :main}

$ docker pull archivebox/archivebox:latest
$ docker run archivebox/archivebox:latest version
0.7.2
ArchiveBox v0.7.2 COMMIT_HASH=315c9f3 BUILD_TIME=2024-04-24 22:43:51 1713998631
...

$ docker pull archivebox/archivebox:stable
$ docker run archivebox/archivebox:stable version
0.7.2
ArchiveBox v0.7.2 COMMIT_HASH=315c9f3 BUILD_TIME=2024-04-24 22:43:51 1713998631
...

$ docker pull archivebox/archivebox:main
$ docker run archivebox/archivebox:main version
0.7.2
ArchiveBox v0.7.2 COMMIT_HASH=315c9f3 BUILD_TIME=2024-04-24 22:43:51 1713998631
...

$ docker pull archivebox/archivebox:0.7
$ docker run archivebox/archivebox:0.7 version
0.7.2
ArchiveBox v0.7.2 COMMIT_HASH=315c9f3 BUILD_TIME=2024-04-24 22:43:51 1713998631
...

$ docker pull archivebox/archivebox:0.7.2
$ docker run archivebox/archivebox:0.7.2 version
0.7.2
ArchiveBox v0.7.2 COMMIT_HASH=315c9f3 BUILD_TIME=2024-04-24 22:43:51 1713998631
...

These ^ are all built from the 315c9f3 commit (the 0.7.2 release commit from a few months ago), but as a side-effect of re-publishing them today, it bumped the unpinned dependency versions so it'll have the latest yt-dlp, chrome, wget, curl, etc. versions from 2024-04.

---

:dev == :sha-b4c3aa5 (for now, will updated once latest auth/import errors on dev branch are fixed)

$ docker pull archivebox/archivebox:dev
$ docker run archivebox/archivebox:dev version
0.7.3
ArchiveBox v0.7.3+editable COMMIT_HASH=b4c3aa5 BUILD_TIME=2024-03-26 22:15:46 1711491346
...

---

The docs and scripts now exclusively recommend archivebox/archivebox (aka :latest aka :stable):

• https://github.com/ArchiveBox/ArchiveBox/blob/dev/bin/setup.sh
• https://github.com/ArchiveBox/ArchiveBox/blob/dev/docker-compose.yml#L15
• https://github.com/ArchiveBox/ArchiveBox/wiki/Docker
• ... and all other places in the codebase/docs (via findandreplace all) ...

I've entirely removed references to main in the docs, the stable branch will now be known as stable and the tag will be :stable (equivalent to :latest). _{(it was only ever main because Github recommended everyone switch away from the old master convention)}

stable is just a better user-facing release channel name than main, so I'll centralize on that going forward and use it for both the Docker tag and Github branch name.

[pirate]

Out of curiosity, what setup are ya'll running that pulls new docker tags so quickly?

Are you using watchtower or some other auto-updater? And is the interval set to instant or daily?

Knowing how updates roll out and which users get them first helps me plan releases and tag updates in the future. (ArchiveBox collects no analytics, so even seemingly small info like this is very useful to me)

[rcarmo]

Watchtower, yes. Daily updates for some containers, weekly for others.

[stanthewizzard]

Watchtower running twice a day

[sclu1034]

Daily watchtower runs for me also.

[pirate]

Ok thanks for your patience ya'll! Closing this for now.

btw the big PR that broke all these tags ^ is bringing some exciting new features soon, including the new REST API:

I'll roll a real release when it's stable, so you should all get it automatically when it's out!