Docker scheduling docs questions

[thelfensdrfer]

Hi, I have a few questions about this wiki page: https://github.com/ArchiveBox/ArchiveBox/wiki/Scheduled-Archiving#docker-usage

1. I read somewhere, but can't find it anymore, that you shouldn't use docker-compose exec archivebox_scheduler ... but docker-compose run archivebox_scheduler .... Is that true, and if so, why?
2. As far as I understand, without --overwrite one URL is only archived once, no matter if I also provide e.g. a --every=day argument. At least that's how I understand this example: https://github.com/ArchiveBox/ArchiveBox/wiki/Scheduled-Archiving#example-archive-a-reddit-subreddit-and-discussions-for-every-post-once-a-week
   But if that's true, the following examples with Reddit, hacker news, the RSS feed, etc. are all missing the overwrite flag.
3. Since now I also added all my URLs (the content did not change) without the overwrite flag and I suspect that's the reason I didn't see any new archive results in the web UI. But I can see that the scheduler worked when reading the docker logs. In the web UI the logs do not contain any new information since the initial archive result 5 weeks ago. For me it always seemed like it didn't run properly until I checked the docker logs.

[pirate]

I read somewhere, but can't find it anymore, that you shouldn't use docker-compose exec archivebox_scheduler ... but docker-compose run archivebox_scheduler .... Is that true, and if so, why?

Because docker compose run goes through the ./bin/docker_entrypoint.sh, which is needed to do a bunch of filesystem checks on startup and drop down to the archivebox user with the correct permissions. docker compose exec/docker exec skip the entrypoint entirely and don't use the right permissions.

(side note: docker compose without the - is much faster and has a nicer-looking CLI than docker-compose these days)

---

As far as I understand, without --overwrite one URL is only archived once, no matter if I also provide e.g. a --every=day argument. At least that's how I understand this example: ...

That is correct, but the intention is to use it with --depth=1 so that all new URLs found within the page are archived, even if the root URL is not. This is perfect for archiving HN, subreddits, etc. any kind of feed. I don't need a snapshot of the HN homepage every day, what I really want is a snapshot of all the articles and comment sections linked to from the homepage since thats where the real content is.

---

In the web UI the logs do not contain any new information since the initial archive result 5 weeks ago [...] For me it always seemed like it didn't run properly until I checked the docker logs.

This could be caused by a number of things, will need more info to debug. Please share your full docker-compose.yml, and the output of docker compose run archivebox version and docker compose run archivebox schedule --show.

[thelfensdrfer]

$ docker compose ps
NAME                                IMAGE                         COMMAND                  SERVICE                CREATED        STATUS                    PORTS
archivebox-archivebox-1             archivebox/archivebox:dev     "dumb-init -- /app/b…"   archivebox             19 hours ago   Up 19 hours (unhealthy)   8000/tcp
archivebox-archivebox_scheduler-1   archivebox/archivebox:dev     "dumb-init -- /app/b…"   archivebox_scheduler   19 hours ago   Up 19 hours (unhealthy)   8000/tcp
archivebox-sonic-1                  valeriansaliou/sonic:latest   "sonic -c /etc/sonic…"   sonic                  7 weeks ago    Up 19 hours               1491/tcp

It's only unhealthy because I have not added localhost to the ALLOWED_HOSTS config yet:

archivebox-1  | Invalid HTTP_HOST header: 'localhost:8000'. You may need to add 'localhost' to ALLOWED_HOSTS.
archivebox-1  | Bad Request: /health/
archivebox-1  | "GET /health/ HTTP/1.1" 400 143
archivebox-1  | "GET / HTTP/1.1" 302 0
archivebox-1  | "GET /admin/login/?next=/ HTTP/1.1" 200 11702

And are you sure the URLs you've scheduled to archive are actually changing between each run?

As I've said in the initial question, the content does not change. But I have no clue if everything works as expected since the logs have not changed since the initial archive. I understand why the date of the latest snapshot did not change, there were no new snapshots since nothing changed, but why is there no new log entry? Something like: "Tried to archive https://www.example.com, but nothing changed -> skipped". Or there should be a log entry but my setup isn't working. My problem is that I can't figure out which of both options is correct.

[pirate]

Ah if the content is not changing then that explains it, it wont create any new entries on the Log DB / ADMIN UI if no new URLs are added. It ignores any URLs that are already in the index, so nothing will show up if it finds only previously-seen URLs.

It should still output some CLI stdout/stderr every time it runs though, can you share the contents of ./data/logs/schedule.log.

[thelfensdrfer]

Yeah, I posted the docker logs above. Every week, like I wanted to, the scheduler container runs Running 6 ArchiveBox jobs in foreground task scheduler.... And that's great. But why is that not reflected in the log in the Django admin panel?

I am happy to provide a pull request if this is a bug, I am only trying to understand the reasoning behind this. Perhaps I am missing something. But from what I can tell right now, "logs" can be combined with "snapshots" (at least in the navigation bar), because without a snapshot there is no new log entry.

[pirate]

But why is that not reflected in the log in the Django admin panel?

There are only 2 main models in ArchiveBox currently:

• Snapshot 1:1 with each URL added to ArchiveBox, not created unless a new unique URL is added
• ArchiveResult (aka Log in the UI), several of these exist underneath each snapshot, one for each new extractor output that is generated inside the snapshot (e.g. one for favicon, one for headers, one for singlefile, etc. see screenshot below)

If you re-add URLs that archivebox has already seen previously, it doesn't create any new Snapshots, so no new ArchiveResults are generated, so nothing new appears in the UI. There is no record* of imports that produce no new Snapshot or ArchiveResult rows.

* aside from ./data/sources and ./data/logs

Note the ./data/schedule.log will have much more detail than the output of docker compose logs archivebox_scheduler.

[thelfensdrfer]

Got it, thanks!

The only question that remains for me is, what happens if the contents of the URL changes? For example I replace <h1>Hello</h1> with <h1>Hello Update</h1>?

As I understand it now*, nothing is saved because the URL is already added. I guess this is when the "re-snapshot" feature comes into place but you have to configure it to store multiple snapshots actively. So what is the purpose of the --every argument when you provide --depth=0?

* it took me a long time to figure out that changes are not stored automatically every. e.g. day, because the first paragraph on the website makes it sound like archivebox is something like an open-source replacement for archive.org. But on archive.org you have multiple snapshots for different dates.

And please don't get me wrong, I like archivebox and appreciate the time you put into it. I want to give the perspective of a first-time user.

Edit: And yes, schedule.log shows that the scheduler is running but no new URLs are found and therefore are not stored.

Edit2: And now I read #179 and understand this feature wasn't planned initially and is simply a planned feature.