Re-running `archivebox init` loses metadata

[shepner]

This is extracting a defect commented in #556

Describe the bug

Running archivebox init is a dangerous task (data is lost when the DB is recreated)

Steps to reproduce

Run archivebox init

Software versions

archivebox/archivebox:latest as of 11/29/2020

Discussion

archivebox init should not be dangerous. Did some folders get wiped, or some entries in the database get lost when you ran it? Can you reliable reproduce it? It would be very helpful if that is the case.

Originally posted by @cdvv7788 in #556 (comment)

I was mainly referring to the Timestamp and Title fields. I havnt used tags yet so I havnt tested that. When archivebox init re-adds the snapshots to the DB, the timestamp gets overwritten and the title is re-generated. The net result is that I have a few hundred entries that claim to be added the same minute and a handful of titles that reverted back to "403 Forbidden" due to the WARC method failing. Finally, the Files indicators dont seem to be re-populating correctly in the admin panel

Files indicators:

Timestamps:

[pirate]

Thanks for reporting, this is definitely a serious bug, and we'll look into it before releasing v0.5.0 to make sure it doesn't affect anyone else.

@cdvv7788 can you investigate further if you have some time.

[cdvv7788]

Yes, I will make this my priority.

[cdvv7788]

@shepner does this happen when you are updating from v0.4 to v0.5? Or just re-running it on v0.5 triggers the issue?

[shepner]

Good question. The script has been pulling the latest version when it runs and I never before looked to see what version Im on. Apparently v0.5.0:

$ sudo docker exec -it archivebox su archivebox -c "archivebox --version"
ArchiveBox v0.5.0
Linux Linux-5.4.0-53-generic-x86_64-with-glibc2.28 x86_64

[i] Dependency versions:
 √  ARCHIVEBOX_BINARY      /usr/local/bin/archivebox                                                    v0.5.0          valid 
 √  PYTHON_BINARY          /usr/local/bin/python3.9                                                     v3.9.0          valid 
 √  DJANGO_BINARY          /usr/local/lib/python3.9/site-packages/django/bin/django-admin.py            v3.1.3          valid 
 √  CURL_BINARY            /usr/bin/curl                                                                v7.64.0         valid 
 √  WGET_BINARY            /usr/bin/wget                                                                v1.20.1         valid 
 √  NODE_BINARY            /usr/bin/node                                                                v15.3.0         valid 
 √  SINGLEFILE_BINARY      /node/node_modules/single-file/cli/single-file                               v0.1.14         valid 
 √  READABILITY_BINARY     /node/node_modules/readability-extractor/readability-extractor               v0.1.0          valid 
 √  MERCURY_BINARY         /node/node_modules/@postlight/mercury-parser/cli.js                          v1.0.0          valid 
 √  GIT_BINARY             /usr/bin/git                                                                 v2.20.1         valid 
 √  YOUTUBEDL_BINARY       /usr/local/bin/youtube-dl                                                    v2020.11.26     valid 
 √  CHROME_BINARY          /usr/bin/chromium                                                            v83.0.4103.116  valid 

[i] Source-code locations:
 √  PACKAGE_DIR            /app/archivebox                                                              21 files        valid 
 √  TEMPLATES_DIR          /app/archivebox/themes/legacy                                                7 files         valid 

[i] Secrets locations:
 -  CHROME_USER_DATA_DIR   None                                                                         -               disabled 
 -  COOKIES_FILE           None                                                                         -               disabled 

[i] Data locations:
 √  OUTPUT_DIR             /data                                                                        7 files         valid 
 √  SOURCES_DIR            /data/sources                                                                4 files         valid 
 √  LOGS_DIR               /data/logs                                                                   0 files         valid 
 √  ARCHIVE_DIR            /data/archive                                                                407 files       valid 
 √  CONFIG_FILE            /data/ArchiveBox.conf                                                        193.0 Bytes     valid 
 √  SQL_INDEX              /data/index.sqlite3                                                          412.0 KB        valid 

Just ran sudo docker exec -it archivebox su archivebox -c "archivebox init" again. I dont see any differences in the times and etc so perhaps I started with v0.4.x and at some point got switched over to v0.5.0? I havnt been using this very long so Im guessing it just occurred a few days ago?

On a related note, I suggest that ongoing archivebox/archivebox:latest should point to your latest stable version. Ie, from the documentation, I was expecting to see v0.4.21 as its what is listed on the releases page. This might help alleviate later confusion/issues as more adopt the Docker container.

[pirate]

Aha that explains it, v0.5.0 is an unreleased working branch, still full of bugs. Definitely downgrade your image back to v0.4.21.

Unfortunately I haven't figured out an easy for us to control the :latest tag because Docker hub assigns it to whatever the last uploaded image is.

We need to be able to build and share images of our working branches (which get pushed more often than stable releases), do you know of a good way to do that without bumping latest each time we push some WIP branch? No worries if not, I'll probably do some research on my own, and worst case we can put our WIP code on a separate docker hub acct entirely.

[shepner]

This part of the conversation prolly should be moved elsewhere but:

Note that I do not work as a developer, my own projects/repos arent to the point where anyone (myself included) particularly care about releases, and professionally I'm a bit removed from that side of the process. That said, Im pretty sure having :latest assigned to the, well, latest image is to be expected. :)

I suspect there are several ways to go about this but I dont gone through the nuances of each. For example:

• One approach is to have a well structured set of branches. The dev branch being where you do your day to day work, a series of release branches, and master being the current stable. Im guessing that is the intent for for this project.
• Another seems doesnt seem to care much about branches at all, and somehow relies on git's release mechanism to trigger scripted automatons so docker build only sees that release version.
  I havnt messed with Jenkins near enough but Im guessing the latter is closer to the ideal, if for no other reason, because it doesnt require code merges or the dev team remembering what branch to use.

In thinking about it, that latter approach could would mean docker hub would only receive the images pushed to it and anyone wanting to play can just run their own build locally. The hard part is figuring out how to configure the Jenkins job (Jenkins itself is easy enough to get running)

[pirate]

We already have Github Actions CI (we don't use Jenkins) set up to auto-push built images on every commit, so it would be easy for us to limit that only to master and work on the dev branch instead.

I think that's probably the way to go, it's what we had before and it worked well, I was just hoping we could autobuild all commits because it makes for easier development if you don't have to build locally, but it seems to be causing more trouble than it's worth with the constant :latest bumping.

[cdvv7788]

@pirate we discussed this approach before. It looks like the way to go...we should do that. If you want to have a build for everything, let's setup another dockerhub repository.

[pirate]

Done b186e98 (we no longer push every commit to docker hub as :latest images, only the full releases)

[cdvv7788]

@shepner do you have the original archive (a copy before you ran archivebox init)?

[shepner]

Sorry no. TBH, it isnt a big deal on my part. Im still getting a feel for how this works and if/how I want to use it. That latter part is prolly best suited for a new thread

[pirate]

I suspect the original issue here is being caused by this #640

[pirate]

I believe these issues are all fixed in the latest versions, if you're still having any issues with corruption please comment back here and we'll investigate.