Replace index.json with index.sql as the main index

[cdvv7788]

Summary

After this PR is ready, the json index will not be considered the main source of truth anymore. Instead, the index.sqlite3 will replace it in that role.
The index.json will still be around, but it will only be written at the end of the processes that run. If the archive is old (no index.sqlite3 is present) running archivebox init --force will be necessary to update it to the latest version.

Changes these areas

• Bugfixes
• Feature behavior
• Command line interface
• Configuration options
• Internal architecture
• Archived data layout on disk

Roadmap Goals

This is one of the main goals of the 0.5 release.

[pirate]

From a high level this looks good, but one I think I was hoping is that we could get rid of Link entirely with this PR and use Snapshot exclusively.

Link was only needed as an in-memory representation of a Snapshot before it was added to the DB, because the DB was not the single source of truth.

Now that the DB is a single source of truth, we can do everything with Snapshot, and if we need an in-memory representation we can just do Snapshot(url='http://...', ...) without calling .save() yet.

[cdvv7788]

Thanks for the review! I will make the required changes, and start the removal of Link from archivebox.

[cdvv7788]

@pirate I think that most of the requested changes were done. There are a couple of them pending, but we can address them later. What else do you want in place to be able to close this PR?