Bugfix: django.db.utils.IntegrityError: UNIQUE constraint failed: core_snapshot.timestamp #412

drpfenderson · 2020-07-31T20:33:18Z

Describe the bug

Y'all helped me with upgrading my super old archive to the django branch before official 0.4.9 release. I recently upgraded to the newest version, so I could start adding links. archivebox said I had to re-init. archivebox init gives me following error, and will not let me add new links.

django.db.utils.IntegrityError: UNIQUE constraint failed: core_snapshot.timestamp

Full log/error below.

Steps to reproduce

git checkout master to switch from django branch.
git pull origin master to pull new release.
pip install -e . (also tried with pip uninstall archivebox && pip install .)
Navigate to archivebox-output directory.
Run archivebox init.
error.

Screenshots or log output

[i] [2020-07-31 17:34:44] ArchiveBox v0.4.9: archivebox init
    > /.archivebox-output/archive-working

[*] Updating existing ArchiveBox collection in this folder...
    /.archivebox-output/archive-working
------------------------------------------------------------------

[*] Verifying archive folder structure...
    √ /.archivebox-output/archive-working/sources
    √ /.archivebox-output/archive-working/archive
    √ /.archivebox-output/archive-working/logs
    √ /.archivebox-output/archive-working/ArchiveBox.conf

[*] Verifying main SQL index and running migrations...
    √ /.archivebox-output/archive-working/index.sqlite3

    Operations to perform:
      Apply all migrations: admin, auth, contenttypes, core, sessions
    Running migrations:
    Applying core.0005_auto_20200728_0326... OK

[*] Collecting links from any existing indexes and archive folders...
    √ Loaded 1376 links from existing main index.
    √ Added 347 orphaned links from existing archive directories.
    ! Skipped adding 239 invalid link data directories.

    X /* SNIP A BUNCH OF BROKEN ARCHIVES /*

    Hint: For more information about the link data directories that were skipped, run:
        archivebox status
        archivebox list --status=invalid

[*] [2020-07-31 18:01:50] Writing 1723 links to main index...
Traceback (most recent call last):
  File "/home/USERNAME/.local/lib/python3.7/site-packages/django/db/models/query.py", line 575, in update_or_create
    obj = self.select_for_update().get(**kwargs)
  File "/home/USERNAME/.local/lib/python3.7/site-packages/django/db/models/query.py", line 417, in get
    self.model._meta.object_name
core.models.DoesNotExist: Snapshot matching query does not exist.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/USERNAME/.local/lib/python3.7/site-packages/django/db/backends/utils.py", line 86, in _execute
    return self.cursor.execute(sql, params)
  File "/home/USERNAME/.local/lib/python3.7/site-packages/django/db/backends/sqlite3/base.py", line 396, in execute
    return Database.Cursor.execute(self, query, params)
sqlite3.IntegrityError: UNIQUE constraint failed: core_snapshot.timestamp

The above exception was the direct cause of the following exception:
Traceback (most recent call last):
  File "/home/USERNAME/.local/bin/archivebox", line 33, in <module>
    sys.exit(load_entry_point('archivebox', 'console_scripts', 'archivebox')())
  File "/home/USERNAME/datahoard/ArchiveBox/archivebox/cli/__init__.py", line 126, in main
    pwd=pwd or OUTPUT_DIR,
  File "/home/USERNAME/datahoard/ArchiveBox/archivebox/cli/__init__.py", line 62, in run_subcommand
    module.main(args=subcommand_args, stdin=stdin, pwd=pwd)    # type: ignore
  File "/home/USERNAME/datahoard/ArchiveBox/archivebox/cli/archivebox_init.py", line 35, in main
    out_dir=pwd or OUTPUT_DIR,
  File "/home/USERNAME/datahoard/ArchiveBox/archivebox/util.py", line 109, in typechecked_function
    return func(*args, **kwargs)
  File "/home/USERNAME/datahoard/ArchiveBox/archivebox/main.py", line 369, in init
    write_main_index(list(all_links.values()), out_dir=out_dir)
  File "/home/USERNAME/datahoard/ArchiveBox/archivebox/util.py", line 109, in typechecked_function
    return func(*args, **kwargs)
  File "/home/USERNAME/datahoard/ArchiveBox/archivebox/index/__init__.py", line 235, in write_main_index
    write_sql_main_index(links, out_dir=out_dir)
  File "/home/USERNAME/datahoard/ArchiveBox/archivebox/util.py", line 109, in typechecked_function
    return func(*args, **kwargs)
  File "/home/USERNAME/datahoard/ArchiveBox/archivebox/index/sql.py", line 42, in write_sql_main_index
    Snapshot.objects.update_or_create(url=link.url, defaults=info)
  File "/home/USERNAME/.local/lib/python3.7/site-packages/django/db/models/manager.py", line 82, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/home/USERNAME/.local/lib/python3.7/site-packages/django/db/models/query.py", line 580, in update_or_create
    obj, created = self._create_object_from_params(kwargs, params, lock=True)
  File "/home/USERNAME/.local/lib/python3.7/site-packages/django/db/models/query.py", line 604, in _create_object_from_params
    raise e
  File "/home/USERNAME/.local/lib/python3.7/site-packages/django/db/models/query.py", line 596, in _create_object_from_params
    obj = self.create(**params)
  File "/home/USERNAME/.local/lib/python3.7/site-packages/django/db/models/query.py", line 433, in create
    obj.save(force_insert=True, using=self.db)
  File "/home/USERNAME/.local/lib/python3.7/site-packages/django/db/models/base.py", line 746, in save
  force_update=force_update, update_fields=update_fields)
  File "/home/USERNAME/.local/lib/python3.7/site-packages/django/db/models/base.py", line 784, in save_base
    force_update, using, update_fields,
  File "/home/USERNAME/.local/lib/python3.7/site-packages/django/db/models/base.py", line 887, in _save_table
    results = self._do_insert(cls._base_manager, using, fields, returning_fields, raw)
  File "/home/USERNAME/.local/lib/python3.7/site-packages/django/db/models/base.py", line 926, in _do_insert
    using=using, raw=raw,
  File "/home/USERNAME/.local/lib/python3.7/site-packages/django/db/models/manager.py", line 82, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/home/USERNAME/.local/lib/python3.7/site-packages/django/db/models/query.py", line 1204, in _insert
    return query.get_compiler(using=using).execute_sql(returning_fields)
  File "/home/USERNAME/.local/lib/python3.7/site-packages/django/db/models/sql/compiler.py", line 1392, in execute_sql
    cursor.execute(sql, params)
  File "/home/USERNAME/.local/lib/python3.7/site-packages/django/db/backends/utils.py", line 68, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "/home/USERNAME/.local/lib/python3.7/site-packages/django/db/backends/utils.py", line 77, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/home/USERNAME/.local/lib/python3.7/site-packages/django/db/backends/utils.py", line 86, in _execute
    return self.cursor.execute(sql, params)
  File "/home/USERNAME/.local/lib/python3.7/site-packages/django/db/utils.py", line 90, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/home/USERNAME/.local/lib/python3.7/site-packages/django/db/backends/utils.py", line 86, in _execute
    return self.cursor.execute(sql, params)
  File "/home/USERNAME/.local/lib/python3.7/site-packages/django/db/backends/sqlite3/base.py", line 396, in execute
    return Database.Cursor.execute(self, query, params)
django.db.utils.IntegrityError: UNIQUE constraint failed: core_snapshot.timestamp

Software versions

OS: Ubuntu 18.04
ArchiveBox version: 0.4.9 (0ac4e12)
Python version: Python 3.7.8

The text was updated successfully, but these errors were encountered:

karlicoss · 2020-08-11T19:03:41Z

Happens for me as well. Archivebox version: v0.4.13 (image from Docker hub).

I experimented a bit and managed to consistently reproduce. I suspect the urls that have a suffix in the timestamp are causing it.

Create a new (empty) archive directory, put it in the compose file and initialise

docker-compose run --rm archivebox init

Archive few URLs

input:

 https://beepb00p.xyz/mypy-error-handling.html
 https://beepb00p.xyz/promnesia.html
 https://beepb00p.xyz/hpi.html

First archiving:

 docker-compose run --rm archivebox add <input

Goes well:

 dco run --rm archivebox add <input
 [i] [2020-08-11 18:46:49] ArchiveBox v0.4.13: archivebox add < /dev/stdin
     > /data
 
 [+] [2020-08-11 18:46:49] Adding 3 links to index (crawl depth=0)...
     > Saved verbatim input to sources/1597171609-import.txt
     > Parsed 3 URLs from input (Plain Text)
     > Found 3 new URLs not already in index
 
 [*] [2020-08-11 18:46:49] Writing 3 links to main index...
     √ /data/index.sqlite3
     √ /data/index.json
     √ /data/index.html
 
 [▶] [2020-08-11 18:46:50] Collecting content for 3 Snapshots in archive...
 
 [+] [2020-08-11 18:46:50] "beepb00p.xyz/promnesia.html"
     https://beepb00p.xyz/promnesia.html
     > ./archive/1597171609
     ...
 
 [+] [2020-08-11 18:46:59] "beepb00p.xyz/mypy-error-handling.html"
     https://beepb00p.xyz/mypy-error-handling.html
     > ./archive/1597171609.0
     ...
 
 [+] [2020-08-11 18:47:10] "beepb00p.xyz/hpi.html"
     https://beepb00p.xyz/hpi.html
     > ./archive/1597171609.1
     ...
 
 [√] [2020-08-11 18:47:20] Update of 3 pages complete (30.33 sec)
     - 0 links skipped
     - 3 links updated
     - 0 links had errors
 
     Hint: To view your archive index, open:
         /data/index.html
     Or run the built-in webserver:
         archivebox server
 
 [*] [2020-08-11 18:47:20] Writing 3 links to main index...
     √ /data/index.sqlite3
     √ /data/index.json
     √ /data/index.html

Now if you rerun the same command, it works well too

 dco run --rm archivebox add <input                              
 [i] [2020-08-11 18:51:29] ArchiveBox v0.4.13: archivebox add < /dev/stdin
     > /data
 
 [+] [2020-08-11 18:51:29] Adding 3 links to index (crawl depth=0)...
     > Saved verbatim input to sources/1597171889-import.txt
     > Parsed 3 URLs from input (Plain Text)
     > Found 0 new URLs not already in index
 
 [*] [2020-08-11 18:51:30] Writing 3 links to main index...
     √ /data/index.sqlite3
     √ /data/index.json
     √ /data/index.html

As expected, just says everything is already in the index

Now try running against on of the urls that has a dot in the timestamp (with a suffix)

 dco run --rm archivebox add https://beepb00p.xyz/hpi.html 
 # results in IntegrityError
 dco run --rm archivebox add https://beepb00p.xyz/mypy-error-handling.html
 # results in IntegrityError

Interesting enough, running against https://beepb00p.xyz/promnesia.html, that has the timestamp 1597171609 works fine and as expected just says it's already in the index.

Now if you try to add a completely different set of links, it works fine again:

 $ dco run --rm archivebox add <input2  

 [i] [2020-08-11 18:59:21] ArchiveBox v0.4.13: archivebox add < /dev/stdin
     > /data
 
 [+] [2020-08-11 18:59:21] Adding 2 links to index (crawl depth=0)...
     > Saved verbatim input to sources/1597172361-import.txt
     > Parsed 2 URLs from input (Plain Text)
     > Found 2 new URLs not already in index
 
 [*] [2020-08-11 18:59:21] Writing 5 links to main index...
     √ /data/index.sqlite3
     √ /data/index.json
     √ /data/index.html
 
 [▶] [2020-08-11 18:59:22] Collecting content for 2 Snapshots in archive...
 
 [+] [2020-08-11 18:59:22] "blog.sigfpe.com/2008/02/what-is-topology.html"
     http://blog.sigfpe.com/2008/02/what-is-topology.html
     > ./archive/1597172361
 ...
 
 [+] [2020-08-11 18:59:35] "blog.sigfpe.com/2006/11/yoneda-lemma.html"
     http://blog.sigfpe.com/2006/11/yoneda-lemma.html
     > ./archive/1597172361.0
 ...

And again, if you try to add http://blog.sigfpe.com/2008/02/what-is-topology.html, it works, if you try http://blog.sigfpe.com/2006/11/yoneda-lemma.html it fails.

pirate · 2020-08-11T20:03:57Z

Very helpful @karlicoss! This is high on our priority list of things to fix.

I'll check in with an update once we've started working on this. I suspect it's a relatively simple bug in the timestamp deduping code, most of the work will be QA and testing to make sure we don't introduce any regressions while we fix it.

For context, timestamp deduping has been one of the most brittle parts of ArchiveBox in the past years, and we already have plans to remove the need for it in a refactoring in the next major version.

jrruethe · 2020-08-15T01:12:01Z

I unfortunately ran into this issue as well. From my testing, I agree with @karlicoss and his assessment that it is related to the timestamp suffixes. I am trying to pin it down further than that, I'll reply if I figure anything out.

Thanks

coisnepe · 2020-08-16T16:08:02Z

it works fine again

Nothing works for me anymore, sadly... Attempting to add any link, whether completely new or already archived, results in django.db.utils.IntegrityError.
What's the least dangerous way to fix it (temporarily disabling the unique constraint, deleting one/some archives etc...)?

apkallum · 2020-08-17T18:06:45Z

Hello @coisnepe @jrruethe @karlicoss @drpfenderson & everyone else, would you mind testing my master branch with a fix here? https://github.com/apkallum/ArchiveBox

drpfenderson · 2020-08-17T20:01:55Z

@apkallum - Using your build, it gets a bit further. Modifies a few entries, and then gives following error:

Traceback (most recent call last):
  File "/home/USERNAME/datahoard/ArchiveBoxTimefix/archivebox/index/sql.py", line 43, in write_sql_main_index
    snapshot = Snapshot.objects.get(url=link.url)
  File "/home/USERNAME/.local/lib/python3.8/site-packages/django/db/models/manager.py", line 82, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/home/USERNAME/.local/lib/python3.8/site-packages/django/db/models/query.py", line 415, in get
    raise self.model.DoesNotExist(
core.models.DoesNotExist: Snapshot matching query does not exist.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/USERNAME/.local/lib/python3.8/site-packages/django/db/backends/utils.py", line 86, in _execute
    return self.cursor.execute(sql, params)
  File "/home/USERNAME/.local/lib/python3.8/site-packages/django/db/backends/sqlite3/base.py", line 396, in execute
    return Database.Cursor.execute(self, query, params)
sqlite3.IntegrityError: UNIQUE constraint failed: core_snapshot.timestamp

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/USERNAME/.local/bin/archivebox", line 11, in <module>
    load_entry_point('archivebox', 'console_scripts', 'archivebox')()
  File "/home/USERNAME/datahoard/ArchiveBoxTimefix/archivebox/cli/__init__.py", line 122, in main
    run_subcommand(
  File "/home/USERNAME/datahoard/ArchiveBoxTimefix/archivebox/cli/__init__.py", line 62, in run_subcommand
    module.main(args=subcommand_args, stdin=stdin, pwd=pwd)    # type: ignore
  File "/home/USERNAME/datahoard/ArchiveBoxTimefix/archivebox/cli/archivebox_init.py", line 33, in main
    init(
  File "/home/USERNAME/datahoard/ArchiveBoxTimefix/archivebox/util.py", line 111, in typechecked_function
    return func(*args, **kwargs)
  File "/home/USERNAME/datahoard/ArchiveBoxTimefix/archivebox/main.py", line 376, in init
    write_main_index(list(all_links.values()), out_dir=out_dir)
  File "/home/USERNAME/datahoard/ArchiveBoxTimefix/archivebox/util.py", line 111, in typechecked_function
    return func(*args, **kwargs)
  File "/home/USERNAME/datahoard/ArchiveBoxTimefix/archivebox/index/__init__.py", line 235, in write_main_index
    write_sql_main_index(links, out_dir=out_dir)
  File "/home/USERNAME/datahoard/ArchiveBoxTimefix/archivebox/util.py", line 111, in typechecked_function
    return func(*args, **kwargs)
  File "/home/USERNAME/datahoard/ArchiveBoxTimefix/archivebox/index/sql.py", line 51, in write_sql_main_index
    snapshot.save()
  File "/home/USERNAME/.local/lib/python3.8/site-packages/django/db/models/base.py", line 745, in save
    self.save_base(using=using, force_insert=force_insert,
  File "/home/USERNAME/.local/lib/python3.8/site-packages/django/db/models/base.py", line 782, in save_base
    updated = self._save_table(
  File "/home/USERNAME/.local/lib/python3.8/site-packages/django/db/models/base.py", line 887, in _save_table
    results = self._do_insert(cls._base_manager, using, fields, returning_fields, raw)
  File "/home/USERNAME/.local/lib/python3.8/site-packages/django/db/models/base.py", line 924, in _do_insert
    return manager._insert(
  File "/home/USERNAME/.local/lib/python3.8/site-packages/django/db/models/manager.py", line 82, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/home/USERNAME/.local/lib/python3.8/site-packages/django/db/models/query.py", line 1204, in _insert
    return query.get_compiler(using=using).execute_sql(returning_fields)
  File "/home/USERNAME/.local/lib/python3.8/site-packages/django/db/models/sql/compiler.py", line 1392, in execute_sql
    cursor.execute(sql, params)
  File "/home/USERNAME/.local/lib/python3.8/site-packages/django/db/backends/utils.py", line 68, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "/home/USERNAME/.local/lib/python3.8/site-packages/django/db/backends/utils.py", line 77, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/home/USERNAME/.local/lib/python3.8/site-packages/django/db/backends/utils.py", line 86, in _execute
    return self.cursor.execute(sql, params)
  File "/home/USERNAME/.local/lib/python3.8/site-packages/django/db/utils.py", line 90, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/home/USERNAME/.local/lib/python3.8/site-packages/django/db/backends/utils.py", line 86, in _execute
    return self.cursor.execute(sql, params)
  File "/home/USERNAME/.local/lib/python3.8/site-packages/django/db/backends/sqlite3/base.py", line 396, in execute
    return Database.Cursor.execute(self, query, params)
django.db.utils.IntegrityError: UNIQUE constraint failed: core_snapshot.timestamp

EDIT: To be clear, this is using archivebox init in the main archive directory.

EDIT 2: Oops. Realized I had switched to Python 3.8 for another project and forgot to update-alternatives. Running archivebox init with Python 3.7, with apkallum's branch, gives me essentially same error.

pirate · 2020-08-18T13:58:27Z

Give the latest master a try:

pip install --upgrade archivebox
# or if you use docker
docker pull nikisweeting/archivebox

drpfenderson · 2020-08-18T16:07:35Z

Used pip install --upgrade archivebox, it upgraded and installed 2 additional packages.

Successfully installed archivebox-0.4.17 croniter-0.3.34 natsort-7.0.1

Went to archive directory to run archivebox init.

[*] [2020-08-18 16:00:00] Writing 1723 links to main index...
Traceback (most recent call last):
  File "/home/USERNAME/.local/lib/python3.7/site-packages/django/db/models/query.py", line 575, in update_or_create
    obj = self.select_for_update().get(**kwargs)
  File "/home/USERNAME/.local/lib/python3.7/site-packages/django/db/models/query.py", line 417, in get
    self.model._meta.object_name
core.models.DoesNotExist: Snapshot matching query does not exist.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/USERNAME/.local/lib/python3.7/site-packages/django/db/backends/utils.py", line 86, in _execute
    return self.cursor.execute(sql, params)
  File "/home/USERNAME/.local/lib/python3.7/site-packages/django/db/backends/sqlite3/base.py", line 396, in execute
    return Database.Cursor.execute(self, query, params)
sqlite3.IntegrityError: UNIQUE constraint failed: core_snapshot.timestamp


Traceback (most recent call last):
  File "/home/USERNAME/.local/bin/archivebox", line 8, in <module>
    sys.exit(main())
  File "/home/USERNAME/.local/lib/python3.7/site-packages/archivebox/cli/__init__.py", line 126, in main
    pwd=pwd or OUTPUT_DIR,
  File "/home/USERNAME/.local/lib/python3.7/site-packages/archivebox/cli/__init__.py", line 62, in run_subcommand
    module.main(args=subcommand_args, stdin=stdin, pwd=pwd)    # type: ignore
  File "/home/USERNAME/.local/lib/python3.7/site-packages/archivebox/cli/archivebox_init.py", line 35, in main
    out_dir=pwd or OUTPUT_DIR,
  File "/home/USERNAME/.local/lib/python3.7/site-packages/archivebox/util.py", line 111, in typechecked_function
    return func(*args, **kwargs)
  File "/home/USERNAME/.local/lib/python3.7/site-packages/archivebox/main.py", line 377, in init
    write_main_index(list(all_links.values()), out_dir=out_dir)
  File "/home/USERNAME/.local/lib/python3.7/site-packages/archivebox/util.py", line 111, in typechecked_function
    return func(*args, **kwargs)
  File "/home/USERNAME/.local/lib/python3.7/site-packages/archivebox/index/__init__.py", line 246, in write_main_index
    write_sql_main_index(links, out_dir=out_dir)
  File "/home/USERNAME/.local/lib/python3.7/site-packages/archivebox/util.py", line 111, in typechecked_function
    return func(*args, **kwargs)
  File "/home/USERNAME/.local/lib/python3.7/site-packages/archivebox/index/sql.py", line 46, in write_sql_main_index
    Snapshot.objects.update_or_create(url=link.url, defaults=info)
  File "/home/USERNAME/.local/lib/python3.7/site-packages/django/db/models/manager.py", line 82, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/home/USERNAME/.local/lib/python3.7/site-packages/django/db/models/query.py", line 580, in update_or_create
    obj, created = self._create_object_from_params(kwargs, params, lock=True)
  File "/home/USERNAME/.local/lib/python3.7/site-packages/django/db/models/query.py", line 604, in _create_object_from_params
  
  File "/home/USERNAME/.local/lib/python3.7/site-packages/django/db/models/query.py", line 433, in create
    obj.save(force_insert=True, using=self.db)
  File "/home/USERNAME/.local/lib/python3.7/site-packages/django/db/models/base.py", line 746, in save
    force_update=force_update, update_fields=update_fields)
  File "/home/USERNAME/.local/lib/python3.7/site-packages/django/db/models/base.py", line 784, in save_base
    force_update, using, update_fields,
  File "/home/USERNAME/.local/lib/python3.7/site-packages/django/db/models/base.py", line 887, in _save_table
    results = self._do_insert(cls._base_manager, using, fields, returning_fields, raw)
  File "/home/USERNAME/.local/lib/python3.7/site-packages/django/db/models/base.py", line 926, in _do_insert
    using=using, raw=raw,
  File "/home/USERNAME/.local/lib/python3.7/site-packages/django/db/models/manager.py", line 82, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/home/USERNAME/.local/lib/python3.7/site-packages/django/db/models/query.py", line 1204, in _insert
    return query.get_compiler(using=using).execute_sql(returning_fields)
  File "/home/USERNAME/.local/lib/python3.7/site-packages/django/db/models/sql/compiler.py", line 1392, in execute_sql
    cursor.execute(sql, params)
  File "/home/USERNAME/.local/lib/python3.7/site-packages/django/db/backends/utils.py", line 68, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "/home/USERNAME/.local/lib/python3.7/site-packages/django/db/backends/utils.py", line 77, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/home/USERNAME/.local/lib/python3.7/site-packages/django/db/backends/utils.py", line 86, in _execute
    return self.cursor.execute(sql, params)
  File "/home/USERNAME/.local/lib/python3.7/site-packages/django/db/utils.py", line 90, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/home/USERNAME/.local/lib/python3.7/site-packages/django/db/backends/utils.py", line 86, in _execute
    return self.cursor.execute(sql, params)
  File "/home/USERNAME/.local/lib/python3.7/site-packages/django/db/backends/sqlite3/base.py", line 396, in execute
    return Database.Cursor.execute(self, query, params)
django.db.utils.IntegrityError: UNIQUE constraint failed: core_snapshot.timestamp

Note: I'm not sure if you need the entire traceback each time, since most of it is identical, but figured more is better when hunting down bugs. Apologies if it's too much.

coisnepe · 2020-08-19T08:01:00Z

Deployed the latest Docker image and it seems to have fixed the issue. Thanks so much!

pirate · 2020-08-19T15:54:03Z

@drpfenderson let me know if you're still having any issues and we can reopen the ticket.

drpfenderson · 2020-08-19T16:30:16Z

@pirate Updated to newest.

[i] [2020-08-19 15:55:47] ArchiveBox v0.4.21: archivebox init

same error, exactly, as my last log.

[*] [2020-08-19 16:20:40] Writing 1723 links to main index...
Traceback (most recent call last):
  File "/home/USERNAME/.local/lib/python3.7/site-packages/django/db/models/query.py", line 575, in update_or_create
    obj = self.select_for_update().get(**kwargs)
  File "/home/USERNAME/.local/lib/python3.7/site-packages/django/db/models/query.py", line 417, in get
    self.model._meta.object_name
core.models.DoesNotExist: Snapshot matching query does not exist.

The rest of the log is exactly the same as well, line references and all.

EDIT: I thought maybe I could try nuking it, starting from scratch. No dice, same error. I tried with docker and docker-compose as well, after removing the original package from pip. Same error in both, but with python3.8 instead.

jrruethe · 2020-08-23T21:26:33Z

For what it is worth, v0.4.21 fixed the issue I was having regarding sqlite3.IntegrityError: UNIQUE constraint failed: core_snapshot.timestamp. Thank you!

drpfenderson · 2020-09-02T21:42:08Z

With the changes present in the cdvv7788:sql_index branch, reflected in PR #452, it fixed my issue! I was able to archivebox init on the old index, updated with some broken directories, but ultimately wrote everything to the index. Looks to be intact! I'll just add the "invalid link data directories" through a .txt file.

cdvv7788 · 2020-09-02T21:46:41Z

@pirate I added a final check to avoid duplication in the PR when migrating the index. Check it when reviewing the PR #452

pirate · 2022-04-12T21:58:06Z

Note I've added a new DB/filesystem troubleshooting area to the wiki that may help people arriving here from Google: https://github.com/ArchiveBox/ArchiveBox/wiki/Upgrading-or-Merging-Archives#database-troubleshooting

Contributions/suggestions welcome there.

pirate added type: bug report touches: data/schema/architecture size: medium status: wip Work is in-progress / has already been partially completed labels Aug 1, 2020

pirate assigned cdvv7788 Aug 1, 2020

apkallum mentioned this issue Aug 17, 2020

fix database constraint issue #447

Closed

6 tasks

pirate closed this as completed Aug 19, 2020

pirate reopened this Aug 19, 2020

drpfenderson closed this as completed Sep 2, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Bugfix: django.db.utils.IntegrityError: UNIQUE constraint failed: core_snapshot.timestamp #412

Bugfix: django.db.utils.IntegrityError: UNIQUE constraint failed: core_snapshot.timestamp #412

drpfenderson commented Jul 31, 2020

karlicoss commented Aug 11, 2020 •

edited

pirate commented Aug 11, 2020

jrruethe commented Aug 15, 2020

coisnepe commented Aug 16, 2020 •

edited

apkallum commented Aug 17, 2020

drpfenderson commented Aug 17, 2020 •

edited

pirate commented Aug 18, 2020

drpfenderson commented Aug 18, 2020

coisnepe commented Aug 19, 2020

pirate commented Aug 19, 2020

drpfenderson commented Aug 19, 2020 •

edited

jrruethe commented Aug 23, 2020

drpfenderson commented Sep 2, 2020 •

edited

cdvv7788 commented Sep 2, 2020

pirate commented Apr 12, 2022

Bugfix: django.db.utils.IntegrityError: UNIQUE constraint failed: core_snapshot.timestamp #412

Bugfix: django.db.utils.IntegrityError: UNIQUE constraint failed: core_snapshot.timestamp #412

Comments

drpfenderson commented Jul 31, 2020

Describe the bug

Steps to reproduce

Screenshots or log output

Software versions

karlicoss commented Aug 11, 2020 • edited

pirate commented Aug 11, 2020

jrruethe commented Aug 15, 2020

coisnepe commented Aug 16, 2020 • edited

apkallum commented Aug 17, 2020

drpfenderson commented Aug 17, 2020 • edited

pirate commented Aug 18, 2020

drpfenderson commented Aug 18, 2020

coisnepe commented Aug 19, 2020

pirate commented Aug 19, 2020

drpfenderson commented Aug 19, 2020 • edited

jrruethe commented Aug 23, 2020

drpfenderson commented Sep 2, 2020 • edited

cdvv7788 commented Sep 2, 2020

pirate commented Apr 12, 2022

karlicoss commented Aug 11, 2020 •

edited

coisnepe commented Aug 16, 2020 •

edited

drpfenderson commented Aug 17, 2020 •

edited

drpfenderson commented Aug 19, 2020 •

edited

drpfenderson commented Sep 2, 2020 •

edited