Can't recreate database.

[TheCake21]

Version info

Duplicati Version: 2.0.1.35
Operating System: Ubuntu 16.04 Mono 4.6.1
Backend: Any

Bug description

Cannot recreate the database on a huge backup. The Backup wasnt't finished yet and as it allready took around 1,5 months to create i can't redo it.

debug log

{"ClassName":"Mono.Data.Sqlite.SqliteException","Message":"Abort due to constraint violation
UNIQUE constraint failed: BlocksetEntry.BlocksetID, BlocksetEntry.Index","Data":null,"InnerException":null,"HelpURL":null,"StackTraceString":"  at Mono.Data.Sqlite.SQLite3.Reset (Mono.Data.Sqlite.SqliteStatement stmt) [0x00096] in <57d07b96bbef430fb73bd5de676da784>:0 
  at Mono.Data.Sqlite.SQLite3.Step (Mono.Data.Sqlite.SqliteStatement stmt) [0x00046] in <57d07b96bbef430fb73bd5de676da784>:0 
  at Mono.Data.Sqlite.SqliteDataReader.NextResult () [0x00129] in <57d07b96bbef430fb73bd5de676da784>:0 
  at Mono.Data.Sqlite.SqliteDataReader..ctor (Mono.Data.Sqlite.SqliteCommand cmd, System.Data.CommandBehavior behave) [0x00051] in <57d07b96bbef430fb73bd5de676da784>:0 
  at (wrapper remoting-invoke-with-check) Mono.Data.Sqlite.SqliteDataReader:.ctor (Mono.Data.Sqlite.SqliteCommand,System.Data.CommandBehavior)
  at Mono.Data.Sqlite.SqliteCommand.ExecuteReader (System.Data.CommandBehavior behavior) [0x00006] in <57d07b96bbef430fb73bd5de676da784>:0 
  at Mono.Data.Sqlite.SqliteCommand.ExecuteNonQuery () [0x00000] in <57d07b96bbef430fb73bd5de676da784>:0 
  at Duplicati.Library.Main.Database.ExtensionMethods.ExecuteNonQuery (System.Data.IDbCommand self, System.String cmd, System.Object[] values) [0x0005f] in <c6d4a3718b924f65909eb0a2af46f7c9>:0 
  at Duplicati.Library.Main.Database.LocalRecreateDatabase.FindMissingBlocklistHashes (System.Int64 hashsize, System.Int64 blocksize, System.Data.IDbTransaction transaction) [0x0019b] in <c6d4a3718b924f65909eb0a2af46f7c9>:0 
  at Duplicati.Library.Main.Operation.RecreateDatabaseHandler.DoRun (Duplicati.Library.Main.Database.LocalDatabase dbparent, System.Boolean updating, Duplicati.Library.Utility.IFilter filter, Duplicati.Library.Main.Operation.RecreateDatabaseHandler+NumberedFilterFilelistDelegate filelistfilter, Duplicati.Library.Main.Operation.RecreateDatabaseHandler+BlockVolumePostProcessor blockprocessor) [0x00e8c] in <c6d4a3718b924f65909eb0a2af46f7c9>:0 
  at Duplicati.Library.Main.Operation.RecreateDatabaseHandler.Run (System.String path, Duplicati.Library.Utility.IFilter filter, Duplicati.Library.Main.Operation.RecreateDatabaseHandler+NumberedFilterFilelistDelegate filelistfilter, Duplicati.Library.Main.Operation.RecreateDatabaseHandler+BlockVolumePostProcessor blockprocessor) [0x00035] in <c6d4a3718b924f65909eb0a2af46f7c9>:0 
  at Duplicati.Library.Main.Operation.RepairHandler.RunRepairLocal (Duplicati.Library.Utility.IFilter filter) [0x000c2] in <c6d4a3718b924f65909eb0a2af46f7c9>:0 
  at Duplicati.Library.Main.Operation.RepairHandler.Run (Duplicati.Library.Utility.IFilter filter) [0x00015] in <c6d4a3718b924f65909eb0a2af46f7c9>:0 
  at Duplicati.Library.Main.Controller+<Repair>c__AnonStorey3.<>m__0 (Duplicati.Library.Main.RepairResults result) [0x0001c] in <c6d4a3718b924f65909eb0a2af46f7c9>:0 
  at Duplicati.Library.Main.Controller.RunAction[T] (T result, System.String[]& paths, Duplicati.Library.Utility.IFilter& filter, System.Action`1[T] method) [0x00047] in <c6d4a3718b924f65909eb0a2af46f7c9>:0 ","RemoteStackTraceString":null,"RemoteStackIndex":0,"ExceptionMethod":null,"HResult":-2147467259,"Source":"Mono.Data.Sqlite"}

[TheCake21]

@kenkendk Are you planning to do something about this issue in the next week or so? If not i might take a look at this myself. This broke 2 huge Backups for me so it's quite important to be fixed.

[kenkendk]

For the next week, my plan is to look at recovering a backup after loss of dblock files.
If you have the resources and skills to look at this, it would be much appreciated.
I have this and the related two issues #1699 and #2140 marked as required before a 2.0 beta can go out, so I will look at it soon. But I do not have the data that you have that cause the error, so it is much harder for me to figure out.

[Athep]

Tested with 2.0.1.39_canary_2017-01-08. Please see #2224

[Athep]

Tested with 2.0.1.42_canary_2017-01-16

Blockset insert failed, this is likely issue #2140, #1699, #2048 or #2178, comitting temporary tables as permanent
System.Data.SQLite.SQLiteException (0x80004005): constraint failed
UNIQUE constraint failed: BlocksetEntry.BlocksetID, BlocksetEntry.Index
at System.Data.SQLite.SQLite3.Reset(SQLiteStatement stmt)
at System.Data.SQLite.SQLite3.Step(SQLiteStatement stmt)
at System.Data.SQLite.SQLiteDataReader.NextResult()
at System.Data.SQLite.SQLiteDataReader..ctor(SQLiteCommand cmd, CommandBehavior behave)
at System.Data.SQLite.SQLiteCommand.ExecuteReader(CommandBehavior behavior)
at System.Data.SQLite.SQLiteCommand.ExecuteNonQuery(CommandBehavior behavior)
at Duplicati.Library.Main.Database.ExtensionMethods.ExecuteNonQuery(IDbCommand self, String cmd, Object[] values)
at Duplicati.Library.Main.Database.LocalRecreateDatabase.FindMissingBlocklistHashes(Int64 hashsize, Int64 blocksize, IDbTransaction transaction)

[kenkendk]

@Athep: The sqlite database should now contain two extra tables (TempBlocklist-*-Failed and TempSmallist-*-Failed).

Can you create a "bug-report" file and email it to me?

More detailed explanation: #2140 (comment)

[kenkendk]

I have made a new build that should fix the issue:
https://github.com/duplicati/duplicati/releases/tag/v2.0.1.43-2.0.1.43_canary_2017-01-19

[kenkendk]

I am considering this fixed, and collecting performance related issues under #2302