Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ctb files and all ctb~backups concurrently corrupted #2148

Closed
gitnard opened this issue Nov 5, 2022 · 4 comments
Closed

ctb files and all ctb~backups concurrently corrupted #2148

gitnard opened this issue Nov 5, 2022 · 4 comments

Comments

@gitnard
Copy link

gitnard commented Nov 5, 2022

--- issue

My main.cbt and all 40 main.cbt~backups have become corrupted at the same time.

Three other .cbt files and their backups also stored in the same directory and used over the same several week time period did not become corrupted.

All 41 corrupted files now share the same Last Data Modification timestamp, matching the likely time of incident. Timestamps previously would have spanned several weeks.

All 41 corrupted files are of similar size. Seven separate sizes between 1372132 B and 1417188 B.

All 41 corrupted filed are SQlite Not Protected .cbt. and have following tree summary info: Rich Text Nodes 113, Plain Text Nodes 0, Code Nodes 0, Images 0, Embedded Files 0, Tables 2, CodeBoxes 0, Anchors 0.

--- sqlite info

In terminal, using cherrytree to open these files produces similar but not identical messages:
=== Database problems report ===
: *** in database main ***
Main freelist: freelist leaf count too big on page 291
Page 149: btreeInitPage() returns error code 11
... then 50 more btreeInitPage() messages , then...
On tree page 89 cell 1: invalid page number 5990766
On tree page 89 cell 0: invalid page number 5701632
Page 87 is never used
... then 40 more never used messages, then ...
[
] [ ] [error]
[
] [ ] [warning] %s Couldn't open file: on_open

In db-browser-for-sqlite-gui, node table for these corrupted files shows all ~113 nodes, but most notes show error loading, and only between 1 and 22 of the nodes do load. The loaded nodes names match cherrytree document nodes that have not been recently edited.

--- settings and system

Cherrytree >Preferences >Miscellaneous >Saving: Not Autosave Every n Minutes, Not Autosave on Quit, Yes Create a Backup Copy Before Saving, Number of Backups to Keep 40.

Using Cherrytree 0.99.41 on MX-Linux 21 (Debian 11, Xfce 16) on UPS protected desktop computer using local drive. The NVMe drive smartclt checks out ok. No other computer issues or file issues.

@gitnard gitnard changed the title cbt files and all cbt~backups concurrently corrupted ctb files and all ctb~backups concurrently corrupted Nov 8, 2022
@giuspen
Copy link
Owner

giuspen commented Nov 18, 2022

The backups are rotated, so it means that for 40 subsequent saves in the same session (cannot be 2 different sessions as you would not have been able to open the document in the first place) the document written to disk was corrupted.

Please confirm the type of error you get, is the one saying that the file is not a cherrytree document?

Maybe I should change the backup strategy to rotate between sessions and not between subsequent saves.

Another option would be that after one rotation, the backup document is opened and the full tree structure tested. I think I will try this second option first.

giuspen added a commit that referenced this issue Nov 18, 2022
…with verification of the backup before rotating (#2148)
giuspen added a commit that referenced this issue Nov 19, 2022
…with verification of the backup before rotating (#2148)
@gitnard
Copy link
Author

gitnard commented Nov 20, 2022

Regarding, "The backups are rotated, so it means that for 40 subsequent saves in the same session (cannot be 2 different sessions as you would not have been able to open the document in the first place) the document written to disk was corrupted.

That precisely may be the issue. Successfully reopened main.ctb files (and some in-session saves), were subsequently saved as incremented main.cbt-backups (without any error message). Then when the main.cbt became corrupted, all 40 main.cbt-backups then were observed to all be corrupt as well. Expected behaviour would be for the .ctb-backup files to have remained independent of the main.ctb corruption, and should not have all received the same Last Data Modification timestamp as the main.ctb.

Regarding improving backup strategy.

I concur with Not implementing "rotate between sessions and not between subsequent saves". Subsequent saves within a session provide an audit-trail and safety-net when pushing hard through substantive updating.

Based on my unexpected problem, I embrace "after one rotation the backup document is opened and the full tree structure tested"

Not that I am clear as to what a "rotation" is. Since I had previously thought that when saving a .ctb, each previously saved same-named .ctb and ctb-backup simply had an additional tilde suffixed to the filename extension. And that each backup gets no further changes.

It would be reassuring to know each newly edited .ctb save is tested for corruption. And it would be unsettling to know all previously saved .ctb-backup might collectively somehow get additional changes.

Here are the type of error I got, during my once only ever incident with Cherrytree.

Type of error, when use gui to open any of the 40 corrupt .ctb files, is a pop up small window that says:

Warning
The database file /filepath/filename.ctb is corrupt, see log for more details
OK

Type of error, when use cli terminal to open the most recently saved .ctb, is:

[2022-11-19 15:23:38.866] [   ] [debug] /home/gn/.config/cherrytree/config.cfg parsed
[2022-11-19 15:23:39.039] [   ] [warning] 
=== Database problems report ===
: *** in database main ***
Main freelist: freelist leaf count too big on page 291
Page 149: btreeInitPage() returns error code 11
Page 301: btreeInitPage() returns error code 11
Page 335: btreeInitPage() returns error code 11
Page 334: btreeInitPage() returns error code 11
Page 333: btreeInitPage() returns error code 11
Page 332: btreeInitPage() returns error code 11
Page 326: btreeInitPage() returns error code 11
Page 290: btreeInitPage() returns error code 11
Page 281: btreeInitPage() returns error code 11
Page 278: btreeInitPage() returns error code 11
Page 277: btreeInitPage() returns error code 11
Page 275: btreeInitPage() returns error code 11
Page 274: btreeInitPage() returns error code 11
Page 271: btreeInitPage() returns error code 11
Page 262: btreeInitPage() returns error code 11
Page 261: btreeInitPage() returns error code 11
Page 260: btreeInitPage() returns error code 11
Page 254: btreeInitPage() returns error code 11
Page 240: btreeInitPage() returns error code 11
Page 227: btreeInitPage() returns error code 11
Page 223: btreeInitPage() returns error code 11
Page 217: btreeInitPage() returns error code 11
Page 212: btreeInitPage() returns error code 11
Page 208: btreeInitPage() returns error code 11
Page 201: btreeInitPage() returns error code 11
Page 199: btreeInitPage() returns error code 11
Page 195: btreeInitPage() returns error code 11
Page 188: btreeInitPage() returns error code 11
Page 187: btreeInitPage() returns error code 11
Page 186: btreeInitPage() returns error code 11
Page 183: btreeInitPage() returns error code 11
Page 181: btreeInitPage() returns error code 11
Page 176: btreeInitPage() returns error code 11
Page 172: btreeInitPage() returns error code 11
Page 168: btreeInitPage() returns error code 11
Page 164: btreeInitPage() returns error code 11
Page 148: btreeInitPage() returns error code 11
Page 143: btreeInitPage() returns error code 11
Page 142: btreeInitPage() returns error code 11
Page 140: btreeInitPage() returns error code 11
Page 139: btreeInitPage() returns error code 11
Page 138: btreeInitPage() returns error code 11
Page 137: btreeInitPage() returns error code 11
Page 136: btreeInitPage() returns error code 11
Page 134: btreeInitPage() returns error code 11
Page 129: btreeInitPage() returns error code 11
Page 127: btreeInitPage() returns error code 11
Page 126: btreeInitPage() returns error code 11
Page 124: btreeInitPage() returns error code 11
Page 123: btreeInitPage() returns error code 11
Page 108: btreeInitPage() returns error code 11
Page 107: btreeInitPage() returns error code 11
Page 106: btreeInitPage() returns error code 11
Page 101: btreeInitPage() returns error code 11
On tree page 89 cell 1: invalid page number 5990766
On tree page 89 cell 0: invalid page number 5701632
Page 87 is never used
Page 88 is never used
Page 91 is never used
Page 92 is never used
Page 93 is never used
Page 94 is never used
Page 95 is never used
Page 96 is never used
Page 97 is never used
Page 98 is never used
Page 99 is never used
Page 100 is never used
Page 102 is never used
Page 103 is never used
Page 104 is never used
Page 105 is never used
Page 109 is never used
Page 110 is never used
Page 111 is never used
Page 112 is never used
Page 113 is never used
Page 114 is never used
Page 115 is never used
Page 116 is never used
Page 117 is never used
Page 118 is never used
Page 119 is never used
Page 120 is never used
Page 121 is never used
Page 122 is never used
Page 125 is never used
Page 128 is never used
Page 130 is never used
Page 131 is never used
Page 132 is never used
Page 133 is never used
Page 135 is never used
Page 141 is never used
Page 144 is never used
Page 145 is never used
Page 146 is never used
Page 147 is never used
Page 150 is never used

[2022-11-19 15:24:46.521] [   ] [error] 
[2022-11-19 15:24:46.522] [   ] [warning] %s Couldn't open file: on_open

Type of error, when use cli terminal to open the earliest saved .ctb40~backup is:

[2022-11-19 15:36:17.224] [   ] [debug] /home/gn/.config/cherrytree/config.cfg parsed
[2022-11-19 15:36:17.472] [   ] [warning] 
=== Database problems report ===
: *** in database main ***
Main freelist: invalid page number 0
Main freelist: invalid page number 0
Main freelist: invalid page number 0
Main freelist: invalid page number 0
Main freelist: invalid page number 0
Main freelist: invalid page number 0
Main freelist: invalid page number 0
Main freelist: invalid page number 0
Main freelist: invalid page number 0
Main freelist: invalid page number 0
Main freelist: invalid page number 0
Main freelist: invalid page number 0
Main freelist: invalid page number 0
Main freelist: invalid page number 0
Main freelist: invalid page number 0
Main freelist: invalid page number 0
Main freelist: invalid page number 0
Main freelist: invalid page number 0
Main freelist: invalid page number 0
Main freelist: invalid page number 0
Main freelist: invalid page number 0
Main freelist: invalid page number 0
Main freelist: invalid page number 0
Main freelist: invalid page number 0
Main freelist: invalid page number 0
Main freelist: invalid page number 0
Main freelist: invalid page number 0
Main freelist: invalid page number 0
Main freelist: invalid page number 0
Main freelist: invalid page number 0
Main freelist: invalid page number 0
Main freelist: invalid page number 0
Main freelist: invalid page number 0
Main freelist: invalid page number 0
Main freelist: freelist leaf count too big on page 10
Main freelist: invalid page number 167772160
2nd reference to page 10
Page 47: free space corruption
Page 319: btreeInitPage() returns error code 11
Page 310: btreeInitPage() returns error code 11
Page 307: btreeInitPage() returns error code 11
Page 306: btreeInitPage() returns error code 11
Page 304: btreeInitPage() returns error code 11
Page 303: btreeInitPage() returns error code 11
Page 300: btreeInitPage() returns error code 11
Page 291: btreeInitPage() returns error code 11
Page 290: btreeInitPage() returns error code 11
Page 287: btreeInitPage() returns error code 11
Page 273: btreeInitPage() returns error code 11
Page 266: btreeInitPage() returns error code 11
Page 261: btreeInitPage() returns error code 11
Page 257: btreeInitPage() returns error code 11
Page 253: btreeInitPage() returns error code 11
Page 247: btreeInitPage() returns error code 11
Page 242: btreeInitPage() returns error code 11
Page 238: btreeInitPage() returns error code 11
Page 231: btreeInitPage() returns error code 11
Page 229: btreeInitPage() returns error code 11
Page 225: btreeInitPage() returns error code 11
Page 218: btreeInitPage() returns error code 11
Page 217: btreeInitPage() returns error code 11
Page 216: btreeInitPage() returns error code 11
Page 213: btreeInitPage() returns error code 11
Page 211: btreeInitPage() returns error code 11
Page 206: btreeInitPage() returns error code 11
Page 202: btreeInitPage() returns error code 11
Page 198: btreeInitPage() returns error code 11
Page 194: btreeInitPage() returns error code 11
Page 180: btreeInitPage() returns error code 11
Page 179: btreeInitPage() returns error code 11
Page 178: btreeInitPage() returns error code 11
Page 173: btreeInitPage() returns error code 11
Page 172: btreeInitPage() returns error code 11
Page 170: btreeInitPage() returns error code 11
Page 169: btreeInitPage() returns error code 11
Page 168: btreeInitPage() returns error code 11
Page 167: free space corruption
Page 166: btreeInitPage() returns error code 11
Page 164: btreeInitPage() returns error code 11
Page 159: btreeInitPage() returns error code 11
Page 157: btreeInitPage() returns error code 11
Page 156: btreeInitPage() returns error code 11
Page 154: btreeInitPage() returns error code 11
Page 153: btreeInitPage() returns error code 11
Page 144: btreeInitPage() returns error code 11
Page 138: btreeInitPage() returns error code 11
Page 137: btreeInitPage() returns error code 11
Page 136: btreeInitPage() returns error code 11
Page 131: btreeInitPage() returns error code 11
Page 119: free space corruption
Page 116: free space corruption
Page 97: free space corruption
Page 95: btreeInitPage() returns error code 11
Page 84: free space corruption
Page 70: free space corruption
Page 61: btreeInitPage() returns error code 11
Page 320: btreeInitPage() returns error code 11
Page 32: btreeInitPage() returns error code 11
Page 27: btreeInitPage() returns error code 11
Page 22 is never used

[2022-11-19 15:36:27.388] [   ] [error] 
[2022-11-19 15:36:27.388] [   ] [warning] %s Couldn't open file: on_open

@giuspen
Copy link
Owner

giuspen commented Nov 20, 2022

As I am currently implementing the sanity check of a document, it would be useful if you could share with me privately one or more of these corrupted documents (if different errors) so that I can also test the code on it. You can find my private email in cherrytree help--about dialog

giuspen added a commit that referenced this issue Nov 22, 2022
…with verification of the backup before rotating (#2148)
giuspen added a commit that referenced this issue Nov 22, 2022
…with verification of the backup before rotating (#2148)
giuspen added a commit that referenced this issue Nov 23, 2022
…with verification of the backup before rotating (#2148)
giuspen added a commit that referenced this issue Nov 24, 2022
…ation of the backup before encryption and rotation (#2148)
@giuspen
Copy link
Owner

giuspen commented Nov 24, 2022

I implemented the document integrity verification in upcoming 0.99.52 before backups rotation. I hope to not see anymore such issue.

@giuspen giuspen closed this as completed Nov 24, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants