"unable to open database file", intranet library, is read only? #257
Comments
|
Phew, this might be an issue with SQLite databases and network storage, though that's "usually" reported as crap happening on DropBox shares and the like (cloud storage). Thanks for the screenshots: what I can glean from there is that I probably need to augment that error report so that we can dig up on what database file exactly SQLite is barfing a hairball. On another note I do recall so old trouble back in '19 with the whole Qiqqa Sync business, which had a few lines from the commercial age being obstructive, but then that should have been gone out the window by window. Got to dig in, so don't expect anything useful before the weekend. Plan of attack in my mind right now:
P.S.: just doublechecking here to make sure I can tick off the scare in my brain: both machines still have a working Qiqqa and Qiqqa libraries, right? |
|
Cheers for the wonderfully thought-out plan! :) Yep - both having working Qiqqa, just getting the intranet library working on the second doesn't work, even though I'm using the same approach as on the first. One thought is that I don't know with 100% certainty that both machines are setup EXACTLY the same way in terms of reaching the NAS. Now, I think they are. But browsing to the NAS as a network drive took more steps on the home machine (that works) than the work machine, which worked straight off the bat. Now they look and function the same but I don't know if there's any small difference behind the scenes. I suspect this isn't the issue but I figured I'd mention it just in case. On a completely separate note, I noticed that the Qiqqa getsatisfaction community has been removed; I wonder if it's possible for us to get an archive of this somehow (maybe Wayback machine)? I figure this could be quite useful since it's got a lot of community ideas for workarounds or bug discussions, so it could be a useful resource... Cheers! |
|
Before I'm off (near 1 AM now): I do recall some trouble with sync behaviour (enabled/disabled) back in '19. Given your problem, I must look into that part of the code again I expect. At least that's my initial hunch without any backing except "gut feeling, no data". We'll see what happens when I do an updated version of qiqqa for you around tomorrow or day after, depending on how things go with the day job. On the separate note:
[OK, that story isn't written in chronologic order. Hope it's intelligible anyway. Anyhoo, I'm off. Gotta go. 👋 ] |
|
Interesting rollercoaster ride of a tale about the GS issues! I suspect there's not tons of useful stuff on there but you never know. I'm sure we'll find all the bugs again anyway!! |
|
OK, looks like it's the sync target, i.e. the SQLite database file on the NAS, which might be damaged and SQLite barfing a hairball about this. Have been digging around as I needed to see where I augment the logging, etc. and not yet reproduced your problem, but that should be fine for the next phase: running a Qiqqa update to test this in more detail. WARNING NOTESince I gather from digging through the code and your screenshots of the failure that the NAS copy of your library is probably corrupt (the metadata store at least as the BibTeX etc. is stored in a SQLite database) and before we commence I urge you to back up your Qiqqa libraries from the local machine: nobody would be happy when that stuff gets damaged by a subsequent (newer) Qiqqa run after installing the Qiqqa test release that's going to be referenced in my next comment. Best practice would be to backup the Qiqqa base directory and everything inside from the local machine and possibly of the NAS as well, just to be on the safe side and my preliminary guess turns out wrong. Personally, I would ZIP (or 7ZIP) the whole kaboodle; might want to use a "multiple volume 7ZIP" archive for that, i.e. tick the box in 7zip when creating the archive and you get archive file segments named archive.001, archive.002, etc. Screenshots of how that might look:
^^^ would produce something like this ==>
(nitpick: those are screenshots from different actions, so base.7z in the first should be read as 'C9F0D079-EE4C-4A15-8547-72164A7A356D-updated-GHO.7z') END OF WARNING NOTELink to Qiqqa update for testing coming up next.... |
… suitable, instead of brutally clipped off. - added the Sync Target column at the right size in there, so you can see what Qiqqa thinks the current Sync Target directory might be. `(null)` when none has been set up. - WARNING: this stuff is not yet editable! - File.Copy() gives out a horrible user-unfriendly exception: path only, no cause, no blurb about it being the source or target either. Hackily fixed for the sync template database now. Must be inspected. MIGHT be just good enough for recovering from a broken sync target database like Simon Dedman in jimmejardine#257 is possibly sadled with. (IFF one goes and *manually* deletes the corrupted XYZ.s3db database file in that sync destination, anyway. Dangerous stuff. :-)
|
OK, new Qiqqa setup for you to try at https://drive.google.com/file/d/13gWzHHtL0dpRAJlSWZjweE3-3RWXFpxY/view?usp=sharing Make sure you've backed up your Qiqqa data before running this installer. Once installed, run Qiqqa and see if you can reproduce the problem and see what Qiqqa says then in the logging. This time around Qiqqa should be a bit more verbose and also tell us which database file exactly triggered that b0rk. (This is an unlabeled release, coded as v82.0.7576.3604 for commit SHA-1: 4433a6a ) Trouble that I expect to find over there:
|
|
Oh, the new Qiqqa should show the Sync Target path in the sync manager dialog. That might also help the diagnosis a bit.
|
|
Howdy! A thousand thanks for helping with this, again. I installed the new version of qiqqa on my work machine. It looks like my original screenshot (second one) was cropped too far down, and the line before was key:
Sync target path is correct, \NAS\Si Work\Papers & Books\QiqqaSync In my home machine (working, just did a full sunc successfully which is weird since you'd think it would announce a problem with Qiqqa.LibraryMetadata.s3db) I'll backup the NAS library then nuke that s3db file and try to sync it back (again from home machine) then see if the work machine, uh, works. To be continued! |
|
So: nuked Qiqqa.LibraryMetadata.s3db, resyned on home machine to rebuild it, successfully (on home machine), but no change on work machine. Thought: could it be that the database file has some form of machine-specific ID that leads it to be readable on the machine that set it up, but not on other machines? Logs attached for the work machine. In 'Properties' for that file, details in General (archive) and Security (Allow for all except 'full control') are the same on both machines. Uninstalled & reinstalled Qiqqa on work machine. Upon reopening, my intranet library is present (with 0 docs as usual). This seems strange; maybe the qiqqa library isn't removed upon uninstall... I still haven't updated my home machine's Qiqqa to the latest version, since it works fine. Let me know if doing so would be beneficial for diagnostics. Cheers again bud! |
Not that I recall. Besides, the .s3db file is a plain Sqlite database file and those are portable across machines as far as I know. (Since Qiqqa only runs on Windows, it's even simpler: every machine will then be little endian and 32-bit execution for Qiqqa runs as a 32-bit .NET application, even when the OS is 64-bit, so I cannot imagine anything that might make it load on one and fail on the other. (Aside: This 32-bittiness is due to the embedded webbrowser (XULrunner) forcing the whole application down to 32-bit exec alas.)
Indeed, Qiqqa does not delete user config upon install: that's a safety measure so folks don't 'loose' their Qiqqa libraries when upgrading. The magic ingredient there is the file listing the discovered libraries, which sits in the libraries base directory:
Might be useful later, but hold off for a bit as I'm working on a new release which should be ready tomorrow there-abouts. Besides, the .s3db file (and the rest) SHOULD BE cross-version binary compatible with all v79/v80/v81/v82 Qiqqa releases: I haven't changed the formats of any (yet). 😄 Consequently my expectation had been that your Qiqqa setups should either yak about the .s3db database file both, or be able to read & write it without errors. 🤔 🤔 🤔 🤔 Given the results so far and the (to me) very curious behaviour as you describe, let me back-pedal and make darn sure we're on the same page about your layout:
That last point can only be easily checked with the Qiqqa version that you got from me, as previous versions didn't yet show the sync path in the Sync Management dialog. Anyway: is the above HOME/OFFICE/NAS layout assumption correct? (If not, please correct me!) There's one more thing you might want to check, just in case: The 'sync target' (in your case the NAS directory) should also contain a file called This is one example how such a file should look: {"Id":"INTRANET_FFFF9774-FFFF-FFFF-FFFF-E67433F7FFFF",
"Title":"Issue 142 Library #1 for Qiqqa",
"Description":"yada yada yada."}The exact format is quite liberal (whitespace & newlines don't matter) as this is a JSON format file. The important bit is the Id: this is the exact name of the directory your library should be stored in on both machines. For example, if your Qiqqa base directory on HOME would be (like I have): then the library from the example should be in so you should find the library database file there, at least: Same for the OFFICE machine, where I'll say, for this example, the base directory is at hence the same library there should be in with its own local library database file (and other data): So far, that's the only other thing I can imagine that may have occurred somehow, is one of the machines (OFFICE?) losing the sync target reference. Do note that I'm grabbing at straws here as I have no good explanation for what's happening over at yours, so this will take some diagnosing iterations before we hit it. 😢 That's worst case scenario and the reason why I requested you back up everything: when we don't know what's going on precisely, stuff can get worse before we get it fixed! Let me know if that Stay tuned for another Qiqqa setup tomorrow. (Will be late-ish as this diagnostics round uncovered other bad bits in testing here locally, which are getting fixed.) |
|
Hi again, My layout is exactly as you described, yes. My
On my HOME machine:
On my WORK machine:
Ok, so, same. But obviously the WORK machine's folder has a lot less in it, 5kb vs 4.5gb. Maybe if I again uninstall qiqqa on WORK, AND delete the user config files?? Edit: I "Forgot" the intranet library on WORK and it did NOT delete that folder (C:\Users\simon\AppData\Local\Quantisle\Qiqqa\INTRANET_0E4255D5-3AFB-4522-BDD4-74E0E5C23A73). So I manually moved the folder, restarted Qiqqa, joined the intranet library at NAS again, same issue.
The folder has been created afresh. Possibly deleting the full user config files folder will work though, I'll give it a whirl. Edit: still no. It does clean out some memories of configurations but the issue remains. The folder is created, again with the correct looking name. The mystery continues! |
|
Quick prelim release for you to try out: https://github.com/GerHobbelt/qiqqa-open-source/releases/tag/v82.0.7578.6369 When you install this one on both machines, it should present the proper Sync Point path ('\\NAS\Si Work\Papers & Books\QiqqaSync\Qiqqa.LibraryMetadata.s3db') as desired -- since you have still trouble with the OFFICE machine, this should be able to check my current working assumption that something on the OFFICE box corrupted/killed the Sync Point setting. |
|
Strange - no change in the path having upgraded the office machine only (so far): Log attached. |
|
Okay. The magic ingredient in that one is this chunk: So the machine is telling us it cannot read the .s3db file. At all, since it's not bad records bubbling up from a query, which would look quite different with the bleeding edge of Qiqqa as you have now running. That does, regrettably, not tell us yet why it finds the thing 'cannot be opened'. To help us diagnose that, I fear I'll have to do a bit more work on the code so Qiqqa can tell us a bit more about the why. Stay tuned for a new setup installer coming your way today. |
- when an SQLite error occurs, a full diagnostics report is included in the logfiles for further diagnosis by the user and/or engineer. + this is applied to both local database files (`*.library`)and the 'Sync Point' (`*.s3db`) to help diagnose jimmejardine#257 and similar. - dialed the Sqlite diagnostics up: Turn on extended result codes via `connection.SetExtendedResultCodes(true);` - Library integrity checks dialed up several notches: report and track data corruption in the log and pop an alertbox for user to decide to either abort the application or continue as-is -- which should be reasonably safe... - done the same on the Sync Point/Share Target side! .s3db is now integrity-checked on access. - disambiguated a few confusing log lines. - quite a bit of work done on metadata record recovery from corrupted/botched/hand-edited SQLite databases (like the ones I still have around from the time of the bad days of Commercial Qiqqa v76-v79, which led to: ) - some records can be recovered in a way as to treat them as pure BibTeX; adding the fingerprint from the first column as needed. - the above is a THIRD option for recovering the metadata and is useful when older databases have been touched by external Sqlite tools: it turns out the previous Qiqqa code did not behave well when a BLOB field was changed in subtle ways by these external tools, including the SQLite CLI itself. - stop yakking about 'don't know how to yada yada' for libraries which landed in the sync list but do not have a Sync Point: these libraries are now *ignored* (but reported in the log, in case sync was intended and we have a very obscure failure on our hands, e.g. jimmejardine#257 ) - add developer JSON5 option "BuildSearchIndex" to shut up and cut off the Lucene index building/updating background task. This was/is important as we're hitting the very limit of 32-bit memory with Qiqqa while testing with 30+ LARGE old databases in various states of disrepair: at a grand total of 200K+ PDF records, it looks like some re-indexing action will quickly drive us into 'Out Of Memory' fatal failure, while we test for other matters and are NOT interested in the index per se, right now. - LoadFromMetaData(): add addition sanity check to WARN in log about record key/document fingerprint NOT matching the fingerprint stored in the metadata BLOB. (This should be a RARE error, but indicative of DB data corruption / manual patching and important for diagnosis in your own log files.) - killed a couple of (even cycling!) 'unhandled exception' dialogs: these buggers are useless when we're already shutting down the app and should therefor only dump the error report in the log. - correction for the 'end of life of application run' checks and consequent log shutdown. An overall stability improvement!
|
Here's the new version to test for you: https://github.com/GerHobbelt/qiqqa-open-source/releases/tag/v82.0.7579.33985 Of particular interest to you & me in this case would be the new diagnostic report appended to every SQLite error: there we should find all available File System info plus a few tests and some preliminary diagnosis of the situation as done by Qiqqa itself. It'll all end up in the log files, so, as before, I am mighty curious to hear what the OFFICE machine has to say re NAS access now. P.S. note that I will have the day job to attend to the entire week, so things may get delayed or late as working hours are 0700-2200 or there-abouts inc. travel (remote real estate project). I'll try to look into anything you send ASAP anyway. Mighty curious what the heck is going wrong. |
|
Cheers mate, attached. Of interest based on my uneducated eye: "--> Is a normal file: False" "SQLite Error Code: 14 "--> VERDICT OK(?): this DOES look like a very normal file." Strange... Ger, absolutely ZERO need to explain nor apologise for delays buddy - I appreciate you're doing this in your spare time as a gift to the community and specifically for me (currently). Day job takes precedent, always, of course. Enjoy the travel - seems like a very unusual thing nowadays! Cheers. |
|
Thanks!
Just did a quick scan of your log file and the plot thickens... I'll
explain in a sec, but the low down of it is that I'll need a look at your
.s3db file and dig through it with SQLite tool as I still don't have an
explanation what's going wrong there...
Hmmmm. Thinking while writing this message... There's two things you can
do, the second of which might "magically" solve this (yup, sometimes
engineers wave a dead chicken and turn the prayer wheels; we're at that
level of non comprendo, alas.)
After those two items, I'll address your notes and the logfile you sent and
what it "says".
## Actions first:
1. If you're okay with sharing that file, I'm mightly interested in a
ZIPped copy of your .s3db file: if you can pack `\\NAS\Si Work\Papers &
Books\QiqqaSync\Qiqqa.LibraryMetadata.s3db` into a ZIP and send it to me (
ger@hobbelt.com; please mention 'Qiqqa' in email subject line or the spam
filters may kill it) so I can have a look at the binary when I have time
(next weekend, probably).
2. What I just thought about (also given your sync_md5, which we'll get to
in the log analysis section below) is that you MIGHT want to try this:
- DELETE or RENAME the `\\NAS\Si Work\Papers &
Books\QiqqaSync\Qiqqa.LibraryMetadata.s3db` file so that Qiqqa won't be
able to find it, e.g. name it `\\NAS\Si Work\Papers &
Books\QiqqaSync\Qiqqa.LibraryMetadata.s3db.bak1`. What Qiqqa on the HOME
machine SHOULD do, is on sync find out that the s3db is missing, copy an
empty template DB in its place and get busy syncing all from scratch.
(Remember those backups I said you should have made? This assumes your data
is in a safe place to get back if this results in 'bad things happening';
not that I expect but we have one 'weird' machine as we're getting crafty,
so better not risk loss of important goods while we EXPECT this to work
out, but cannot guarantee it right now.)
Anyway, the idea here is that you RESET the HOME machine's Qiqqa into a
state of mind where it believes that is has a Sync Point (`\\NAS\Si
Work\Papers & Books\QiqqaSync\Qiqqa.LibraryMetadata.s3db`) BUT also
discovers that it has NEVER synchronized yet and do a full copy to the NAS.
The way to accomplish that change of mind is 2-fold:
A: RENAME or DELETE the .s3db file at the 'NAS' side, so it will at least
not do *something* with a existing file which happens to make the OFFICE
box behave so very oddly. So we tickle Qiqqa (HOME) to create a fresh one
for us from scratch and take it from there.
B: on the HOME box, it is important to RESET Qiqqa into thinking "ow, sync
point but apparently I haven't copied a thing yet! Let's do that now!" and
that is accomplished by DELETING or RENAMING the corresponding .sync_md5
file in the local library directory on the HOME machine: the .sync_md5 file
is where Qiqqa 'tracks' what it has synced previously so it can 'changes
only'. We want EVERYTHING, so no previous work, so get rid of that
.sync_md5 file.
NOTE: one thing I haven't tested yet, ever, is how Qiqqa will react to the
PDF files already existing on the NAS, as Qiqqa should discover those are
already there when it starts to look at HOME + NAS to discover the
necessary sync actions (copy which metadata records, which PDF files, etc.)
EXPECTED end result is HOME syncing to NAS, taking a while maybe, but I
expect that machine to copy all metadata into a fresh .s3db file (which
should be present on the NAS once HOME has synced after killing the old
.s3db and .sync_md5 files.
OW, forgot to mention but just to make sure: DELETING/RENAMING those files
should be done **before** you start Qiqqa on HOME !
That's action 2 on HOME.
Now we hop over to OFFICE, should see a new `\\NAS\Si Work\Papers &
Books\QiqqaSync\Qiqqa.LibraryMetadata.s3db` from there, then
- we do the same DELETE/RENAME local .sync_md5 file
( C:\Users\simon\AppData\Local\Quantisle\Qiqqa\INTRANET_0E4255D5-3AFB-4522-BDD4-74E0E5C23A73\Qiqqa.sync_md5
) -- given the noises Qiqqa is making in the log, it's already gone. No
sweat if it is gone, but if it isn't now's the time to kill the bugger.
- next when you run Qiqqa on the OFFICE box, it should discover the library
with a sync point at
`C:\Users\simon\AppData\Local\Quantisle\Qiqqa\INTRANET_0E4255D5-3AFB-4522-BDD4-74E0E5C23A73\Qiqqa.sync_md5`
like it does now.
Since HOME has been previously enticed into creating a completely fresh
.s3db file for us, the OFFICE sync should run with no errors. If it errors,
well more logs to send me and bundle the 'new' s3db as well, I'd say. I
*hope* I get a 'it's working now!' message instead, because if this worse
case scenario is happening, you got me really stumped. (See also the
explanation about your logfile and your notes in your message. If this
fails without a clear error cause in HOME or OFFICE log, then we got a real
gremlin -- or me missing some horribly wicked detail.)
Anyway, long story short:
Action 1: consider sending me the whole kaboodle, i.e. s3db at least + logs
as you go.
Action 2: tickle Qiqqa on HOME to completely rewrite the NAS DB from
scratch by DELETING
`...\INTRANET_0E4255D5-3AFB-4522-BDD4-74E0E5C23A73\Qiqqa.sync_md5` on HOME
plus DELETING `\\NAS\Si Work\Papers &
Books\QiqqaSync\Qiqqa.LibraryMetadata.s3db` on the NAS. (RENAME is as good
as DELETE: Qiqqa must not find these files at the expected spots and create
them anew.)
Then run Qiqqa on HOME to sync, which should result in a new `\\NAS\Si
Work\Papers & Books\QiqqaSync\Qiqqa.LibraryMetadata.s3db` being created.
(Only bit I keep my fingers crossed about is the PDF files' sync as we did
not DELETE those off the NAS. *should* be okay, but this is the only risk
bit as far as I can tell.)
Once SYNC is complete, go over to OFFICE machine, check if the new s3db on
the NAS is visible,
DELETE the local sync_md5 if it happens to exist,
then start Qiqqa on OFFICE to start the sync there, which SHOULD now
SQLite-open the NEW .s3db file and not report an error.
So far, the actions. Logging review I'll do in the next message as writing
this as I go made this a pretty long text already.
Good luck! (You & me both :-))
Met vriendelijke groeten / Best regards,
Ger Hobbelt
…--------------------------------------------------
web: http://www.hobbelt.com/
http://www.hebbut.net/
mail: ger@hobbelt.com
mobile: +31-6-11 120 978
--------------------------------------------------
On Mon, Nov 2, 2020 at 6:48 PM Simon Dedman ***@***.***> wrote:
Cheers mate, attached. Of interest based on my uneducated eye:
"Could not find file
'C:\Users\simon\AppData\Local\Quantisle\Qiqqa\INTRANET_0E4255D5-3AFB-4522-BDD4-74E0E5C23A73\Qiqqa.sync_md5'."
Maybe normal on first sync attempt?
"--> Is a normal file: False"
Not sure if this is good or bad!
"SQLite Error Code: 14
Reported error code is NOT a SQLite Extended Error Code.
SQLite HResult: -21474816018:X"
"--> VERDICT OK(?): this DOES look like a very normal file."
Strange...
Ger, absolutely ZERO need to explain nor apologise for delays buddy - I
appreciate you're doing this in your spare time as a gift to the community
and specifically for me (currently). Day job takes precedent, always, of
course. Enjoy the travel - seems like a very unusual thing nowadays!
Cheers.
Qiqqa-20201102.173708.LOG
<https://github.com/jimmejardine/qiqqa-open-source/files/5476908/Qiqqa-20201102.173708.LOG>
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#257 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AADCIHTF34P2O7AXPIA6QBLSN3WGPANCNFSM4S36UMAQ>
.
|
OFFICE log: the important stuff starts with this: Sync process is starting. 'Guest' lib is classically now syncable, no surprises there. (This will change in a future release, but we'll stick with what we have right now.) So far, this is all 'good noises'. Expected output. Onwards! Qiqqa inspecting your second library on the OFFICE box. This is the one that comes with a Sync Point (we'll see in a sec). So far, good noises. All clear. Error. About the local Again, 'good noises', IF IT WEREN'T for me wondering aloud: "but Simon has had this setup for a while now! Did he nuke the sync_md5 on the OFFICE previously? (Which is A-okay, Qiqqa should just pick from there and do a full sync on OFFICE, but this is me right now wondering if you did this as part of the ongoing process (might have mentioned to you to do this a while ago) or if this also a 'surprise' to you. If it is a surprise to you, this merrits a "what the heck?" but otherwise won't be able to help us fix the situation. The actions mentioned in my previous message might resolve the matter. So all we can do now with this is raise eyebrows, wonder and continue with the rest of the logfile: That's the sync process starting for real. Just did a quick source code check about those (0/0: 0%) numbers: that's just internal progress counters and we're at the start, knowing nothing yet, so 0/0 is fine. Kaboom! There's that one again. Bad noise! We already know this, but this SQLite barfing a hairball about not being able to connect (not even running a query yet!) to the NAS DB: the s3db file. Your bleeding edge Qiqqa software now adds a diagnostic report, which we'll get into next: 'Archive' attribute. FileSystem legacy stuff from back in the olden days of MSDOS. Almost nobody does anything with that flag, anymore. Qiqqa and SQLite sure wouldn't bother about it being off or on. Here follows the rest of the detailed FileSystem info dump which I included in the diagnostics report. (Last one: that's the archive bit again. Just from another API. ok.) Nothing crazy so far. "Vanilla". Good noises. This one may indeed sound confusion, but it his is the OS telling us that the file is a 'normal file' without any attributes set. Since the (A)=Archive bit has been set, this is reported as "normal=false" at OS level, but nothing crazy or off about this. So, confusion aside, definitely not surprising to me. I should have written this one up as "file without any attributes set" to prevent confusion. Good catch in your notes. Anyway, no cause for concern. First time we see the size: this is straight off the disk from the Windows operating system: this is the number you'ld see in the Windows Explorer when you request the File Properties of the THIS is important as under hood that was me just READING every byte in the s3db file from NAS: this means that the file proven to be readable by the Qiqqa software, at least at a binary level! Meanwhile Qiqqa's built-in SQLite kicks up a ruckus, so from this (plus the next line in the log!), it looks like there's some trouble for SQLite about the database file not being in the correct format? Or is SQLite.NET doing things I don't know yet? I downloaded their sourcecode, but that quite a large chunk of code and I haven't had time to dig through that bunch yet, but this part of the diagnostic tells me Qiqqa can read the s3db, so, following logic, it must therefor be something in the format OR in yet-unidentified problems originating from the exact way SQLite opens a database file. We'll have to dig deep to answer that one. (Incidentally, this is why I refuse to work on projects with closed-source libraries: we you cannot read the sourcecode in cases like this, you've hit a very hard wall. There's a commercial crypto-library way back which cost me dearly in reverse-engineering/diagnostics cost (2 man-weeks of digging with sophisticated rev-eng tools) thanks to closed-source. Anyway, we'll have to dig into SQLiteNET if we want to get to the bottom of this, because my new diagnostics just completely the normal FILE READ process and found no fault what-so-ever: "Successfully scanned the entire file: length scanned = 27233280 bytes": no error report in this line, size reported MATCHES the OS-reported filesystem property 'size' from the previous line. As one would expect for any 'normal' file... Now that line was added by me as my diagnostics code goes through the motions at OS level to open the s3db file for WRITING instead of only READING. The OS just told us: "fine. here you go. Do your thing." Nevertheless, SQLiteNET comes back with "unable to open database file" so somebody is pulling my leg and all parties point at SQLiteNET by now... We'll have to investigate that one further, alas. This is me and my diagnostics report code telling us that SQLiteNET (which, you'll see below, has been configured to output the most elaborate set of errors possible: Given the report output, we also know that SQLiteNET didn't deem this worthy of additional error info: 'NOT a SQLite Extended Error Code'. So 'unable to open database file' is about all the information we get for this. Bugger. RTFC is in order to discover why. (RTFC=Read The F****** Code; geek-speak for if you want to know what's going down, the source code is your only chance. Very dark. So real. 😄 Let's have a look if there's more to learn from the log: All that stuff is me dumping every version number or other release markers I can get my grubby paws on to help me investigate if possibly maybe there's a DLL gone sideways or other installed software crap happenin'. Here? No problem at all. Clean as a whistle. All numbers match the ones on my dev box and are what your Qiqqa installer installed. No surprises there. Good noises. That's just me telling folks that this SQLite has been specifically told to turn on advanced error reporting via the API it has provided for that. Nothing special. Just added in the very last release you got from me. Onwards. That is me combining a few of the geeky bits about the file and saying "this is about what I expect to see at OS level for a file that Qiqqa might be working on". As you noted, here's that "normal" again, but this time it is me with my perspective, instead of a deep Windows API with a very specific, different perspective of what's "normal". Hence this is fine and leads me to conclude that the problem is somewhere hiding in the internals of SQLiteNET (or even SQLite itself!) and I have to find out what might be other reasons for SQLite to dump an error 14 "cannot open" in my lap on one machine, while another does just fine. I allude that, very indirectly, in the next bit of the logfile: For completeness, there's also question what Qiqqa has to say about the sync process after this just happened. So let's pick out the relevant bits of the rest of the log and read those: That's the code in Qiqqa catching that SQLite error, uttering some noises but nothing extra useful. From this we at least know that Qiqqa hasn't given up the ghost entirely, so we should see some more sync noises in the log... That last line is indeed Qiqqa telling us it's now done with SYNC of the SimonDedmanNASsync library: nothing on the remote side, a DB error at the start of that process, so nothing that can be done. Hence no mention of files or records processed. Expected, yet undesired. However, a possibly fun fact are the log lines before that one: that's ANOTHER library of yours, as far as I can tell ('QiqqaSync' instead of 'SimonDedmanNASsync') being synced successfully!? So the error seems very specific to a single library in a larger set, or am I reading this wrong? Anyway, like I said before: the plot thickens and I'll need to dig into SQLiteNET to maybe find out what's going on for real. Meanwhile, I hope you have a spot of luck with Action 2 (RENAME/DELETE s3db + sync_md5 files, reconstructing the sync that way). Keep me posted and thanks for hanging in there!) 👍 Bit of an aside, but that I'll need to do more work on this stuff. (Last weekend's slew of changes saw the old commercial Qiqqa cruft being regarding this being kicked out finally -- it's permeated through the source code. 😞 ) Folks should be able to assign or change the Sync Point of their library and point them at any NAS drive or Cloud Storage they may prefer instead to sync their libs across multiple machines. Alas, that's Future Music. right now we have a very odd Qiqqa library issue here.
|
|
😭 Last message near the end: I'm wrong. The 'fun fact' I mention there is me not watching out carefully enough: it's the same library, only the local side of its sync process still being able to collect the set of PDFs and then saying there's nothing to do (heck, yeah, s3db remote DB could not be opened, so no sync work can be done!) The Library name is SimonDedmanNASsync, while it's sync directory on the NAS is called QiqqaSync. That's fine. Github allows me to EDIT my messages, but I'll the above last message as it is; if I turn this in a document, that 'fun fact' paragraph (and question) should be scratched as it's completely wrong. Time to spin down here. Catch some Z's. 😴 😴 Best of luck with Action 2 (and 1 too, of course) and let me know how it goes. |
|
Cheers again Ger. I have a big work deadline this afternoon so will get
back to this immediately after.
S
…On Mon, 2 Nov 2020 at 15:33, Ger Hobbelt ***@***.***> wrote:
😭 Last message near the end: I'm *wrong*. The 'fun fact' I mention there
is me not watching out carefully enough: it's the *same* library, only
the *local* side of its sync process still being able to collect the set
of PDFs and then saying there's nothing to do (heck, yeah, s3db remote DB
could not be opened, so no sync work can be done!)
The Library name is *SimonDedmanNASsync*, while it's sync directory on
the NAS is called *QiqqaSync*. That's fine.
Github allows me to EDIT my messages, but I'll the above last message as
it is; if I turn this in a document, that 'fun fact' paragraph (and
question) *should be scratched as it's completely wrong*.
Time to spin down here. Catch some Z's. 😴 😴
Best of luck with Action 2 (and 1 too, of course) and let me know how it
goes.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
<#257 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ABDC7RGLL26MAXKN3OC5PDDSN46TDANCNFSM4S36UMAQ>
.
|
|
Hi bud,
Nonetheless, manually open intranet library from the menu, sync details, Start sync: same issue: Log attached. The confusion continues! :) Cheers mate |
|
That empty screen is very weird. At least the Guest lib should be listed
there, even if a Ui update would be buggy. This is definitely inexplicable
and indicative of a bug somewhere deep.
Havent got a clue yet but this is an important finding as it shows more
effects of what i expect to be a single underlying problem.
Writing this from off site, will only have time for this on saturday, so I
hope you have time then as well for a couple of iterations tovdig into this.
Can you mail me the zipped logfiles of both machines? What i am looking for
in there is (a) qiqqa startup logs where it reports about installed NET
versions etc and i want to compare your machines.
I also wonder what happened with the Guest library on your OFFICE machine
as that empty Home screen is theoretically impossible as qiqqa SHOULD have
created a guest lib if it doesnt exist already, so at least one row is
expected, not zero rows. (correct behaviour would be 2 rows at least: Guest
plus Intranet library)
From the sound of it now it looks like a machine specific problem so we
will have to also investigate outside qiqqa itself: which (. NET) libraries
etc have been installed and are there any errors with those?
This will take some digging. A very weird problem indeed.
PS does the Guest library exist on the OFFICE box? It is supposed to always
contain at least 2 articles about Qiqqa.
Thanks for reporting an for your persistence.
…On Thu, Nov 5, 2020, 03:04 Simon Dedman ***@***.***> wrote:
Hi bud,
Right, working through this now,
1. sqdb file just emailed to you
2/A/B. backup everything then rename Qiqqa.LibraryMetadata.s3db & sync
from HOME mechine to have it rebuild the s3db: just done. I did this a few
weeks ago, and it works flawlessly. HOME is back to normal.
2. In OFFICE: Forget intranet library in Qiqqa, delete that folder in
/Quantile/Qiqqa/ (inc sync.md5). Reopen qiqqa, join intranet library, full
sync.
Oddly when I join intranet library, I'm then greeted with a blank Home
screen:
[image: image]
<https://user-images.githubusercontent.com/4599748/98188023-2670e900-1f0a-11eb-8e50-906697c452c4.png>
Nonetheless, manually open intranet library from the menu, sync
details,
[image: image]
<https://user-images.githubusercontent.com/4599748/98188094-4d2f1f80-1f0a-11eb-96ff-5efa06e36182.png>
Start sync: same issue:
[image: image]
<https://user-images.githubusercontent.com/4599748/98188197-7bacfa80-1f0a-11eb-8678-74f7b3d8bb86.png>
Log attached.
Qiqqa-20201105.015324.LOG
<https://github.com/jimmejardine/qiqqa-open-source/files/5491455/Qiqqa-20201105.015324.LOG>
The confusion continues! :)
Cheers mate
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#257 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AADCIHQUCEPMDPAA6D54RP3SOIB27ANCNFSM4S36UMAQ>
.
|
|
Potentially related: since installing the older pre-release version I noticed my watch folder wasn't present. When I go to add it again, the NAS isn't present in Network, in the folder browser. But it's present & open & I'm editing files within it using Windows Explorer: Not sure why this would be, I'll restart to see if that helps, but this might be a clue regarding Qiqqa's problems connecting to the NAS sync library... |
|
Actually, looks like windows doesn't see the NAS now. Joy. Edit: fixed. Windows decided to turn SMB CIFS off, on a whim. Jerk OS. |
|
Re blank PDFs, I have this now on a recent paper: Page 6 is blank, but has a normal thumbnail and the text seems to be indexed since it shows up in searches. Edit: and then some minutes later it appears. Really odd. |
|
That's something I haven't encountered yet. Could you send me that PDF
file, please?
I'm pretty sure it'll be another SORAX library quirk, but I want to have a
look to see what might be special and how mupdf page renderer deals with it.
Thanks!
…On Mon, Apr 5, 2021, 22:02 Simon Dedman ***@***.***> wrote:
Re blank PDFs, I have this now on a recent paper:
[image: image]
<https://user-images.githubusercontent.com/4599748/113620323-180e3100-9652-11eb-90da-7b5a62b12ab4.png>
Page 6 is blank, but has a normal thumbnail and the text seems to be
indexed since it shows up in searches.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#257 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AADCIHRHNTXKNPWEXYSZVL3THIJMNANCNFSM4S36UMAQ>
.
|
|
Hi bud, related to this, I've noticed that the watch folder is missing when I open Qiqqa. That folder is on the NAS also, so it may be that Qiqqa can't connect to it quickly enough and gives up and then deletes the folder location in its settings? |
|
Qiqqa SHOULD never discard watch directories which are unreachable for
whatever reason.
SHOULD: haven't verified this in the source code right now (away from
computer), but then again I also don't recall any sort of "cleanup"
functionality either.
…On Thu, Apr 8, 2021, 00:21 Simon Dedman ***@***.***> wrote:
Hi bud, related to this, I've noticed that the watch folder is missing
when I open Qiqqa. That folder is on the NAS also, so it may be that Qiqqa
can't connect to it quickly enough and gives up and then deletes the folder
location in its settings?
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#257 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AADCIHTRDBKBKBP6KREYPBTTHTKX5ANCNFSM4S36UMAQ>
.
|
…or Qiqqa Sync and further investigation of jimmejardine/qiqqa-open-source#257, jimmejardine/qiqqa-open-source#319, ...) - discarded the unsupported luratech projects - discarded the Artifex/mupdf wix installer project; going to use inno setup for that - added another portion of the owemdjee libraries as MSVC projects -- just plonked the source files in there, did some very rudimentary setup/config work, so don't expect those to compile & run yet. This is about getting them into MSVC and towards building together as a unit.
… Qiqqa Sync and further investigation of jimmejardine/qiqqa-open-source#257, jimmejardine/qiqqa-open-source#319, ...) - Wrote the basic lockfile primitives - Got a start with Google Test to create a unit test set for this: it's too important to get it right to NOT do unit tests for this.
…mitives for Sync: lib_nas_lockfile. TODO: additional tools for performing the new Sync concept and testing viability on multiple platforms.
…k file locking primitives for Sync: lib_nas_lockfile. TODO: additional tools for performing the new Sync concept and testing viability on multiple platforms.
…k file locking primitives for Sync: lib_nas_lockfile. TODO: additional tools for performing the new Sync concept and testing viability on multiple platforms.
|
Just a couple of notes-to-self re this issue - still unsolved 😢 : Talked to someone today (off net, RL) who had tangled with quite similar issues regarding mapped network drives. His problems revolved around non-working MSAccess applications. Environment included BSD servers with Samba (SMB) over VPN and other environments.
|
|
|
Hey bud, happy winter. Have you had any further success with the MuPDF implementation? I'm basically unable to use Qiqqa at the moment because every pdf I open is blank. When I reopen qiqqa to that pdf, I get a blank or blurry first page and nothing else:
If I scroll away and come back, it's blank. If I close & reopen the pdf, blank. Looks like I'm using v82s; I think I rolled back due to other issues in other issue threads. But now I'm only making updates/edits/imports to Qiqqa at home, having given up trying to get the remote/NAS stuff working until I hear from you. Cheers! |
|
Hi Simon,
MuPDF is still being worked on (slow going, mostly due to me working
construction 1.5 years and selling the house, which finalized just a few
weeks ago). Not much progress to report as the late hours I got home have
mostly been spent on investigating how I should move forward with the whole
Qiqqa caboodle.
Reading this (and knowing there's several crazy issues with Qiqqa) I have,
as they say, *my work cut out for me* for next months.
Don't know how fast I'll get stuff done (pretty worn out so taking my
time), but I'm glad you're still here.
Re the NAS issue: many tests later, my conclusion is that thee only way
around is "work-around", which would be to copy the network files to local
temp directory, update them there and then pump them back onto the network
store. Easier said than done, because this would be a very non-atomic
operation and thus inviting a plethora of data loss if I'm not careful
about it: the moment I write back to NAS/network/remote, I neeed to make
darn sure nobody else, not even a background process anywhere, has touched
those files in the meantime. And network-storage file locking is
notoriously b0rked since its inception. A fundamental library to tackle
that *very important* detail has been developed (
https://github.com/GerHobbelt/lib_nas_lockfile) and that will be an
integral part of the coming "work around" for the Qiqqa Sync problems.
(I'll also need it for any "future Qiqqa" that's in the works as the
networked file storage/sync problem is a very fundamental one, so that's
development effort not wasted in the long run.)
Met vriendelijke groeten / Best regards,
Ger Hobbelt
…--------------------------------------------------
web: http://www.hobbelt.com/
http://www.hebbut.net/
mail: ***@***.***
mobile: +31-6-11 120 978
--------------------------------------------------
On Thu, Dec 9, 2021 at 11:24 PM Simon Dedman ***@***.***> wrote:
Hey bud, happy winter. Have you had any further success with the MuPDF
implementation? I'm basically unable to use Qiqqa at the moment because
every pdf I open is blank. When I reopen qiqqa to that pdf, I get a blank
or blurry first page and nothing else:
[image: image]
<https://user-images.githubusercontent.com/4599748/145485050-c499a95d-4c14-41b2-af08-80d4e9974865.png>
If I scroll away and come back, it's blank. If I close & reopen the pdf,
blank. Looks like I'm using v82s; I think I rolled back due to other issues
in other issue threads. But now I'm only making updates/edits/imports to
Qiqqa at home, having given up trying to get the remote/NAS stuff working
until I hear from you.
Cheers!
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#257 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AADCIHR3RTOF6LR74WXMZ7LUQEUBTANCNFSM4S36UMAQ>
.
|
|
Hi @GerHobbelt !
What do i do? |
|
FYI: we've very recently discovered a potential cause for this sync issue (SQLite error 14, when using a NAS). More info later on Monday, but the TL;DR is this: looks like there's a very nasty issue with MS windows local caching for SMB / SAMBA network shares (which is offered by many NAS'es out of the box) in combination with SAMBA oplock (opportunistic locking) settings. We've run into this issue in a completely different environment, not Qiqqa related, but I strongly suspect that this is cause for SQLite error 14 as well when Qiqqa attempts to sync the local database with the one located on the NAS / network-share. This is completely random behaviour: you may experience this, or you may not. Consequences can range from the error 14 to file corruption (which is what we are investigating for a DBase4 environment ATM.) More to follow in a few days. TL;DR of this TL;DR: any syncing on NAS shares is at risk of error 14 SQLite errors (or worse), rarely but unpredictably and a proper solution will require fundamental changes to the Qiqqa software.
The same fundamental risks/issues are expected with any drive-mapped cloud storage as well: DropBox, OneDrive, GoogleDrive, etc. |
|
Credits for SMB / SAMBA issue discovery, root cause analysis and initial work-around (disable SAMBA oplocks): @Lion251 (Henk de Leeuw). Qiqqa will need a different approach as Qiqqa will not have full control of the SAMBA configuration nor MS Windows system settings on every site; a more rigorous fix is required for our sync issues (not just this issue #). |
|
Hi @GerHobbelt ! I am very glad that you responded so quickly and despite everything, the improvement of Qiqqa continues. I want to let you know that while I was enjoying a couple of extra days off, my colleagues solved the problem by installing an older version of v.80s. I hope you manage to overcome this annoying issue. |
|
Hi mate, hope you're well. Very small diagnostic element: blank page problem: Qiqqa was working fine with 3 papers open, only 2 of which I'd interacted with since opening the program, both of which are quite big. Opening another paper was blank, and then the previous papers were too. Wondering if it's a memory issue?? Or similar allocation/space kinda problem? Possibly compounded by my network setup? TBH now I'm WFH full time I should just migrate my Qiqqa setup to the windows install and be done with the network setup... |
|
To add to this: I jumped down a paper to the bottom, which loaded, then scrolled up and got teh blank page, and an unresponsive Qiqqa. When I minimise the window, it disappears, and seems to have crashed. |
|
Hi Ger, hope you're well. I wondered if you might have any idea why something's not working for me -
However this results in no papers after syncing, despite there being 8gb of stuff in that folder:
Any ideas why this might me? I'm connected to my NAS via RaiDrive mapping the NAS to a local folder, which I can navigate to fine in Windows Explorer. Cheers for any ideas bud! Edit: looks like my main Qiqqa, in Win10 in a VM, which was syncing before I believe, is now greyed out. Possible a version issue with v80? |
|
Quick answer, so YMMV:
the directory listing shows a '...s3db' file: this is a Qiqqa metadata
database file BUT it's the database variant for the 'sync point' i.e. for
the REMOTE directory that's synced with the *local* database.
What I would do/try is this:
- FIRST making a BACKUP of that directory you showed (would be a lot of
tears if the subsequent initial sync would inadvertently decide the remote
has stuff that "should be deleted"; I haven't tested the functionality in a
while, so better safe than sorry)
- then go and create a NEW local library on the C: drive (or any local SSD
or harddrive on your new machine)
- then point *that* new (empty) library's sync point at the remote NAS path
you showed above.
- run a sync
That *should* entice qiqqa into comparing local (`*.library`) with remote
(`*.sdb`) databases and create a sync list, which it will then use to copy
all the documents and metadata across.
## Background + Asides:
Qiqqa expects a metadata database in each local 'sane' library, called
something like `*.library`, i.e. a database file (really an SQLite
database) with the `.library` filename extension. When i's not there, it
considers that directory an invalid/empty library, which would explain what
you're seeing.
Ditto for remote library copies: there Qiqqa looks for a `*.s3db` file as
the database storage, so any 'valid remotes' should have an `.s3db` file
(that must be a valid SQLite database).
The two databases store almost the same data and have different layouts, so
you can't outsmarten Qiqqa by renaming a `.library` file to `.s3db` or
vice-versa.
A *clean* remote library directory tree SHOULD NOT have a `.library` file,
which might have been created when you pointed Qiqqa directly at that
directory as being your library basedir: tread carefully with deleting
files (as always), but it might be handy to check if Qiqqa added such a
(small!) `*.library` file on the NAS there and then delete it; the rest
should remain as those documents are stored in the documents subdirectory
and referenced by the `*.s3db` database file.
Another potential hazard is that I don't see a `documents` subdirectory
directly, which MAY indicate that you pointed Qiqqa at the parent directory
instead at the library sync directory itself: the `.s3db` is rather small
to my taste, but YMMV as I haven't looked at that part of the code lately
and my memory is unreliable. ;-)
Met vriendelijke groeten / Best regards,
Ger Hobbelt
--------------------------------------------------
…On Thu, Oct 12, 2023 at 7:47 PM Simon Dedman ***@***.***> wrote:
Hi Ger, hope you're well. I wondered if you might have any idea why
something's not working for me -
I've just installed Qiqqa v80 on a windows machine, and am trying to
connect to my intranet library, which is the same location as previously
detailed above in this thread, \NAS\Si Work\Papers & Books\QiqqaSync
[image: image]
<https://user-images.githubusercontent.com/4599748/274685899-9a5bbcc0-5c92-4a4e-b5b8-151ab6e9b3b6.png>
However this results in no papers after syncing, despite there being 8gb
of stuff in that folder:
[image: image]
<https://user-images.githubusercontent.com/4599748/274686018-f4f7ed8d-750b-41b7-9a18-9b8015a5a44a.png>
Any ideas why this might me? I'm connected to my NAS via RaiDrive mapping
the NAS to a local folder, which I can navigate to fine in Windows Explorer.
Cheers for any ideas bud!
—
Reply to this email directly, view it on GitHub
<#257 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AADCIHSQQU535MF5A5FNGFLX7AUJVANCNFSM4S36UMAQ>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
Screenshots of issues above. Setup is: watch folder & qiqqasync directory both in a local network directory which opens and can be edited fine, as far as I can tell. I'm able to setup the intranet library but when I try to full sync I get the above errors.
I have another machine with the same setup and it works fine. I'd been using that machine (home) all day, then tried to sync the other machine (work). Qiqqa was closed on the home machine.
Any ideas gratefully received. Cheers!
The text was updated successfully, but these errors were encountered: