Initial testing #3
Comments
|
I deleted a book. And transferred a book :D I haven't really changed anything @NiLuJe , so if it doesn't work, make sure Calibre is listening, and the ports aren't being blocked. Issues so far:
|
|
I'll do a round of testing tomorrow ;). What are you currently using to generate the thumbnails? I don't know if that's doable in this context, but it might be worth letting Nickel handle all that, because otherwise, doing it right is going to be expensive, no matter what ;). |
|
I'm using a Go library. Just noticed that I asked for it to use Lanczos algorithm. Perhaps (ok, almost certainly) this was a tad ambitious, and the quality is probably lost on such small images anyway. Will try switching to bilinear and see if that helps (it will, just not sure by how much...) I really want to get thumbnail generation working at an acceptable performance level. Nothing more annoying (for me) to wait at the library screen while Nickel takes its sweet time generating thumbnails... I'm using a go routine for image generation, so it doesn't block the rest of the code, at least not until we're ready to exit. So long as I can get performance good enough that it's all done by the time the user "ejects" from Calibre, that's good enough for me. Note when you are having a go, the correct way to exit is to "eject" the device in Calibre, like you would with a USB connection. |
|
So, I've got thumbnail generation to an acceptable performance level. Mainly by deciding not to bother with the full cover at all. KU now requests a (much) smaller cover size (the dimensions of the Also fixed an issue around updating the Nickel DB with NULL values. I was sending the string And I haven't had any troubles exiting and returning from Nickel. Thanks for making the shell scripts so much more robust @NiLuJe |
|
After a false start because I have no idea how Go works, and despite a I'd forgotten that you had to enable a service in Calibre for the Wireless thingy, which probably explains my earlier failures (speaking of, that probably warrants a user-visible warning on-screen, something along the lines of: couldn't connect to Calibre, check that it's running and the Wireless connection service has been started).
You don't need explicit timestamps, the syslog handles those ;).
|
|
Yeah, when Calibre is down, I see a |
|
Aaaaand, semi-success: I apparently successfully downloaded a book, I can see it and everything, but I get a "Oops! This document couldn't be opened" warning when I actually try to read it ;). I did get the processing pass, I did get the second KU run (it also took a few minutes to start). I couldn't track everything because I of course had to take a call right at this moment -_-". Side-note: this is a KePub. I have both an ePub and a KePub for this book in my Calibre library, and it preferred the KePub (which is fine by me). That may be because of my format preferences in Calibre, in which case, perfect ;). |
|
FWIW, I can open that same book on my Forma, where I transferred it the usual way. |
|
I haven't actually tested kepub yet. it SHOULD work, but obviously there may be some kinks. The reason it takes so long to before it connects to Calibre, is because before connecting, I gather all the metadata, which is a 2.5 pass process. First, if it exists, I read the As you can see, depending on how many books you have, and how in sync the Nickel DB and calibre cache file are, this step can take a while. I'm not sure how to speed this step up, and still present calibre with an accurate list of books on device (as seen by Nickel). As far as preferring kepub, that's because that's the extension listed first in the code. I should probably make that configurable. As far as the timestamps on the logs go, that's automatically added by Go's default logger which I'm using. Agreed I need to better display the state to the user. It's the book corruption I'm most worried about however. Will need to look into that. Thanks so much for helping to test this. |
|
Ok, sent a kepub to my H2O, and it opened fine. So not a kepub issue. If the kepub is a known good file that Nickel reads correctly, then there's probably some corruption going on in the book saving code, or networking code. Joys... |
|
Okay, take two: Still no dice on that book. The main content DB entry looked like this the first time around: Here's what Nickel says when it fails to open it: And here's the entry from the second send (I did tweak the book a bit, because of a page-break issue I caught on the Forma), where it gets assigned to book 6 of (Not sure if relevant: that's probably the latest Series created/updated on that device, and the latest book sent to the device before that. If not the very latest, in the latest batch, at least). No issues with another book, in English, part of a series. It opens and was assigned to the proper series :). |
|
Oh, and if I actually read the error messages instead of just copy/pasting 'em: the failure to open it definitely appears to be because of the bang ;p. So I guess sanitizing filenames to be FAT-friendly should be enough? No idea what happened with the Series metadata the second time around, though. FWIW, I deleted the first copy from within Nickel, and not via Calibre. |
|
And about the startup delay: progress bar? ;p. I get that keeping things in sync can be tricky, but maybe some stuff could be cached? And then naively checked for changes via timestamps, as that's probably going to be the fastest. Walking the full onboard tree will take its toll no matter what, though :/. Also, no idea how feasible that would be, but if the metadata pass could avoid having to re-do the whole thing that would probably help, too ;). |
|
Thanks for continuing your testing. I had hoped that Calibre would sanitize the filename before sending it. Seems I'll have to do it on device instead :( As far as the wrong series info, that was a bug I fixed locally last evening, I noticed it myself. Simple silly error of forgetting to null variables in a loop. And I specifically don't walk the onboard tree, merely check the DB. The only time it should open a book is if the book exists in Nickel, but not in the metadata cache. Will instrument this to make sure though.I'll definitely have a bit of a think on how to speed this up though, as currently it is definitely slower than I'd like. Thanks for the continued testing! |
|
Well, here's the relevant comment from the calibre driver that it doesn't sanitize filenames: Going to have to have a bit of a rethink as to how to handle this. There looks to be two possible options. I can ask calibre to send UUID filenames. Or there's the option |
|
Well, that wasn't too hard. The driver API accounts for this scenario, and allows us to change the 'lpath' if the Calibre generated one isn't suitable. Hopefully shermp/UNCaGED@cb4fc32 and 41798b2 fixes the filepath issue. Since this fix updates UNCaGED as well, you will probably need to update that manually. A Next up, will have a think about how to make metadata collection/management more efficient... |
|
Ok, bff9e8a should dramatically speed up metadata collection. In hindsight, performing individual single row DB queries for potentially hundreds or even thousands of books really was a rather silly idea wasn't it? Now there's only one multi-row query out to the DB, and I create the final map there instead. Both simpler, and MUCH faster. |
|
Success! :) |
|
Finally have printing to screen working properly. It's probably a little (ok, more than just a little) fancier than is strictly required, but hey gotta use at least some of my FBInk feature requests :p If you want to test it, I've updated go-fbink-v2 with a new function to aid printing a Go I have to say, the Anyways, it's working now, and everything looks a lot nicer. |
|
On a related note, how much faster is providing print_raw_data with an 8bpp grayscale image instead of an RGBA image? If the speed up significantly faster, it might be worth it for me to switch to using that. |
|
It won't be in that context, as the FB is 32bpp, so it would ultimately be
expanded to RGBA again ;).
So starting with RGBA input is the right thing to do here ;).
Unless working with RGBA on your end is so much slower than working in Y8, and that despite the cost of a single conversion to RGBA at draw time, it would still be faster, but I doubt it ;).
…On Tue, May 7, 2019, 14:01 Sherman Perry ***@***.***> wrote:
On a related note, how much faster is providing print_raw_data with an
8bpp grayscale image instead of an RGBA image? If the speed up
significantly faster, it might be worth it for me to switch to using that.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#3 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AAA3KZQMMPAQ2432RTB3N3TPUFVSJANCNFSM4HKXERBA>
.
|
|
Yeah, I've been caught unaware by *align stuff, mainly when I forget that I've set it earlier... >_<". c.f., the dump/restore test for fun interactions between that and the H2O viewport ;p. Besides not forgetting to reset the flag and/or bracket stuff that needs with a set/reset, one could also theoretically use a different FBInkConfig instance, but that's potentially trickier, unless you set it up at the same time you _init/_reinit, so it gets the same values that _init will enforce in there... I usually temporarily switch to full-screen updates to make sure I don't do something stupid when I start playing with stuff like that ;). |
|
Gave it a go, looks neat :). Ann here's a couple of nitpicky suggestions on how to make use of that space ;). I'd probably clear the footer on Eject in the first pass, and possibly print something there in the metadata pass when it's over. |
|
Sidebar: I have no idea how the go get/build stuff works, but the build script will probably need some tweaks to make sure it always pulls/builds the latest stuff, even on rebuilds ;). (Also, the same kind of info pointed out in the go-fbink-v2 README about how to set up the env probably needs to be shown somewhere ;)). |
|
Yeah, I'm aware some the content needs more work, Yesterday was about making sure the thing actually functions properly :) Not sure about downloading updates from the build script. If people are doing any dev work, they may want to specifically use their current version on disk, and not update on every run of the build script. Will have a think about it. |
|
Have you got any ideas on making the
|
|
I have no idea how those are supposed to work, but a generic "are we connected" check might be enough? (i.e., ping/check if there's a gateway/try to read something over HTTP). I think all of those are implemented one way or another in KOReader ;). |
|
Determining if we have an IP address should be easy enough. Parsing But there's no point trying to get an IP address if wpa_supplicant doesn't connect to a Wifi network. That's what I'd like to try and determine if possible. |
|
IIRC, it logs what happened to the syslog. Granted, parsing that wouldn't
be a terribly pretty solution ;).
…On Wed, May 8, 2019, 08:16 Sherman Perry ***@***.***> wrote:
Determining if we have an IP address should be easy enough. Parsing ip
addr show ${INTERFACE} should do the job there. KU doesn't need an
internet connection, just a LAN connection.
But there's no point trying to get an IP address if wpa_supplicant doesn't
connect to a Wifi network. That's what I'd like to try and determine if
possible.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#3 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AAA3KZUPGONDOQYNWCEW4JTPUJV5FANCNFSM4HKXERBA>
.
|
|
If this error doesn't appear after the first import, I don't think it is an issue. @shermp? And ... what exactly is @kobothingy doing? 😕 |
|
FWIW: |
|
Right, after doing a bit of repository maintenance, I can actually start looking at things. While somewhat harmless, I do consider that an actual error. Basically, if you are replacing a book, it means its in the metadata map, and therefore it SHOULD also be in the database. That check is supposed to be a just-in-case test. I'll have a look to see if anything jumps out at me. |
|
That was on an initial import of a book (well, not exactly initial, as it was on the device at one point, but had been deleted from the UI, which should pretty much wipe any trace of it). |
|
Hmm, so that would indicate to me that has snuck (back) into the metadata cache somewhere. I'm having another look over the "read metadata" code now. |
|
Just another thought, did you send the book more than once in the same session? That could also explain the behavior. |
|
I don't think so? I can reproduce it pretty much 100% of the time across different sessions, and I should only be sending books once per session, unless there's something seriously wonky with my "Send To Device" button ;). Let me try with a file that the device's actually never seen, ever ;). |
|
That works better when I don't forget to launch the service... I got to see the new "You're stupid and you forgot to enable the Wireless Service" message, at least :D. But alas, same thing with a book I'm fairly sure the device's never ever heard of (also, an ePub and not a KePub, in case that makes a difference). EDIT: And still no full-screen cover, either. Putting the device to sleep, I don't even get to see the placeholder text-only cover get replaced on-the-fly, like you usually do for unprocessed side-loaded content, it just happily appear to be directly upscaling one of those tiny thumbnails :/. NOTE: I don't recall if I ever tested this in my original tests, though. Given how much I care about this stuff, I would think I did, but, err, this is starting to make me wonder... |
|
Aww crap. I'm beginning to wonder that if Nickel sees any type of pre-generated thumbnail, but no full cover, it uses the thumbnail instead of whatever's in the ebook :( Starting to wonder if finding a small C resize library with ARM NEON optimizations might be the way to go. |
|
@shermp it may be even better to port rez's assembly resizing: https://github.com/bamiaux/rez/blob/master/vscalers_amd64.s https://github.com/bamiaux/rez/blob/master/hscalers_amd64.s. Update: Actually, I don't think it would be possible due to golang/go#7300. Another update: Maybe it might if CL 57470 has all the instructions. |
|
I ripped out Qt/imlib2's scaler for FBInk, and it has a marginally faster NEON implementation (as in, faster than its own C implementation, no idea how that compares to what you currently have), FWIW, c.f., https://github.com/NiLuJe/FBInk/tree/master/qimagescale |
|
My knowledge of assembly is essentially zilch. Let alone SIMD stuff. So it wouldn't be me doing the porting! |
|
Facepalms Yeah, that SQL error is harmless. The function that produces is called regardless of whether it's actually needed. Need to tweak tweak the logic slightly so it doesn't produce an error though. |
|
I'm wondering if it's not a combination of populating the Nickel db AND shipping partial thumbnails. I'd need to check if manually sideloading an ePub and manually shipping only a few of the thumbnails behaves the same. That vaguely looks like something I might already have done in the past, and I fear that the answer is, yeah, it's all or nothing. But, still, FOR SCIENCE! :D. |
|
Note, that KU does not actually add any new rows to the Nickel DB. Or modify anything to do with images in the DB either. The only thing we do regarding images is create the directory structure that nickel would create and populate (some) of the images. |
|
Random fun fact: I can't shutdown my H2O :D.
|
|
Can you |
|
Probably not easily, as it appeared dead to the world, except for the power button (unlike the halt deadlock, where I had to pinhole hard reboot it). By "crazy", I meant it properly turned off after a while, then started flickering on and off at a very slow frequency (like very sluggish keyboard LEDs during a kernel panic, so I'm going to assume it was a kernel panic ;p). |
|
Now I probably know why the SD card is mounted RO: because you don't have any other solution than ripping it out like a savage :D. |
I hope this is not a common issue, or an issue with KU! |
|
Nah, probably one of the the usual "everything is racy as hell" NTX kernel issue ;). |
|
Okay, yeah, bad news: same "no fullscreen cover" behavior when sideloading stuff by hand with half the thumbnails :/. I took the opportunity to try ImageMagick's version of Starting from a fairly narrow cover @ 1391x2200, I ended up with a FULL @ 355x561 and a GRID @ 149x236. Not quite sure where each of 'em is used, but it's pretty much only downscaling at display time, which is good :). Crappy test script to make sure I did things right: #!/bin/env python2
import os
import sys
def qhash(inputstr):
instr = b""
if isinstance(inputstr, bytes):
instr = inputstr
elif isinstance(inputstr, unicode):
instr = inputstr.encode("utf8")
else:
return -1
h = 0x00000000
for x in bytearray(instr):
h = (h << 4) + x
h ^= (h & 0xf0000000) >> 23
h &= 0x0fffffff
return h
ContentID = "file://" + "/mnt/onboard/TEST/" + sys.argv[1]
ImageID = ContentID.replace('/', '_')
ImageID = ImageID.replace(' ', '_')
ImageID = ImageID.replace(':', '_')
ImageID = ImageID.replace('.', '_')
hash1 = qhash(ImageID)
dir1 = hash1 & (0xff * 1)
dir2 = (hash1 & (0xff00 * 1)) >> 8
path = "/run/media/" + os.getenv("USER") + "/KOBOeReader/" + ".kobo-images/"
path = os.path.join(path, "%s" % dir1, "%s" % dir2)
im_args_pre = "-colorspace Lab -filter LanczosSharp -distort Resize"
im_args_post = "-colorspace sRGB -grayscale Rec709Luminance -colorspace sRGB -dither Riemersma -remap /home/niluje/SVN/Configs/trunk/Kindle/Touch_Hacks/ScreenSavers/src/linkss/etc/kindle_colors.gif -quality 75 png:"
im_args_pre_lb = "-colorspace sRGB -background black -gravity center -extent"
im_args_post_lb = "-grayscale Rec709Luminance -colorspace sRGB -dither Riemersma -remap /home/niluje/SVN/Configs/trunk/Kindle/Touch_Hacks/ScreenSavers/src/linkss/etc/kindle_colors.gif -quality 75 png:"
print("{}".format(path))
print
print("convert cover.jpg {} {} {} {} {}'{}'".format(im_args_pre, "1080x1429", im_args_pre_lb, "1080x1429\!", im_args_post_lb, ImageID + " - N3_FULL.parsed"))
print("convert cover.jpg {} {} {}'{}'".format(im_args_pre, "355x530\^", im_args_post, ImageID + " - N3_LIBRARY_FULL.parsed"))
print("convert cover.jpg {} {} {}'{}'".format(im_args_pre, "149x223\^", im_args_post, ImageID + " - N3_LIBRARY_GRID.parsed")) |
|
I'm thinking of releasing another public version with all the changes and improvements that have been made. @NiLuJe when will you be ready to release a new version of kfmon? SD card users will need a version with the slightly relaxed mountpoint change limits. |
|
I'm away for a few days, and I was in the middle of refactoring some stuff
in kfmon, but I should be back basically early this week end to finish it I
think ;).
…On Thu, May 30, 2019, 00:38 Sherman Perry ***@***.***> wrote:
I'm thinking of releasing another public version with all the changes and
improvements that have been made.
@NiLuJe <https://github.com/NiLuJe> when will you be ready to release a
new version of kfmon? SD card users will need a version with the slightly
relaxed mountpoint change limits.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#3?email_source=notifications&email_token=AAA3KZQGEGVU7GVLKPJO5N3PX4AW5A5CNFSM4HKXERBKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODWQ2VJI#issuecomment-497134245>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AAA3KZSUW5CTIPOLAJ72IUTPX4AW5ANCNFSM4HKXERBA>
.
|
Thanks for the update. I've decided to do a bit of refactoring (mostly around code organisation, I'm moving stuff out of main into separate packages), so no rush. |
|
I've finally released KFMon 1.3.0 ;). Relevant to your interests, it will pickup new configs right after an USBMS session, meaning new users won't have to reboot after a brand new UNCaGED install (... as long as it's done via USBMS, and not through other, more esoteric means ^^) ;). The release was slightly delayed because of a last minute experiment with replacing the boot progress bar, after I realized it was chewing through > 50% CPU on my H2O, which seemed a bit much to essentially draw 5 big pixels in the middle of the screen ;p. |
|
Nice! I'll be suggesting that as the recommended minimum version for the next release (When I get time to finish my current PR). |
Starting an issue thread to gather early testing results.
@NiLuJe if you have any issues or comments, leave 'em here. Probably better than continuing the discussions in PR's
The text was updated successfully, but these errors were encountered: