[Enhancement] [Kobo] one-step installer

[pazos]

Project: integrate a (minimal)launcher based on inotify (like fmon) and koreader in KoboRoot.tgz
Goals: provide an alternative for new users, or users that don't like to install a full blown launcher, like KSM.

launcher is coded in c, it just wait for IN_OPEN events on /mnt/onboard/koreader.png
when our target is opened a helper script is launched. The helper script finds a match for file:///mnt/onboard/koreader.png in nickel's db and runs koreader just in that case.

I submit a dirty way of doing this, as an example:
kmon-sample.zip

What is your opinion?

[pazos]

There are some typos in the code, but I hope the concept is clear :)

[NiLuJe]

Just for the record, since I've never used KSM nor fmon:

• PNG files dropped somewhere in onboard are parsed by Nickel and create dedicated tiles? If not, I don't exactly get how one would "launch" stuff ;).
  Same question if it only creates tiles and not entries in the Library: what happens once the tile gets phased out by others?
• Can't remember if the FW ships w/ the sqlite3 client, but doing a proper SQL query (either via the CLI client, or merging everything in the C code and using libsqlite3 directly) seems "cleaner", as a goal ;).

[pazos]

I don't understand SQL at all, but the PNG seems to be referenced in some places:
grep file:///mnt/onboard/koreader.png /mnt/onboard/.kobo/KoboReader.sqlite | wc -l
= 17

grep faulkner /mnt/onboard/.kobo/KoboReader.sqlite | wc -l <-- just one ebook
= 179

[NiLuJe]

sqlite3 /mnt/onboard/.kobo/KoboReader.sqlite "SELECT EXISTS(SELECT 1 FROM content WHERE ContentID = 'file:///mnt/onboard/koreader.png' AND ContentType = '6');"

Returns 1 if found (i.e., in the Library), 0 otherwise.

[NiLuJe]

Note that, yep, my memory wasn't wrong, the Kobos do NOT ship w/ the sqlite3 client...

┌─(ROOT@(none):pts/0)───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────(/)─┐
└─(0.54:53%:06:32:86%:#)── ls -lash $(which sqlite3)                                                                                                                                                                                                        ──(Thu, Apr 28)─┘
     0 lrwxrwxrwx    1 root     root          39 Mar  2 23:19 /usr/bin/sqlite3 -> /mnt/onboard/.niluje/python/bin/sqlite3

[NiLuJe]

And to answer my initial question, since I've now checked: yeah, PNGs create both a tile and an entry in the Library....

(... and are wrongly assigned the application/x-cbz mimetype, but, hey... :D).

[NiLuJe]

TL;DR: I'm all for it!

We have non-intrusive ways to launch daemons at boot, so, yeeep, I like it.

[NiLuJe]

And whether opened from a tile or the Library, the string of inotify events is the same:

/mnt/onboard/koreader.png OPEN 
/mnt/onboard/koreader.png ACCESS 
/mnt/onboard/koreader.png ACCESS 
/mnt/onboard/koreader.png ACCESS 
/mnt/onboard/koreader.png ACCESS 
/mnt/onboard/koreader.png ACCESS 
/mnt/onboard/koreader.png ACCESS 
/mnt/onboard/koreader.png ACCESS 
/mnt/onboard/koreader.png ACCESS 
/mnt/onboard/koreader.png ACCESS 
/mnt/onboard/koreader.png ACCESS 
/mnt/onboard/koreader.png ACCESS 
/mnt/onboard/koreader.png ACCESS 
/mnt/onboard/koreader.png ACCESS 
/mnt/onboard/koreader.png ACCESS 
/mnt/onboard/koreader.png ACCESS 
/mnt/onboard/koreader.png CLOSE_NOWRITE,CLOSE

(I happen to have an inotify-tools build in my stuff ;p).

[NiLuJe]

The SQLite API looks dead simple and easy to use, so I'm all for doing everything in C and ditching the extra wrapper script :).

[NiLuJe]

Speaking from experience on Kindles, one question I'm left with is what happens to inotify watches when onboard gets unmounted to be exported over USBMS?

[houqp]

Speaking from experience on Kindles, one question I'm left with is what happens to inotify watches when onboard gets unmounted to be exported over USBMS?

We should probably reinitialize the watch every time it's mounted again...

[KenMaltby]

If this, like fmon, is dependent upon Nickel's processing of the .png, to put it in the db and library, then it retains the obstacle most new users fall prey to. If you have totally automated the process, so it functions without requiring user intervention, that would make the install more reliable.

[NiLuJe]

What obstacle would that be? (never used fmon, sorry for rehashing old
stuff ;))
On Apr 28, 2016 2:35 PM, "Ken Maltby" notifications@github.com wrote:

If this, like fmon, is dependent upon Nickel's processing of the .png, to
put it in the db and library, then it retains the obstacle most new users
fall prey to. If you have totally automated the process, so it functions
without requiring user intervention, that would make the install more
reliable.

—
You are receiving this because you commented.
Reply to this email directly or view it on GitHub
#1996 (comment)

[KenMaltby]

Where users most often ran into problems, when using the install guides I posted at MR, was with getting the .png files properly processed for fmon's use.

[NiLuJe]

Guess we'll have to check if putting it in the KoboRoot tarball gets it
processed at boot?
On Apr 28, 2016 3:15 PM, "Ken Maltby" notifications@github.com wrote:

Where users most often ran into problems, when using the install guides I
posted at MR, was with getting the .png files properly processed for fmon's
use.

—
You are receiving this because you commented.
Reply to this email directly or view it on GitHub
#1996 (comment)

[KenMaltby]

Nickel's processing is triggered upon return from a USB connection. This makes the monitoring process a two step install process for the user.

Yes, if it could be checked before the monitoring is implemented.

You know, the Nickel processing step that scans for new media files, might be bypassed and the referenced file added directly to the db and library, that would make it "one step".

One thing to keep in mind might be the problem the "old" fmon based install method encountered where a loop was established that required removing the koreader.png file to get back control of the device.

[NiLuJe]

The idea behind the SQL query would be to avoid such cases. (Note to self: for the same reason, check IN_OPEN events and not IN_CLOSE, to avoid catching the eventual CLOSE of nickel's processing, at which point the DB may already be updated, but we certainly don't want to start anything).
I'll double-check, but I don't think we trip anything when removing the PNG from Nickel? So that would take care of the looping issue.

I thought about adding the SQL entry ourselves, but that seems a bit overkill when nickel does it perfectly well and with user feedback for us. And in any case, the only thing I can think of where we could run that code from on a vanilla device requires a reboot, and the timing risks being iffy (onboard potentially not yet mounted, or already mounted and nickel with its hooks in the DB. I'm not sure writing in a SQLite DB from two concurrent processes is considered safe).

---

To get back to the install:

• Check if bundling the PNG in KoboRoot indeed gets it processed at boot...
• Barring that, we've had good experience on KIndles to ship a zip file with the full directory structure prepared, with a simple instruction of: unpack to the USB root. That would have the added benefit of unanonimyzing the KoboRoot tarball since it'd be inside that zip.

[NiLuJe]

Deleting from the Library:

/mnt/onboard/koreader.png ATTRIB 
/mnt/onboard/koreader.png DELETE_SELF

So I'm not quite sure how one could get "trapped", @KenMaltby, do you have real-world examples?

---

Given how fast it boots, I'm assuming Nickel doesn't spend its time trying to stat() every file in onboard at boot, so, that rules that out that too as a potential source of stray OPEN events.

EDIT: And stat doesn't trigger anything anyway :).

[pazos]

Whoooaa! @NiLuJe , when do you sleep?

The loop that @KenMaltby refers is found if fmon is active before koreader.png is indexed. This happens because fmon install triggers a reboot, which comes first than nickel db. When nickel tries to open koreader.png it runs koreader, and this happens again and again after exit koreader

This is why I think about checking nickel db in first place.

@houqp : we don't need to reinitialize the watch. If file isn't found there isn't a way to open the file. The program will run even if /mnt/onboard/koreader.png doesn't exist

[NiLuJe]

@pazos: Haven't slept much tonight :D.

As for the issue with reinitializing the watch, the watch automatically gets destroyed when the underlying FS is unmounted (which is what should happen when it gets exported over USBMS).

cf. man 7 inotify, section BUGS (among others):

When a watch descriptor is removed by calling inotify_rm_watch(2) (or because a watch file is deleted or the filesystem that contains it is unmounted)
[...]

fmon doesn't have any code to deal with this, so I'm wondering if this ever was an issue, or if Kobo somehow handles the USBMS thing differently than what I expected?

[houqp]

In this case, sounds like it should be a trivia fix for the infinite loop? i.e. just wait until it's indexed in the db before performing a restart.

sqlite uses flock, so it's safe for multiple processes to write to it at the same time: https://www.sqlite.org/faq.html#q5. That said, I would still avoid doing it because I don't really trust sqlite ;P

[NiLuJe]

@houqp: I'm more concerned about further USBMS events down the line...

I'll check if fmon implemented a workaround it the easy way (i.e., hook into something else relating to mounts and simply restart the fmon process when onboard is ready). (Which, incidentally, is how I went about it on legacy Kindles when using inotify-tools ;p).

Barring that, there are ways to wait for FS events, so, look into that as well. (Which is what i'd prefer anyway, centralizing everything in a single C daemon, without any wrapper/monitoring shell scripts involved).

[NiLuJe]

Okay, I have all the info I need... I'm going to take a stab at this.

Prepare to be amazed by how terrible I am at C :D.

[NiLuJe]

Okay, WIP in NiLuJe/kfmon ...

Should currently handle unmounts/remounts properly!
Going to tackle SQLite next :).

[NiLuJe]

And now with added SQL! :)

[NiLuJe]

And it's now a true little daemon :).

[NiLuJe]

Okay, did a testrun on a Kobo.

It basically works as intended and as well as it did on my local box when playing with a bunch of tmpfs ;p.

The only major issue left is what happens when opening koreader.png from inside KOReader... It's... not good.
Since we block on system(), nothing happens while KOReader is up... but as soon as you exit it, nickel restarts... but so does KOReader. And then you're basically screwed (I had to dig out a toothpick for the reset button).

So far my less sucky idea is twofold:

• Background the call to koreader.sh in the system() call (that way we should not be blocking anymore, which helps us avoid the catastrophic timing of KOReader exit -> Nickel STartup + KOReader startup...
• Update the koreader script to prevent lauching concurrent instances of itself.

[NiLuJe]

But first, it's sammich time!

[KenMaltby]

Wouldn't the "return 0" get you back to the launching program? The "reboot" seems unneeded, if things were handled properly when reader.lua was called. As it is now when I use KOReader's Home Icon I return to KSM without rebooting. It is a sensible measure returning to something that requires so much background functioning/setup, like Nickel. It now looks like you have handled enough of Nickel's environment to not need it there now as well. I could be missing something, just an observation.

[NiLuJe]

@KenMaltby: Indeed, except in this instance we have nothing to return to, since we basically kill everything at the beginning of the KOReader startup script ;).

Hence we either need to restart Nickel ourselves, or reboot.

[KenMaltby]

I see, but is a reboot needed to return to KSM or Advboot ?

[NiLuJe]

Not with KSM, since it's still running in the BG. Which is why we don't kill it and simply return to it.

[NiLuJe]

As for AdvBoot, it requires a reboot also.

[NiLuJe]

Okay, got a Kobo test build running, appears to behave properly...

Going to write some doc, tag a release, and push this to the world...

[NiLuJe]

Okay, done!

Updated the install package.
Updated the docs.
Committed my KOReader script tweaks.

I'm waiting for those to hit a nightly before pushing this to MR :). [And even launching the PR is pending feedback from another issue, because I mixed different stuff in my tree without working in branches... :o)].
In the meantime, feel free to test & comment :).

@houqp: I'm a jerk. I opted to vendor SQLite in the end, because I realized I needed a minimal build and managed to find usable git mirrors to submodule, and with the switch to a user-configurable model, it could be uncoupled from KOReader... Sorry about the hassle :/.

EDIT: Updated the binaries w/ the latest fixes. Current build should be v0.9-13.

[KenMaltby]

Not to put a damper on the Great work @NiLuJe has accomplished in creating this "Kute File Monitor", improved fmon, But is there a one step installer for KOReader? The new launcher looks neat, for those who use Nickel, but I was hoping that the automated installation of KOReader (mostly for newcomers) was being included in this issue.

[NiLuJe]

@KenMaltby: If you basically merge a koreader zip's content in that install package's zip, you've got it down to one step ;).

[houqp]

Awesome work on kfmon! Looking forward to the MR post. No worry about the sqlite dep, i think it makes more sense to vendor it inside kfmon :)

I assume you are going to send out a PR for the tweaked start script?

[houqp]

NOTE to myself or anyone who is interested to help maintaining the doc:

Update the installation guide in wiki after kfmon is officially launched.

[NiLuJe]

@houqp: Yep, I'm waiting on feedback from #2005 for the PR, since I have a mix of stuff relevant to that in my tree :p.

[KenMaltby]

The MR post will be a "Sticky", when I see it. The "One Step" thread for the KOReader Forum, I mean. I have suggested that the Kobo Developer's Corner thread be made "Sticky", also.

[NiLuJe]

MR Thread posted ;).

As for this issue, I'm thinking maybe do a true 'one-step' package for the next KOReader stable release?
(Provided nothing drastically broken emerges on the KFMon side of things ;)).

[KenMaltby]

It looks like they have a preferred way to handle this sort of thing:

"Hi Ken,

Thank you for your suggestion of a post to be made sticky. We would recommend that posts of this nature - instructions for doing something - be put in the MR Wiki rather than in the forums. The Wiki is a far better place for factual information of a fairly static nature such as this. It doesn't really belong in the forums, which are more suited for discussions.

Kind regards,

HarryT (for the MobileRead moderating team)"

Luck;
Ken

[NiLuJe]

Fair enough ;). I'm not overly fond of stickying everything either (in fact, it's sometimes proved to be counter-intuitive in the Kindle Dev subforum, where some stickied stuff sometimes seems to go unnoticed).

Granted, the fact that a bunch of the active devs got mod powers in the forum help for this kind of boring quality of life stuff ;).

[pazos]

@NiLuJe @houqp : Sooo, enhancement done!. Should I close the issue or wait for documentation updates on https://github.com/koreader/koreader/blob/master/platform/kobo/fmon/README.txt ?

I think installation based on fmon should be deprecated and documentation moved elsewhere. We can make a lean zip without Koboroot.tgz, koreader.png...

[KenMaltby]

@pazos The first ten lines should be retained, (with the references to the old method removed). They could be moved to follow the new KFMon method, though.

[houqp]

yeah, we should update that README to replace fmon with kfmon. BTW, why is that README in fmon directory? Shouldn't it be moved to its parent dir?

[Frenzie]

Also note that the text has acquired a few discrepancies with the wiki: https://github.com/koreader/koreader/wiki/Installation-on-Kobo-devices

[KenMaltby]

I think I may be falling behind, So will KOReader, for Kobo, now come with its own launcher? Will it be included in the download install package? Then the download package would include all that is needed to install and use KOReader, no need to download and install a launcher program? I take it then that the install process would install KFMon and copy over a "koreader folder" ?

For those of us who would prefer to use KSM, will KFM coexist with KSM? fmon does, (at least if you have an almost empty DB) and KSM makes use of a fmon process for some of its features. If not; would just disabling or removing the KoboRoot.tgz, from the .zip, still allow the install?

[NiLuJe]

KFMon is completely orthogonal to KSM, one doesn't care about the other, and vice-versa. As long as you don't somehow open any of the trigger files in KSM, nothing happens. If you somehow have to run both KFMon and KSM, just kill KFMon on KSM's startup to be super-double sure, and that's it.
If KSM uses fmon internally, that doesn't have to change, KFMon is there to deal with nickel, not anything else.

Nothing is likely to change on KOReader's side. I'll probably just make a one-step zip available somewhere of the next stable release, because I agree that the situation is nothing short of terrible for new users, but that's it.

I'm not terribly fond of "super-" packages, and of taking choice away from users. I'd much rather let them decide how and when they want to launch stuff without sneakily sideloading stuff under their nose.

[pazos]

@NiLuJe : bug found!

Changing from rebooting to restarting does not work. Nickel won't launch (stuck in animation)

log said:

[KFMon] [2016-05-22 @ 00:22:53] Trying to load config file '/mnt/onboard/.adds/kfmon/config/kfmon.ini' . . .
[KFMon] [2016-05-22 @ 00:22:53] Daemon config loaded from 'kfmon.ini': db_timeout=450, use_syslog=0
[KFMon] [2016-05-22 @ 00:22:53] Beginning the main loop.
[KFMon] [2016-05-22 @ 00:22:53] Initializing inotify.
[KFMon] [2016-05-22 @ 00:22:53] Setup an inotify watch for '/mnt/onboard/koreader.png' @ index 0.
[KFMon] [2016-05-22 @ 00:22:53] Listening for events.
[KFMon] [2016-05-22 @ 00:24:00] Initializing KFMon v0.9-16-g9b914a8 | Built on May 16 2016 @ 23:24:16 | Using SQLite 3.12.2 (built against version 3.12.2)
[KFMon] [2016-05-22 @ 00:24:00] Trying to load config file '/mnt/onboard/.adds/kfmon/config/koreader.ini' . . .
[KFMon] [2016-05-22 @ 00:24:00] Watch config @ index 0 loaded from 'koreader.ini': filename=/mnt/onboard/koreader.png, action=/mnt/onboard/.adds/kfmon/my_koreader.sh, do_db_update=0, db_title=KOReader, db_author=KOReader Devs, db_comment=An eBook reader application
[KFMon] [2016-05-22 @ 00:24:00] Trying to load config file '/mnt/onboard/.adds/kfmon/config/kfmon.ini' . . .
[KFMon] [2016-05-22 @ 00:24:00] Daemon config loaded from 'kfmon.ini': db_timeout=450, use_syslog=0
[KFMon] [2016-05-22 @ 00:24:00] Beginning the main loop.
[KFMon] [2016-05-22 @ 00:24:00] Initializing inotify.
[KFMon] [2016-05-22 @ 00:24:00] Setup an inotify watch for '/mnt/onboard/koreader.png' @ index 0.
[KFMon] [2016-05-22 @ 00:24:00] Listening for events.
[KFMon] [2016-05-22 @ 00:24:14] Tripped IN_OPEN for /mnt/onboard/koreader.png
[KFMon] [2016-05-22 @ 00:24:14] Tripped IN_CLOSE for /mnt/onboard/koreader.png
[KFMon] [2016-05-22 @ 00:24:14] Spawning /mnt/onboard/.adds/kfmon/my_koreader.sh . . .
[KFMon] [2016-05-22 @ 00:24:14] . . . with pid: 895

[NiLuJe]

@pazos: I assume you're of course running an up to date KOReader snapshot? ;).

I vaguely remember that happening once during testing, but I can't remember if it's because I had botched the KOReader startup script at the time or not...

According to the logs, KOReader's startup script never returned, which is... odd, since the nickel startup script is immediately backgrounded, and is the last thing KOReader's startup script does...
And if you see the animation, then the nickel startup script was launched...

And even if the sigchld handler somehow deadlocked, that shouldn't prevent nickel from starting up properly. At worse KFMon deadlocks and you get a zombie process, but that's it.

The only other time I distinctly remember hitting something weird like that was when I updated the KOReader startup script while KOReader was running. It promptly crapped out with a bogus syntax error on exit. But that doesn't sound exactly like what's happening here, since in your case it never returned.

I'm afraid I can't tell much more without a more detailed look at the state of the system (i.e., over usbnet).

How long was KOReader running? Did the device suspend at any point?

(And, just to confirm: did switching from a reboot to restarting nickel ever work properly for you/that specific device?)

[pazos]

@NiLuJe:

Yes, lastest nightly

With old kfmon (the one without .ini support) and old koreader.sh (without .ini logic to decide between launch nickel and reboot) changing reboot to ./nickel.sh & in koreader.sh did the work.

With stock device + your tools + lastest kfmon + lastest koreader restarting don't work. Reboot is fine.

I have no time right now to look at it so I changed back to reboot. Everything worked perfect here.

[NiLuJe]

Mmh, thanks for the details, I'll look into it... I probably did something
stupid somewhere along the way ;).
On May 29, 2016 7:52 PM, "Martín Fernández" notifications@github.com
wrote:

@NiLuJe https://github.com/NiLuJe:

Yes, lastest nightly

With old kfmon (the one without .ini support) and old koreader.sh (without
.ini logic to decide between launch nickel and reboot) changing reboot to ./nickel.sh
& in koreader.sh did the work.

With stock device + your tools + lastest kfmon + lastest koreader
restarting don't work. Reboot is fine.

I have no time right now to look at it so I changed back to reboot.
Everything worked perfect here.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#1996 (comment),
or mute the thread
https://github.com/notifications/unsubscribe/AAG1Zn0Z6r77z_dbZi0GymOVuHEKmxImks5qGdJFgaJpZM4IRik6
.

[NiLuJe]

Tiny bit of necromancy to mention that I fixed KFMon, and it now deals with nickel restarts properly (along with a host of other bugfixes/improvements) ;p.

(Better late than never!).

[NiLuJe]

And two years later, one-step packages for real! :D.