wallabako integration #2621
Comments
|
From @anarcat's comment:
CREngine has it's own heuristic on parsing out document hierarchy structures for txt file, which is wrong most of the time :P So yeah, don't use txt, you can just rename the ext to html and it will be rendered literally as what's in file. Let me know if you run into another blockers. I would love to use it on my kobo too :) |
|
the problem is i need to settle on one file extension - i haven't tested the i have seen Koreader completely crash on wallabako log files. it's pretty disturbing, because then it crashes again when you try to restart it ;) the only workaround is to delete the logfile... note that KSM can read those logfiles correctly through its native file browser... my blockers are documented on the gitlab issue here. basically, the biggest issue right now is #2620: because Koreader doesn't read file metadata, the only thing you see is the Wallabag article ID, which says nothing about the content. i understand why the file manager is built that way, but it's a serious user interface regression compared to nickel. i have tried implementing a way to set article titles on the filename, but then you are still missing authorship, a title page and other metadata that could be useful. i am considering dropping this approach completely and just wait for koreader to have a better epub browser. next in line is read status synchronization, but this should be fairly easy to implement... if a bit annoying because of the feature duplication, but there's nothing you can do about this, really. |
|
Things might be a little bit better with #2702. |
|
to clarify, #2702 will help Koreader show the wallabako logfile. the most urgent issue, on top of #2614 (which will take more time) is to have some way of firing wallabako from the Koreader menus somehow. is there an easy way I could hook into koreader that way? right now i'm relying on udev firing the script when the wifi comes up, but since we control the UI now, the best would be to have a "sync" button that would start the script and show the output... ideally, it would also auto-configure wallabako: ie. it would prompt for a username and password if missing and so on. i could do that within wallabako itself, but that would basically mean requesting a username/password on the commandline, and i'm not sure how that would interoperate with koreader. thanks! |
|
GUI-wise there's probably a lot of overlap with https://github.com/koreader/koreader/blob/c73a1c1fcf2c83bf44edfad3d06d48199f61f1ff/plugins/goodreads.koplugin/main.lua There's also a JSON parser. local JSON = require("json")Then I guess there'd be an Showing terminal output… @Hzj-jie is #2646 (re)usable for that? |
|
in other words, what I want to do is write a Lua plugin for koreader. :) as for the terminal output - i could probably just show the output of a file somehow... |
|
Yeah, it should be pretty straight forward to write a plugin that calls a shell command. The timesync plugin can be a good example. We can also provide support for deeper integration like automatically spin up wallabako when network is connected etc. |
|
it would be great if someone made some scaffolding i could just test instead of having to learn all this stuff myself. ;) thanks for the pointers! |
|
I just realized you wrote that series on password managers I read a few weeks ago. That was very interesting. :-) Do you have the emulator set up? That's really the key to making this a simple and not a harrowing experience. For basic scaffolding I think it should be enough to just copy what happens here? One step at a time is how I take it, anyway. ;-) |
|
timesync plugin is a good start if you just want to shell out to spin up wallabako. Quick example: local Device = require("device")
-- only enable in kobo
if not Device:isKobo() then return { disabled = true, } end
local InfoMessage = require("ui/widget/infomessage")
local UIManager = require("ui/uimanager")
local WidgetContainer = require("ui/widget/container/widgetcontainer")
local _ = require("gettext")
local Wallabako = WidgetContainer:new{
name = "wallabako",
}
function Wallabako:init()
self.ui.menu:registerToMainMenu(self)
end
function Wallabako(menu_items)
menu_items.wallabako = {
text = _("Enable wallabako"),
callback = function()
os.execute("./wallabako&")
UIManager:show(InfoMessage:new{
text = _("Wallabako started.")
})
end
}
end
return Wallabako |
|
amazing, thanks! i'll look into this within a week or two.
how about configuration? wallabako supports a configuration file in
standard location - should i make users edit that through Koreader
somehow or prompt or are there config facilities through menus or what?
i need a username and password, at least, and there are other optional
flags...
https://gitlab.com/anarcat/wallabako#configuration-file-details
what should i use for configs?
thanks!
|
|
GitLab is currently erroring out, but that's where the GoodReads plugin I mentioned comes in. It contains a GUI that I assume is quite similar to what you want: https://github.com/koreader/koreader/blob/master/plugins/goodreads.koplugin/main.lua#L74-L144 The config is in JSON iirc? (Like I said, GitLab isn't working for me atm.) There's already a JSON parser which you can call with local JSON = require("json")
local ok, json_data = pcall(JSON.decode, lang_data_file:read("*all"))I assume there's also a JSON.encode and such, but I've never actually used it. |
|
the config is indeed JSON. Gitlab should be back online as well, it was a small blip... thanks for the references! |
|
@anarcat Have you been able to make any progress with the integration? |
|
On 2018-01-09 11:38:28, Simon van der Veldt wrote:
@anarcat Have you been able to make any progress with the integration?
i have not, unfortunately.
|
|
so, for what it's worth, i'm looking into this again, after abysmal failures at getting access to Kobo's firmware internals directly. BTW, if some were worried about using sqlite in koreader, i can tell you there is much, much worse in the depths of Nickel: their use of sqlite made my life just miserable... so anyways... @Frenzie by emulator, i assume you mean https://github.com/koreader/koreader#building-prerequisites and so on? i'm must admit i'm a little lost as to where to go from here... there's been some changes to the file browser if i remember correctly, is there anything i should take into account there? or i just need to get off my ass and write that plugin already? :) i should probably start with the sample plugin above and see if that works. then i can try to hook the output in, and maybe have a configuration dialog and so on. this would be great! |
|
so i've tried to build the "emulator" and boy that thing is huge! it builds its own copy of libpng, openssl, mupdf... all stuff that's already in debian! but anyways... i'm stuck now at the run step: i figured this was a problem with translations, so i went into the clearly, there's something wrong if we can't build koreader without registering on a proprietary service. i understand transifex may be an attractive solution, but if you can't pull translations without begging for mercy, I think there's a problem. there are also free software alternatives to transifex, in particular weblate which directly write po files into git repositories, which I would recommend. anyways, is there something i did wrong? |
While the environment is different regardless, for better or for worse that's part of what makes it an "emulator" likely to exhibit the same behavior as the version on a real device and not just a regular program. libpng probably shouldn't matter, OpenSSL might be okay but could be different in some cases (see here), and MuPDF has some patches for our needs.
Translations are never a problem in that sense. You'll just get English instead. The error worth noting here is
I'd love to, but as an initial reaction I'm a bit frightened of losing the concordance/history and some of the community. The Transifex pulling/pushing thing only works when you have sufficient access rights, annoyingly. (Okay, makes sense for pushing, but pulling seems pretty basic… and that illustrates the general problem, I suppose.) Anyway, you make it sound as if it were a recent decision. ;-) It probably dates back some six, maybe even seven years. Do you have any experience in switching over? |
Well, I guess I don't need the full emulator here: just a similar enough running copy would be enough to develop my plugin. So recompiling the universe seems way overkill... It would be nice if there was a way to link against existing libraries: we already link against stuff like SDL for example... :) I assume the MuPDF patches were sent back upstream?
Gah. Well, I guess that could have been made a little more obvious. ;) To be honest, the README had so many different steps that I got lost and didn't install everything... So I guess I got what I deserve...
makes sense, i guess...
yeah, that's a huge problem... surely there's a way to pull without an account... right??
oh, i didn't mean to imply anything. :)
I do not, but weblate is my goto solution for free software projects, so I never had to switch. :) ElementaryOS switched and it seemed pretty straightforward... Nextcloud stayed on Transifex without much explanation even though Weblate was prefered by translators... For what it's worth, Transifex used to be partially free software, but they closed it all down in 2013... In your place, I would be worried about such a silo and import at least the .po files in the git repo so that the public can operate on those without a transifex account. This kind of workflow is, by the way, perfectly easy (because git) in Weblate: you don't need a weblate account to contribute translations and people can work on their own machines directly without using the web interface... |
You'll have to ask the patch author, mostly @TnS-hun in this case. But note that MuPDF has changed various functions between versions too, so you definitely can't just mix & match versions. Anyway, PRs are of course the best way to get something implemented. ;-) No one's against it (afaik).
Blame LuaJIT I guess? :-P We could consider changing the severity of the language stuff from error to warning. Lines 53 to 56 in bfd55c7 The real error could include some debugging advice about SDL if you're on PC.
I wanted to do some unit tests with the translation last year and there didn't seem to be a way.
Setting that aside, I don't care for it much regardless. To me it's not really the GitHub/GitLab of translation and I prefer Poedit or even just going in with a text editor. Do you have any examples of the Weblate to git workflow in practice? |
This is not fatal and can be mistaken by new users for the cause of all their problems, like I did in koreader#2621.
"hardware abstraction" didn't mean anything for me when I tried the emulator at first on Debian. In koreader#2621, it was proposed to suggest installing SDL: do so here, although we do it for all platforms...
Done in #3752.
Done in #3753.
See, that's a big concern to me.
Should I do this here or in another issue? :) |
This is not fatal and can be mistaken by new users for the cause of all their problems, like I did in #2621.
"hardware abstraction" didn't mean anything for me when I tried the emulator at first on Debian. In #2621, it was proposed to suggest installing SDL.
The encrypted zip and a better version of the external font support patches could be upstreamed, but for example the patch that changes the ICC profile error to warning is KOReader specific, so you won't be able to use MuPDF provided by the system. |
|
sure, but that's all stuff that's unnecessary for the reader... i guess i should just open a different issue for that wishlist, right? :) |
|
i guess that debian support (#3108) implies cleaning up those depends, but i made that explicit there. i wonder if we shouldn't explicitly add a todo for merging the patches upstream though... |
|
Would you mind looking into that, @TnS-hun? The less patches the better. :-) |
|
i filed #3777 to followup on upstream patch merging. |
|
just to go back on track with the original wallabako support here... i've identified the following todo list on my side:
About the latter
Thanks so much for the hand holding. There's progress, and i'm confident we can see this throught! :) |
|
Pinging @poire-z as to what's stored by CoverBrowser.
Incidentally, I notice some hating on Lua there. Quoting from the old (but free) edition of PIL: https://www.lua.org/pil/12.html
Notice in what sense?
I don't see why it wouldn't since that's just a kernel thing? :-) Alternatively you should also be able to use |
Not sure this was Lua "hating" as much as questioning the use of Lua as a data storage format. :p I understand the reasoning, but I disagree: I believe that language-neutral, non-executable data formats are better for storage. But if that's all there is, I'll deal with it the best I can.
To make sure the new books show up in the (new) UI.
Thanks! |
|
I mean it looks all fine and easy in PIL: But actually, it's more like: but really, what's actually going on in that .lua file is: notice how different those syntaxes are? to be safe, i would need to support all of those, which means I basically need to implement a CFG (context-free grammar) in Golang... @houqp was nice enough to actually do that and write a fairly complete one with peg.js, which apparently has go bindings: ... but really, i just want to get this one field, so I'll probably go with a regex and deal when the breakage when/if it happens. :) a more complete discussion of file formats can be seen in @houqp's response here: https://gitlab.com/anarcat/wallabako/issues/15#note_25908362 |
Yes, we might change our serialization any second. Muhahahahahaha! xD
@poire-z and Roberto Ierusalimschy convinced me to disagree with @houqp See discussion in #2789 (comment) and following. |
|
I'm not sure I understand what wallabag/wallabako are :) From what I get, you select some articles elsewhere, and there's wallabako that downloads them automatically and make them appears as epub on your device? The new UI if you're talking about the covers/metadata display is just some hacks to the FileManager, to make it display the files (in a directory) with a new dress. Details on what it does and how in #2940. So, koreader has not evolved much: it's just a file manager. The metadata and cover images are extracted as you browse your directories, and stored in a sqlite database (it's just a cache database, like those Thumbs.db you see on windows when you select Mosaic/Minitatures display), so next time you revisit these directories, you don't need to reparse them. This db is defined in: koreader/plugins/coverbrowser.koplugin/bookinfomanager.lua Lines 19 to 23 in f1aae6a but I don't see how that would help with this topic.
The point is that in koreader, you don't notify when new books are available: the user sees them as he browses his directories. If I got that right (my first sentence above), and if you want it in a plugin, and if you have another process that is doing the syncing, your plugin would only need to:
|
|
On 2018-03-19 15:34:28, poire-z wrote:
I'm not sure I understand what wallabag/wallabako are :) From what I get, you select some articles elsewhere, and there's wallabako that downloads them automatically and make them appears as epub on your device?
Wallabag is a read-it-later app. You bookmark articles (websites) and a
"readable" version is saved for later. It used to be called "in the
poche", a pun on the french translation of the "Pocket" word, "Pocket"
being of course a closed-source read-it-later app. Some other folks
wrote Wallabag, in PHP.
Wallabako is a commandline client for Wallabag. I wrote it so that I
could synchronise my unread Wallabag articles on my Kobo reader. It
reads a JSON config file, logs into the wallabag website, lists the
unread articles, and download the ePUB version. It then does some
horrible black magic to tell the Kobo UI ("Nickel") to refresh its
database to see those new books. It also pushes back the "read" status
from the Nickel database back to Wallabag and removes read articles from
the reader.
The _new UI_ if you're talking about the covers/metadata display is
just some hacks to the FileManager, to make it display the files (in a
directory) with a new dress. Details on what it does and how in #2940.
Right. I remember playing around with this, good work! :)
So, koreader has not evolved much: it's just a file manager. The metadata and cover images are extracted as you browse your directories, and stored in a sqlite database (it's just a cache database, like those Thumbs.db you see on windows when you select Mosaic/Minitatures display), so next time you revisit these directories, you don't need to reparse them. This db is defined in:
https://github.com/koreader/koreader/blob/f1aae6a03cc1167400096d1c92f99eb7b794bc18/plugins/coverbrowser.koplugin/bookinfomanager.lua#L19-L23
but I don't see how that would help with this topic.
Indeed, looking at the db format, I don't think it can help.
The stats I am looking for are basically:
* is the article present
* is the article read or unread
* what percentage of the book is read (not currently supported in
wallabako)
* annotations (not supported either)
> To make sure the new books show up in the (new) UI.
The point is that in koreader, you don't notify when new books are available: the user sees them as he browses his directories.
Anyway, you don't need, as in nickel, to notify anything when you put
books in directories.
Yay. :)
(May be you could hack the history file (lua again :) to put them at top, so they are displayed in History.)
Okay, that's a good trick. It's something like that I'm looking for, and
that I generally found lacking in Koreader... When I add a new book, I
don't immediately notice it in the UI, because everything is organised
by folder. I need to "remember" which book I put in there. When I
manually transfer books, that's okay: I'm supposed to know what I want
to read. But wallabako is different: there could be a bunch of articles
coming in (and even more already there unread), so it's important to
show the latest entries, somehow.
I guess that someone browsing to the "wallabako" folder could sort
entries by modification date?
If I got that right (my first sentence above), and if you want it in a
plugin, and if you have another process that is doing the syncing,
your plugin would only need to:
- notice when we are online, and watch (reschedule a function every 10
seconds, while we are online) some log file about the new books that
are downloaded
- and just pop up an InfoMessage or a TextViewer with what's
available, with may be a button to close the current book and open
filemanager in the target directory where the new files were saved.
That's an interesting idea, I didn't consider it that way. Note that I
have full control over wallabako's implementation: this is what I mean
by "notifying" koreader... Instead of having koreader do a busy loop
when online (which could take precious battery power), couldn't there be
a way to have wallabako notify it when it's done downloading?
I already have ways of firing up wallabako automatically when we go
online. It's not great, because it's also based on a timer that
sometimes fail, but that's kind of orthogonal to communication with
koreader. Besides, I was told there are also hooks for that in
koreader.
You wouldn't need to extract yourself metadata: the coverbrowser
plugin would do that when your user goes to that directory (and with
the way it works, displaying `Filename...` during the few seconds it
need to extract metadata, your user would get some visual feedback
about what's new).
Ah that's a good point. But why I wanted to extract metadata is not to
inform user of progress, it's to report back read status to wallabag, so
that read articles on the reader are marked as read on the wallbag
server as well.
Thanks for all your feedback, it was very useful!
|
The sort option is global. There's been some wish for per-directory sort option (#3428), not implemented, so it went with naming files with date. But may be that could be added (like if there's a
Well, if you're writting a plugin in lua, and all your wallabag articles sit in a single koreader/plugins/coverbrowser.koplugin/listmenu.lua Lines 332 to 356 in f1aae6a
We don't busy loop :) Most of the stuff does as described in #2500 (comment) I don't think we have an easy way to have koreader be notified from the outside (except via /dev/ events, fakevents, which are too device specific).
That shouldn't be too expensive, neither in battery power nor in your lua hacking time :) |
|
On 2018-03-20 05:42:10, poire-z wrote:
> I guess that someone browsing to the "wallabako" folder could sort entries by modification date?
The sort option is global. There's been some wish for per-directory sort option (#3428), not implemented, so it went with naming files with date. But may be that could be added (like if there's a `.sdr/diroptions.lua`, have filemanager read some options from it and use them. You would then just need to create it when creating the folder. Or cleaner: just have plugins notify FileManager that this or that directory has to be sorted in this non-standard way.
Understood. I think i can live with global sort-by-date, it kind of
makes sense anyways... Not that the wallabako plugin would override this
or anything, but it could live with the global settings. Furthermore,
filenames are numeric, based on the server's article ID, which is an
increasing integer. So sorting by *that* is already a good start
anyways...
Well, if you're writting a plugin in lua, and all your wallabag articles sit in a single `Wallabag/` directory, your plugin can just list that directory content and use our existing functions to get these infos, like:
https://github.com/koreader/koreader/blob/f1aae6a03cc1167400096d1c92f99eb7b794bc18/plugins/coverbrowser.koplugin/listmenu.lua#L332-L356
Nice. I didn't consider designing a Wallabako-specific interface like
this, but I guess it would make sense.
The one thing I do not want to do however, is to reimplement all of
wallabako in Lua. So I would rather parse the .lua metadata from
Wallabako than to do this from Lua directly.
And thanks for the updates regarding how to notify koreader... That
would be a useful addition, but I understand why it's not been necessary
yet...
I've already got plenty on my plate as it is with the current todo, so
I'll stick to that for now and look at fancy notification stuff later. ;)
|
|
On a slightly related note, with Mozilla's acquisition of Pocket, there are apparently plans in motion to make it OSS. |
|
On 2018-07-07 16:12:06, NiLuJe wrote:
On a slightly related note, with Mozilla's acquisition of Pocket, there are apparently plans in motion to make it OSS.
That's what they said when they acquired it (now over a year ago,
2017-02-27) but they never stated a timeline and I have not heard of any
progress since then. I am therefore skeptical and would love to see an
explicit commitment from Mozilla to clarify these things.
|
|
Hence my use of "apparently" ^^. Last I checked, I think there's a few things somewhere on GH, and a tracking BZ issue, but that's the extent of the public-facing side of this ;). |
ref: #1580
This is the master ticket wallabako integration discussion.
The text was updated successfully, but these errors were encountered: