Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Help message: Offline Wiki bundled with koreader-updates #1371

Open
Markismus opened this issue Dec 21, 2014 · 40 comments
Open

Help message: Offline Wiki bundled with koreader-updates #1371

Markismus opened this issue Dec 21, 2014 · 40 comments

Comments

@Markismus
Copy link
Member

After reading @chrox last wiki updates --much needed because I had forgotten what Zsync was about-- I realize that the help function still point to the website for bug reports.

I would love it if it also referred to the to-be-constructed offline wiki of koreader. Every time a new update for koreader is created, the current wiki is again bundled. It could be read in crengine without the need to wait for Wifi.

Thus the work in the wiki becomes more available and an integral part of koreader.

Of course other ways of opening up the wiki content inside koreader could be nice, too: a menu item for help with a topic list and a scrollable popup much like the dictionary windows.

@Frenzie
Copy link
Member

Frenzie commented Dec 21, 2014

Okay, I just did a quick test:

git clone git@github.com:koreader/koreader.wiki.git
cd koreader.wiki
pandoc *.md --toc --from markdown_github -o help.epub

The resulting TOC looks slightly confusing, but overall it's not bad at all.

@Markismus
Copy link
Member Author

Got it, thanks!

I think that the *.md part might be a great part in the confusion. As a test I tried to find the pages for gestures and chrox's new entries for calibre wireless connection. I found the gestures.

I would suggest more than one epub. Separating the Chinese from the English might be a good start. Separating user from developer seems sensible. Separating all install instructions from the usage instructions seems wise as most will already be reading on their device.

@Markismus
Copy link
Member Author

Unfortunately pandoc *.md --toc -o *.epub doesn't generate a thing.

@Frenzie
Copy link
Member

Frenzie commented Dec 21, 2014

I don't know if Pandoc is the best tool for this particular job, although I do consider it the best for actually writing stuff in general. (In the rare case that it falls short you can just include straight LaTeX, while maintaining Pandoc Markdown's legibility for all your other stuff.)

Something involving Gollum might make more sense in this particular case.

@houqp
Copy link
Member

houqp commented Mar 30, 2016

Feature wise, pandoc will no doubt be the best tooling out there. It might take awhile to configure pandoc to do what we want though. Someone is already trying to do similar stuff: jgm/pandoc#1857.

Another alternative is to write a tiny script in lua, leveraging the markdown parser used by ldoc.

Either way, I think this automated wiki to book tooling can be turned into a decent open-source project that we can share with the community. Given the fact that a lot of projects out there are using github wiki as documentation, it will help many people ;)

@houqp
Copy link
Member

houqp commented Mar 30, 2016

BTW, If anyone volunteers to do this, we can add this ticket to the next stable release milestone too.

@houqp
Copy link
Member

houqp commented Mar 30, 2016

Ideally if the notes feature is developed, we could edit the wiki in koreader and send the changes directly from koreader.

In that sense I do wonder whether conversion of the wiki to epub, notes in epub synced to what-was-the-name-of-that-server, downloaded and converted to a wiki edit proposition is a good workflow.

Haha, this is getting fancy ;) What concern do you have with this workflow?

@Markismus
Copy link
Member Author

Too many intermediaries: Seems like begging for high maintenance.

@houqp
Copy link
Member

houqp commented Mar 30, 2016

It will be simpler if you think of it as two parallel workflows:

  1. convert wiki markdown to epub
  2. convert note to wiki markdown

And we bridge them with git commits. Then again, this is a nice to have feature that I don't expect it to happen in the near future ;)

@Frenzie
Copy link
Member

Frenzie commented Feb 23, 2017

I noticed an interesting blog post: http://blog.terryburton.co.uk/2015/02/20/Creating-PDF-Documentation-From-a-GitHub-Wiki-Using-Pandoc.html (see https://gist.github.com/rogersachan/40e4f69439b26e1f096f8c91e98b1dee for a potentially updated version)

Filter: rewritewikilinks.hs

#!/usr/bin/env runhaskell

import Text.Pandoc.JSON
import Data.Char

main = toJSONFilter rewritewikilinks

rewritewikilinks :: Inline -> [Inline]
rewritewikilinks ( Link txt ('#':url,title) ) = [ ( Link txt ("#"++url,title) ) ]
rewritewikilinks ( Link txt ('h':'t':'t':'p':':':url,title) ) = [ ( Link txt ("http:"++url,title) ) ]
rewritewikilinks ( Link txt ('h':'t':'t':'p':'s':':':url,title) ) = [ ( Link txt ("https:"++url,title) ) ]
rewritewikilinks ( Link txt (url,title) ) = [ ( Link txt ("#" ++ map toLower url,title) ) ]
rewritewikilinks x = [x]

Adding the whole haskell-platform stuff would be pretty heavy and when I just tried it I got an error I don't quite understand:

$ pandoc *.md --toc --from markdown_github --filter rewritewikilinks.hs -o help.epub
pandoc: Error running filter rewritewikilinks.hs
fd:4: hPutBuf: resource vanished (Broken pipe)

Still, it's the main missing link.

Edit: I'm not entirely sure what the problem is, but the quite elaborate GitHub wiki to PDF system can be admired by checking out the postscriptbarcode wiki: git clone https://github.com/bwipp/postscriptbarcode.wiki.git

@pazos
Copy link
Member

pazos commented Feb 23, 2017

@Frenzie I've tried pandoc *.md --toc --from markdown_github -o help.epub and the result is, IMO, pretty bad. It mixes english and chinese and the wiki articles are in mixed order. Also wiki articles have a lot of hyperlinks and cross references.

Why not gitbook?

@Frenzie
Copy link
Member

Frenzie commented Feb 23, 2017

@pazos That was just a quick proof of concept illustration of the principle that probably took me about half a minute to come up with back when I wrote it. Obviously it would need the kind of work that can be seen in the __pandoc directory of the postscriptbarcode.wiki repository to be usable.

There are two points I'd like to keep in mind:

  1. This is what we've got and I think it's pretty decent. Improving and reusing it would be best. Whatever issues there are with the actual content won't go away by using a different platform.
  2. Any potential replacement should be easier, not harder to use than the GitHub wiki.

A quick look at Gitbook suggests it's harder to use than the GitHub wiki. Documentation is the perfect area to contribute without needing to be able to read code or work with git. On GitHub, all you need is a GitHub account.

@pazos
Copy link
Member

pazos commented Feb 23, 2017

There are two points I'd like to keep in mind:

  1. This is what we've got and I think it's pretty decent. Improving and reusing it would be best. Whatever issues there are with the actual content won't go away by using a different platform.
  2. Any potential replacement should be easier, not harder to use than the GitHub wiki.

true

Also, not all the pages of the wiki are relevant for an "getting started" epub. Just something like:

  1. intro
    1.1 product
    1.2 feature list
  2. getting started
    2.1 filemanager
    2.2 reader app
    2.3 taking notes
    2.n ...
  3. plugins
    3.1 evernote
    3.n ...

@houqp
Copy link
Member

houqp commented Feb 24, 2017

I feel like we might need to maintain two separate documentations, one for developers and one for users. The current wiki is a mix of both :(

@Frenzie
Copy link
Member

Frenzie commented Feb 24, 2017 via email

@Frenzie Frenzie mentioned this issue Mar 27, 2017
Frenzie added a commit to Frenzie/koreader that referenced this issue Apr 4, 2017
This commit seeks to improve the paltry help menu item. It breaks the "try to have at least three items in a submenu" rule in anticipation of an upcoming quickstart guide and show quickstart guide setting. This guide will explain that (soon) you will have to swipe south at the top of the screen to activate the menu.

Also see the somewhat related koreader#1371, which would also go in the submenu.
@Frenzie Frenzie mentioned this issue Apr 4, 2017
houqp pushed a commit that referenced this issue Apr 4, 2017
This commit seeks to improve the paltry help menu item. It breaks the "try to have at least three items in a submenu" rule in anticipation of an upcoming quickstart guide and show quickstart guide setting. This guide will explain that (soon) you will have to swipe south at the top of the screen to activate the menu.

Also see the somewhat related #1371, which would also go in the submenu.
@Frenzie
Copy link
Member

Frenzie commented Apr 6, 2017

To make life easier I'm cleaning up the wiki a little. Here's a list of dev pages. Checkmarks indicate they should be safe to be deleted (soon).

The base wiki should also be migrated but that's of less immediate concern.

@Frenzie Frenzie mentioned this issue Apr 6, 2017
@Frenzie
Copy link
Member

Frenzie commented Apr 7, 2017

Hm, working with the wiki I'm inclined to agree with @pazos that the GitHub wiki is kind of terrible. The main problem is no folder structure in the online interface, although you can do it offline. I brought some order into the whole shebang offline but if you use folders with individual _Sidebar.md documents you can't edit those in the web GUI, for example.

As I pointed out before, the main problem with many alternatives is that they're too developer-oriented. That's fine for the developer docs, but the user docs should be easier to edit than that. Other than a GitHub account (preferably not a separate account) the entry barrier should be as low as possible.

It occurred to me that the likes of http://daux.io (and maybe https://www.gitbook.com/ too?) are potentially significantly less bad than I alluded above because GitHub includes an online editor interface. It's potentially a little less obvious than the wiki, but you shouldn't actually have to figure out Git. Some more possible inspiration here: https://github.com/PharkMillups/beautiful-docs

screenshot_2017-04-07_17-37-07

Actually I'm not sure why that links to https://github.com/justinwalsh/daux.io/blob/master/docs/00_Getting_Started.md and not directly to https://github.com/justinwalsh/daux.io/edit/master/docs/00_Getting_Started.md

The probably more ubiquitous RTD has the same feature combined with what I consider the same oddity: http://docs.sylius.org/en/latest/

RTD looks very promising because it comes with automated EPUBs and localization. The less work for me/us the better. ;-) NB They advise reStructuredText (Sphinx) although they also support Markdown. Anyway, Markdown to rST is a piece of cake with Pandoc, so that's certainly no problem.

What are your thoughts on or experiences with these systems?

@Frenzie
Copy link
Member

Frenzie commented Apr 8, 2017

http://www.mkdocs.org/ also looks interesting, demo at http://doc.freshrss.org/en/ (which I mistakenly thought was RTD because it uses the same theme)
FreshRSS must be using a slightly older version because it does support direct edit links as well (mkdocs/mkdocs#269)

It would require more effort:
https://github.com/jgrassler/mkdocs-pandoc
https://github.com/gisce/markdown-i18n (via mkdocs/mkdocs#211)

@houqp
Copy link
Member

houqp commented Apr 14, 2017

mkdocs and daux.io seem to be focused more on developers.

gitbook looks promising although I am not sure how the collaborator restriction is going to affect us.

I would be looking for the following features for the solution:

  • version control
  • easy for users to edit
  • can be rendered into epub, pdf and a beautiful website
  • focused on writing book like documentation, not programmer manual

@Frenzie
Copy link
Member

Frenzie commented Apr 14, 2017

5 collaborators (Gitbook) is pretty minuscule. So far I think RTD is the closest that I've come across, although it's not quite ideal.

@houqp
Copy link
Member

houqp commented Apr 16, 2017

RTD requires the doc to be written in python tooling and managed by git. In that case, it would be more flexible for us to just setup the automated doc build in gitlab CI.

RTD is really just a doc CI ;P

@pazos
Copy link
Member

pazos commented Aug 25, 2018

awesome work on dev documentation @Frenzie !!!

Any update on user documentation? Can we setup a separate repo for a quickstart guide, keep the documentation on markdown and convert it to html or latex using pandoc/scidown?

@Frenzie
Copy link
Member

Frenzie commented Aug 25, 2018

@pazos Could you please clarify what you mean precisely in contrast to the present situation?

@pazos
Copy link
Member

pazos commented Aug 26, 2018

@Frenzie: I'm not sure if we can handle the github wiki as a normal repo (I mean being able to review PRs, submit issues, have visible discussion, etc) or even submit new code outside the markdown articles.

I mean a full repo with contributing guides, a bit of discussion about guide/book toc, and a Makefile to automate the process of building the output formats (at least html and epub).

I find github's wiki pretty decent as a hub that redirect users to proper documentation. Required wiki contents are installation for different devices, and building koreader from source, the user guide can be stored elsewhere an linked to koreader.rocks and to the wiki, as dev docs are.

@Frenzie
Copy link
Member

Frenzie commented Aug 27, 2018

@pazos

or even submit new code outside the markdown articles

You can pretty much put whatever you want in there, but indeed it's a pain.

It's true that converting the wiki to a regular GH repo shouldn't really be much worse in any case. You'd lose the sidebar, but you'd gain visible folder structure. (I already put the wiki into a folder structure, but you can only see if if you clone the repo.)

We could consider tapping into Jekyll (GitHub Pages)?

@pazos
Copy link
Member

pazos commented Sep 12, 2018

@Frenzie:

We could consider tapping into Jekyll (GitHub Pages)?

Sure.

I cloned the wiki and I'm going to spend some time testing outputs with pandoc. For future sake I would use pandoc's --table-of-contents switch and let it generate the toc from #h1-h3 headers.

@pazos pazos self-assigned this Mar 6, 2019
@Frenzie
Copy link
Member

Frenzie commented Aug 29, 2019

As an aside, the pandoc *.md --toc --from markdown_github -o help.epub line I wrote a few years ago will now come out nicer using the newer pandoc *.md --toc --from gfm -o help.epub.

@Frenzie
Copy link
Member

Frenzie commented Aug 30, 2019

Also of potential interest, Prose.io. See https://bugherd.com/blog/building-a-better-github-wiki/

Gitbook has added something about contacting them for FOSS.

@Frenzie
Copy link
Member

Frenzie commented Dec 19, 2020

This might be of use, but it seems to be slightly inactive recently.
https://github.com/yakivmospan/github-wikito-converter

Also of interest, though I'm not sure I want to pollute the main repo with that:
https://github.com/marketplace/actions/wiki-sync

@Frenzie
Copy link
Member

Frenzie commented May 27, 2021

Cf. #7757 (comment) and following for some related discussion.

@offset-torque

I was specifically talking about the content of manual

So change the content. ;-)

I find it funny that everyone is trying to produce the best solution for exporting wiki where %90 of pages in it are 2-3 years old (24-36 releases ?!). SCREENSHOTS are 4 years old, none of the icons even look like that. BASIC USAGE category is missing and says TODO.

You have a unique talent for writing things that don't land well with me, lol. ;-D I think the vast majority of my discussion was focused on contributing.

The discussion is four years old. Four years ago I put in a fair bit of cleaning effort, albeit not with regard to screenshots (incidentally the icon change was a few months back; other visual changes have been significantly more minor): #1371 (comment). I've half-heartedly attempted to keep the sidebar vaguely organized but I don't have the time to keep a close eye on the code and the (user) documentation. Especially since I do directly manage the localization effort. I depend on @AlanSP1 for example to convert information from the release notes I write into wiki content/updates, and I very much appreciate that.

Maybe github is not a suitable place to prepare this type of document. I can understand the appeal of "everybody can edit easily" but from the update frequency it is obvious that no one is interested in it.

No one's happy with GitHub wiki. [Edit: cf. #7742]

NB Ideally an alternative would also integrate with Weblate.

I have published a few books (on paper), edited some and used digital publishing software like QuarkXpress and Scribus so I can say I know a few things about creating user facing documents.

Me too! ;-) I've worked as an editor. While I haven't used Scribus professionally, I've done so with InDesign and Acrobat.

I personally think that a collaborative word document that is regularly exported to PDF will be much more easier and it would look more professional and helpful. I can volunteer for this. But if YOU (whoever is making these kind of decisions) say "Nah, we are ok like this" I have nothing to do.

Could you please expand on some of the advantages and disadvantages compared to GH wiki? (Pay no mind to the fact that you'll mostly be saying stuff I already know. Others may not have worked much with comments and track changes and whatnot. And even so, I'm still curious about your knowledge of the commit log equivalent since I've only worked with Word/Google Docs in relatively formal step by step or small <5 contributor informal workflows, how to export/integrate these things with other platforms, what kind of automation may be available as an option, etc.)

@offset-torque
Copy link

@Frenzie

You have a unique talent for writing things that don't land well with me, lol. ;-D I think the vast majority of my discussion was focused on contributing.

My writing style is short and effective so it might come as offending to some people. I got this reaction before. Please don't take it personally. I am not attacking anyone. Writing and maintaining code leaves little time for documentation. I am not criticizing developers for not updating wiki. I am pointing out to the lack of contribution by other users/members.

Could you please expand on some of the advantages and disadvantages compared to GH wiki? And even so, I'm still curious about your knowledge of the commit log equivalent since I've only worked with Word/Google Docs in relatively formal step by step or small <5 contributor informal workflows, how to export/integrate these things with other platforms, what kind of automation may be available as an option, etc.

After pazos and NiLuJe's encouragement in the referenced thread I started building a document. After going around wiki I can say this job doesn't need github wiki or any other kind of collaborative tool.

I have the impression that KOReader nearly reached maturity (correct me if I am wrong). Maybe some new device support or color e-ink addition in the future. Maybe a few plugins for extra services. But majority of changes will be fixes for issues and bugs and maybe some optimisations. I can't think of anything that will require heavy contributions of many people. And KOReader is not (and will not be) at the size of DARKTABLE or GIMP or BLENDER. So this is more like a one-man job, not community effort.

My plan:

  • I started building a USER GUIDE that doesn't contain developer documentation or installation instructions. That type of frequently updated info needs github. This document assumes the user already installed and just want to learn how to use KOReader. It will be offline readable. It will be one piece with a TOC.

  • Aim of this document will be providing basic usage information to users that they currently (ineffectively) seek by asking questions in the forums, by posting issues, by asking developers.

  • I am using last version of LibreOffice (7.1.3.2). Document is saved in OpenDocument Text format (.odt). Libreoffice can export PDF perfectly and EPUB pretty ok.

  • After finishing manual, I can continue updating it. Or I can pass it to some other volunteer.

So to answer your question:
Advantages:

  • There will be a manual after 8 years
  • It will be much prettier than markdown. I use markdown for my personal knowledge database but it is not a suitable format for publication. The effort that goes into making it presentable defeats its main advantage of low maintenance. My approach also will be much easier than LaTex. Any office suite user can assume the role of maintainer of this document which lowers the bar for volunteers.
  • Document will be self contained. No repo, no cloning, no image resources folder, no deprecating static site generators, no converters or toolchains. Any office suite (that respects OpenDocument format) can open and edit and export it.

Disadvantages:

  • Everyone will not be able to contribute freely, changes have to go through document maintainer. But considering the current state, "everyone is not contributing" already so this won't change anything actually :)

@Frenzie
Copy link
Member

Frenzie commented May 28, 2021

I have the impression that KOReader nearly reached maturity (correct me if I am wrong).

Feature wise yes, UI wise less so. Or at least that's my opinion, ymmv.

But considering the current state, "everyone is not contributing" already so this won't change anything actually :)

New wiki page from yesterday, btw. :p

https://github.com/koreader/koreader/wiki/Clipboard

Document will be self contained. No repo, no cloning, no image resources folder, no deprecating static site generators, no converters or toolchains. Any office suite (that respects OpenDocument format) can open and edit and export it.

I didn't realize you meant it so traditionally. I figured you meant Google Docs or equivalent, with an editorial team. You're welcome to do it that way of course, but you might run into a lot of merge conflicts. That is, I suspect at the very least @NiLuJe and I will make copious use of track changes and comments. The nature of online editing streamlines that slightly better.

@offset-torque
Copy link

offset-torque commented May 28, 2021

New wiki page from yesterday, btw. :p

There are much better options for extracting text from documents. Evernote export, KOHighlights and Calibre plugin (4 if we count manual extraction from metadata.lua). I would say any user that bypasses these 4 easier options to manually copy paste text into a file on a slow e-reader should be pretty advanced and should not need any instructions for copy/paste.

Also if I would add this to the guide I am preparing, title would be "How can I save parts of my book into a text file on my device ?" which denotes a process, a problem that user needs to solve. How discoverable is the title "CLIPBOARD" for a user with a question like this ?

By the way here is a rough draft of document to show you what I have in my mind.
USER GUIDE.pdf

You're welcome to do it that way of course, but you might run into a lot of merge conflicts. That is, I suspect at the very least NiLuJe and I will make copious use of track changes and comments. The nature of online editing streamlines that slightly better.

And regarding this: You can edit the file directly. I don't have to be the gatekeeper. I am sure you both have office installed on your computers. If you can write in wiki you can surely write it into an office document.

@Frenzie
Copy link
Member

Frenzie commented May 28, 2021

Just, you can't say people don't care. ;-)

By the way here is a rough draft of document to show you what I have in my mind.

Great! 👍

And regarding this: You can edit the file directly. I don't have to be the gatekeeper. I am sure you both have office installed on your computers. If you can write in wiki you can surely write it into an office document.

But you have to take up the coordinator role ("gatekeeper") for a more traditional workflow. At a publisher's you (or at least we) use something like shared Excel sheets or Jira to track who's working on what. A bit primitive perhaps but it works. That's overkill, but the basic issue it solves still exists. The problem is it's actually worse, because the nature of our community is more half a page here and half a page there because, well… it's not work. ^_^

@offset-torque
Copy link

After some thinking I decided to clarify my position regarding documentation. I suspect I might be misunderstood:
I am not suggesting all documentation should switch to a single offline document. And I am not positioning this guide as a replacement or parallel to wiki (or any other collaborative effort).

This guide just aims to help new users to get introduced to KOReader basics.
Specifically to answer these questions:

  • How can I put content into KOReader ?
  • How can I control KOReader ?
  • How can I customize my reading experience to my liking ?
  • Some important tips

These are pretty basic things that users need just after installing. For example "long press to set as default" should be one of the first things a user needs to learn because someone transferring from stock reader apps would like to set the fonts and styling according to their habits so new books will be opened with these defaults.

My guide will be helpful conveying this kind of basic information. For advanced usage cases, guide will direct users to wiki where deeper and fresher information can be found.

For example this info I found in wiki is pretty critical so it should be included in my guide:

Reflowing tweaks
The most important parameter for page reflowing is "Word Gap". Tweaking it "small" or "large" will yield completely different results. It's recommended to set the value to "large" for western languages and "small" for CJK languages.

But ssh access is pretty advanced so I can just say:
"You can transfer files wirelessly via SFTP. Check wiki for more info."
This will help with discoverability. Users will be aware of the capabilities and if they are interested they can go to wiki.

@Frenzie
Copy link
Member

Frenzie commented May 28, 2021

These are pretty basic things that users need just after installing. For example "long press to set as default" should be one of the first things a user needs to learn because someone transferring from stock reader apps would like to set the fonts and styling according to their habits so new books will be opened with these defaults.

As not quite an aside, both long-press for more information and long-press to set default are mentioned in the quickstart guide.

It's recommended to set the value to "large" for western languages and "small" for CJK languages.

Perhaps that's more true for Chinese? I don't feel that the results differ that much between auto and large (as they do with small). In fact at a quick glance I only see a worse result with a quick random test.

And small for good measure:

@offset-torque
Copy link

It's recommended to set the value to "large" for western languages and "small" for CJK languages.

I wonder who wrote this ? Maybe we should contact the author so he can fix his entry :)

Reflowing tweaks
Frans de Jonge edited this page on Aug 30, 2019

@Zeyadas
Copy link

Zeyadas commented May 28, 2021

Sorry, I think I shouldn't be here but months ago I asked exactly for this (user guide) file. #7082 (comment)
I think it should be SO (describing), and maybe later an advanced user guide for non programmers do describe something like this #7244 (comment) and the advanced settings
Screenshot_2021-05-29-01-07-39

@NiLuJe
Copy link
Member

NiLuJe commented May 29, 2021

@Zeyadas: Most, if not all, of the stuff that is in there should never ever be touched by a human being anymore. We took pain over the years to make most of the relevant ones exposed somewhere in the UI.

The glaring exception would be the tap zones (I keep intending to write a real wiki page about it), and what's left in https://github.com/koreader/koreader/wiki/Change-defaults (which I did a pass on a while back, but may still not be entirely up to date).

@Frenzie
Copy link
Member

Frenzie commented May 29, 2021

I wonder who wrote this ? Maybe we should contact the author so he can fix his entry :)

NB That's a GH wiki issue. I merely organized it into a folder in 2017. It was written by chrox in 2014. ;-)

With regard to the content, presumably I reasoned the same way as above. If I felt large were better I'd push to make it the default instead of babbling about it somewhere hidden in the wiki.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

7 participants