Skip to content
Search locale wikipedia with kiwix HTTP server.
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
.gitignore
LICENSE
README.org
kiwix.el

README.org

Intro

Searching offline Wikipedia through Kiwix.

This kiwix.el supports query kiwix-tools’s kiwix-serve server through URL API.

The kiwix-serve server can be started from command-line if you have kiwix-tools installed, or from Docker container [fn:1].

How does this extension work?

integrate with Emacs

core

I found Kiwix will return a URL like this:

http://127.0.0.1:8000/wikinews_en_all_2015-11/A/Big_Linux_Beta_3_released.html
____________________  _____________________  __  _____________________________

< server address >    < library >                <one of the returned results>

steps

  1. auto start kiwix-serve HTTP server.
  2. query/search on kiwix server.
    1. open kiwix server index page to input to search. (But this is slow, waste time)
    2. use http language binding library to query on kiwix HTTP server.
      1. select library in library list page.
      2. after load a library, simulate type query string in the search input box, the submit to search.
      3. return the result page HTML or page URL.
      4. view the result with page URL or page HTML with Emacs browser.

auto start kiwix HTTP server

Here is a simple script, you can put it in Linux ”auto-start”.

#+BEGIN_SRC sh :tangle "~/scripts/kiwix-server.sh"
#!/usr/bin/env sh

/usr/lib/kiwix/bin/kiwix-serve --library --port=8000 --daemon ~/.www.kiwix.org/kiwix/8ip89lik.default/data/library/library.xml
#+END_SRC

search

  1. kiwix-search command -> return a list of results.
    #+BEGIN_SRC sh
    /usr/lib/kiwix/bin/kiwix-search ~/.www.kiwix.org/kiwix/8ip89lik.default/data/index/wikinews_en_all_2015-11.zim.idx linux
    #+END_SRC
        
  2. use one element of list as part of the URL.

    http://127.0.0.1:8000/wikinews_en_all_2015-11/A/Big_Linux_Beta_3_released.html

    #+BEGIN_SRC emacs-lisp
    (browse-url (concat "http://127.0.0.1:8000/" "LIBRARY" "/A/" "RESULT"))
    #+END_SRC
        

more advanced?

If you want more advanced functions, you can use communicate kiwix HTTP server with RESTful API.

  • I don’t know what Emacs library to use.
  • Or you can use other language to do this, like Ruby or Python etc.

Install

Install Kiwix

Flatpak

#+begin_src sh :dir /sudo::/tmp
# Install Flatpak (on Debian/Ubuntu)
sudo pacman -S flatpak

# Install Flathub (for the dependencies)
flatpak remote-add --if-not-exists flathub https://flathub.org/repo/flathub.flatpakrepo

# Download the Kiwix Desktop Flatpak
wget https://download.kiwix.org/release/kiwix-desktop/org.kiwix.desktop.2.0-beta2.flatpak
#+end_src

#+begin_src sh :dir /tmp :eval no
# Install Kiwix Desktop
flatpak install org.kiwix.desktop.2.0-beta2.flatpak
#+end_src

#+begin_src sh :eval no
# Run Kiwix Desktop (but Kiwix should be available through your app launcher anyway)
flatpak run org.kiwix.desktop
#+end_src

Download

Linux

Arch

#+begin_src sh :dir /sudo:: :results none
aurman -S --noconfirm kiwix-bin
#+end_src

Web Browser

Firefox

https://addons.mozilla.org/en-US/firefox/addon/kiwix-offline/

Chrome

https://chrome.google.com/webstore/detail/kiwix/donaljnlmapmngakoipdmehbfcioahhk

MELPA

#+begin_src emacs-lisp :eval no
(use-package kiwix
  ;; :ensure t
  :load-path "~/Code/Emacs/kiwix.el/"
  :after org
  :commands (kiwix-launch-server kiwix-at-point-interactive)
  :bind (:map document-prefix ("w" . kiwix-at-point-interactive))
  :init (setq kiwix-server-use-docker t
              kiwix-server-port 8080
              kiwix-default-library "wikipedia_zh_all_2015-11.zim"))
#+end_src

Setup

If you use kiwix-serve Docker container, you can create an Systemd unit service to auto start Docker container. Here is the systemd unit config file:

Dockerize kiwix-tools (kiwix-serve, etc)

docker pull kiwix/kiwix-serve

#+RESULTS[<2019-03-24 08:33:29> ace542940af6e465f90f0a3a8515e876fd267ad5]:

Using default tag: latest
latest: Pulling from kiwix/kiwix-serve
8e402f1a9c57: Pulling fs layer
7024865ce0e2: Pulling fs layer
ad4c9cfc45dc: Pulling fs layer
c4d62acdb073: Pulling fs layer
c4d62acdb073: Waiting
8e402f1a9c57: Verifying Checksum
8e402f1a9c57: Download complete
7024865ce0e2: Verifying Checksum
7024865ce0e2: Download complete
8e402f1a9c57: Pull complete
7024865ce0e2: Pull complete
c4d62acdb073: Verifying Checksum
c4d62acdb073: Download complete
ad4c9cfc45dc: Verifying Checksum
ad4c9cfc45dc: Download complete
ad4c9cfc45dc: Pull complete
c4d62acdb073: Pull complete
Digest: sha256:8837effa1a4fce750dc373d58b47063b368228331ccacb267c6ae7d3e311e66c
Status: Downloaded newer image for kiwix/kiwix-serve:latest

https://github.com/kiwix/kiwix-tools/blob/master/docker/server/Dockerfile

#+begin_src dockerfile
FROM alpine:latest
LABEL maintainer Emmanuel Engelhart <kelson@kiwix.org>

# Install kiwix-serve
WORKDIR /
RUN apk add --no-cache curl bzip2
RUN curl -kL https://download.kiwix.org/release/kiwix-tools/kiwix-tools_linux-x86_64-1.1.0.tar.gz | tar -xz && \
    mv kiwix-tools*/kiwix-serve /usr/local/bin && \
    rm -r kiwix-tools*

# Configure kiwix-serve
VOLUME /data
ENV PORT 80
EXPOSE $PORT

# Run kiwix-serve
WORKDIR /data
ENTRYPOINT ["/usr/local/bin/kiwix-serve", "--port", "$PORT"]
#+end_src

How to run?

Given wikipedia.zim (Zim database files) resides in /tmp/zim/, execute the following command:

docker container run -v /tmp/zim:/data -p 8080:80 kiwix/kiwix-serve wikipedia.zim

If you put ZIM files in other places not /tmp/zim/, you can use follow my command:

docker container run -d \
       --name kiwix-serve \
       -v ~/.www.kiwix.org/kiwix/nsz6b6tr.default/data/library:/data \
       -p 8089:80 \
       kiwix/kiwix-serve wikipedia_zh_all_2015-11.zim

Visit http://localhost:8080 or http://localhost:8089 (if you exposed different port), you will see:

data/e8/2e194f-2cc8-45eb-a378-f8bd6d7c6b1a/screenshot_1.png

Load the wikipedia.zim library, you will see:

data/e8/2e194f-2cc8-45eb-a378-f8bd6d7c6b1a/screenshot_2.png

For easy launch the docker run command, you can add command alias in shell profile:

alias kiwix-docker-wikipedia_zh_all="docker container run --name kiwix-serve -d -v ~/.www.kiwix.org/kiwix/nsz6b6tr.default/data/library:/data -p 8089:80 kiwix/kiwix-serve wikipedia_zh_all_2015-11.zim"
alias kiwix-docker-wikipedia="docker container run --name kiwix-serve -d -v ~/.www.kiwix.org/kiwix/nsz6b6tr.default/data/library:/data -p 8089:80 kiwix/kiwix-serve wikipedia.zim"

create a systemd unit for kiwix-serve Docker service

#+begin_src systemd :tangle "~/.config/systemd/user/kiwix-serve.timer"
[Unit]
Description=Start kiwx-serve Docker container server at system startup after 5 minutes

[Timer]
OnBootSec=2min
Unit=kiwix-serve.service

[Install]
WantedBy=default.target
#+end_src

#+begin_src systemd :tangle "~/.config/systemd/user/kiwix-serve.service"
[Unit]
Description=kiwix-serve Docker server
After=docker.service

[Service]
Type=simple
ExecStart=/usr/bin/docker container start -i kiwix-serve
ExecStop=/usr/bin/docker container stop kiwix-serve

[Install]
WantedBy=default.target
#+end_src

NOTE: You need to use option -i for docker container start command to avoid systemd auto exit and stop kiwix-serve container.

systemctl --user enable kiwix-serve.timer
systemctl --user status kiwix-serve.timer | cat

#+RESULTS[<2019-03-24 11:45:40> 6470584177f091e79067f9fd96a97c340e00a41f]:

● kiwix-serve.timer - Start kiwx-serve Docker container server at system startup after 5 minutes
   Loaded: loaded (/home/stardiviner/.config/systemd/user/kiwix-serve.timer; enabled; vendor preset: enabled)
   Active: inactive (dead)
  Trigger: n/a
systemctl --user start kiwix-serve.service
systemctl --user status kiwix-serve.service | cat

#+RESULTS[<2019-03-24 12:00:49> 10a33f8521fa2c72e8c1107559e1fb18b58d7da2]:

● kiwix-serve.service - kiwix-serve Docker server
   Loaded: loaded (/home/stardiviner/.config/systemd/user/kiwix-serve.service; disabled; vendor preset: enabled)
   Active: active (running) since Sun 2019-03-24 12:00:14 CST; 34s ago
 Main PID: 2587 (docker)
   CGroup: /user.slice/user-1000.slice/user@1000.service/kiwix-serve.service
           └─2587 /usr/bin/docker container start -i kiwix-serve

Mar 24 12:00:14 dark systemd[694]: Started kiwix-serve Docker server.

NOTE: Because kiwix-serve.service use command docker container start kiwix-serve, so that the container kiwix-serve must already been created by this command, you can check whether the container is created:

docker container ls | head -n 1
docker container ls --all | grep "kiwix-serve" | cat

#+RESULTS[<2019-03-24 11:50:36> e28015e8e78015623bd53ae596015949dc80c549]:

CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
b47533ecd7f6        kiwix/kiwix-serve               "/usr/local/bin/kiwi…"   3 hours ago         Exited (137) 2 minutes ago                                      kiwix-serve
e2f201e655ac        kiwix/kiwix-serve               "/usr/local/bin/kiwi…"   3 hours ago         Created                                                         distracted_hofstadter

Usage

Use in Emacs

[M-x kiwix-at-point-interactive]

Org-mode integration

[C-c C-l] to insert link.

The link format is like this:

[[wikipedia:(library):search][description]]

The (library) can be wikipedia_en, wikipedia_zh, wiktionary_en, or en, zh etc.

Test

Operations Research
query contains space.
Operations research
the second word is not capitalized.
中国
non-english query
meta-circular interpreter
only capitalize the first word.

Footnotes

[fn:1] https://github.com/kiwix/kiwix-tools/issues/257

You can’t perform that action at this time.