Scraper Plugins

Scraper Plugins

Some sites are better served by a downloader that already exists than by mirror_base.py. A plugin runs that other program for a comic, then takes in what it fetched, so the comic is kept like any other: loose pages in Uncompressed/<Comic>, archives on the CBZs shelf, chapters, ComicInfo, the Reader, the schedule, and The Web Page.

Where plugins live

Each plugin is a folder in the config folder's plugins/ (config/plugins/<id>/), and usually a git repository of its own. The folder is named by the id in its plugin.json, and that name is how a comic names its plugin.

config/plugins/<name>/
  plugin.json     required: how to run the program and what it leaves behind
  plugin.py       optional: functions that replace the defaults
  dialog.js       optional: its own form on the web page (else one is built from its fields)
  dialog.css      optional: the look of that form
  tool/           optional: the program itself, usually a git submodule of its own repository

Installing one

On The Web Page, Plugins (beside Settings, on Add comics) lists the plugins, with Update and Remove for each. Install from takes a plugin repository's git address, and optionally a branch or tag. Each action is a job in the queue, and git's and pip's output goes to the log.

• Install clones the repository with its submodules, checks its plugin.json, names its folder by its id, and installs the Python packages it asks for. A repository that is not a usable plugin leaves nothing behind.
• Update pulls the branch it follows and its submodules, then installs its packages again if what it asks for has changed. A plugin installed at a tag stays at that tag; install it again to move it.
• Remove is refused while any comic still uses the plugin.

Installing runs whatever the repository holds, so the page offers it only when MIRROR_WEB_PASSWORD is set. To allow it on a page with no password, set MIRROR_ALLOW_PLUGIN_INSTALLS=yes in the container's environment. The page itself cannot switch this on.

From a shell: run_plugin.py --install <git address> [--ref <branch or tag>] and run_plugin.py --update <id>.

A plugin can also be put in by hand: clone it (git clone --recurse-submodules) or copy its folder into config/plugins/. A plugin put in by hand shows up without a restart, but it cannot be updated from the page unless it is a git clone.

examples/plugins/example/ in this repository is a complete plugin to copy from. Its program draws a made-up series rather than fetching one, so it can be tried with nothing downloaded: copy the folder into config/plugins/.

plugin.json

{
  "id": "example",
  "name": "Example",
  "version": 1,
  "about": "One line shown beside the Scraper picker.",
  "command": ["{python}", "{tool}/scraper.py", "{url}", ["--start", "{last_chapter}"], "{reverse}"],
  "output": "chapter-cbz",
  "fields": [
    {"key": "url", "type": "url", "label": "Series address"},
    {"key": "reverse", "type": "flag", "label": "Oldest first", "option": "--reverse"}
  ]
}

• id names the plugin's folder, and so the plugin every comic that uses it names. It is letters, digits, ., _ and -, and it never changes once comics use it. Installing from a repository needs one. A plugin whose folder has another name is refused, with a message saying to rename the folder.

• install is the Python packages its program needs beyond what the image has: "pip" lists them and "requirements" names a requirements file inside the plugin, either or both (see below).

• needs lists the other programs it runs, such as ["ffmpeg"]. These are only checked: a plugin needing one the image lacks has its runs fail with a message naming it.

• output says what the program leaves in the folder it is run in:

  • pages: pictures, possibly in subfolders. Videos (.mp4, .webm, .m4v, .mov, .mkv) and Flash (.swf) are pages too: the folder keeps them as they are, and archives hold a stand-in for each. The same goes for the pages of chapter-folders and chapter-cbz.
  • chapter-folders: a folder of pictures per chapter, named for the chapter.
  • chapter-cbz: an archive (.cbz or .zip) per chapter, named for the chapter.

• fields are the comic's settings for this plugin. Each has a key, a type (url, text, count or flag), and optionally a label, a default, a placeholder, a help and required. A url field is required unless it says otherwise. Its value is the comic's address (settings.url), the one the library shows.

• command is the program and its arguments, run in the comic's staging folder. Placeholders:

  • {python} is the Python to run it with: the plugin's own if it has an install, otherwise the updater's.
  • {plugin} is the plugin's folder, and {tool} is its tool/ folder.
  • {staging} is where the program is run.
  • Any field's {key}.
  • {last_chapter} and {last_page}: where the last run got to, as the default remember keeps it.

  A word that is only a placeholder is left out when that placeholder is empty. So is a group of words given as a list, ["--start", "{last_chapter}"], which leaves nothing on a comic's first run. A flag field puts in its option when ticked and nothing otherwise. A placeholder that names nothing is an error, so a typo is caught.

• remembers: names that a plugin.py remember keeps, so the command may use them.

• setup: the plugin's own settings, the same for every comic of it, such as an account on the site and its key. They are listed like fields, with one more type, secret, and an optional env (see below).

The plugin's own settings

fields are a comic's settings. setup is for what belongs to the plugin as a whole, set once and used by every comic of it:

"setup": [
  {"key": "username", "type": "text", "label": "Account", "env": "SITE_USERNAME"},
  {"key": "api_key", "type": "secret", "label": "API key", "env": "SITE_API_KEY"},
  {"key": "agent", "type": "text", "label": "User-Agent", "default": "MyPlugin/1.0"}
]

• Where they are set: the Settings button beside the plugin in the Plugins dialog. A secret is shown masked until Show is pressed; whoever can reach the page can see and change it, as with everything else there. Saving takes effect from the plugin's next run.
• Where they are kept: config/plugin-setup/<id>.json, readable only by the user the updater runs as. Not in the plugin's own folder, so updating or reinstalling the plugin keeps them.
• How the program gets them: in its environment, each as its env name, or ZCM_ and its key in capitals (ZCM_AGENT). A flag is 1 when ticked; a setting with no value is not set at all. A setting nothing was saved for is handed over as its default.
• How plugin.py gets them: as settings["setup"], a dict by key, in every function it defines.
• In a command: a non-secret setting can be a placeholder, {agent}, as a field can. A secret cannot: the command is written into the log and the comic's run record, so a plugin.json that names one is refused with a message saying which environment variable to read instead.

plugin.py

Each of these is optional and replaces the default of the same name:

Function	Default
build_command(settings, state, staging)	Fill in plugin.json's command.
chapter_order(labels)	Chapter names sorted with numbers as numbers, so Chapter 9 comes before Chapter 10.
remember(state, taken)	Keep the last chapter and the last page taken in, as last_chapter and last_page. taken is {"pages", "chapters", "skipped"}.
ended(settings, state, taken)	None: a comic is only ever marked ended by hand.

ended is for a plugin whose site says when a series is finished. It is asked after every run whose program finished (never one that failed or ran out of time) of a comic not already ended, with the comic's settings, the state remember just kept, and what the run took in. It answers True (complete), False (not complete) or anything else (no answer):

• True ticks the comic's ended box, so later updates leave it alone, and the metadata notes it in history.plugin_ended.
• Unticking it by hand while the plugin still says True sticks: the comic is checked every run, and not marked again.
• False, a series that came back, clears that note, so the next True marks the comic ended again. A series can die, come back and die again as often as it likes.
• No answer changes nothing. An ended that raises is reported in the run's log and is no answer too, so a check that fails once neither ends a comic nor makes it look as if the series came back.

plugin.py is run only by the process that runs one of the plugin's comics. The web page reads only plugin.json.

dialog.js

The web page builds a form from fields. A plugin that wants its own puts itself in pluginForms under its folder's name:

pluginForms["example"] = {
  render(container, values, fields) { /* fill container with inputs showing values */ },
  read(container) { return { chapters: "3" }; },  // the settings as typed, by key; the updater checks them
};

The page loads it the first time the plugin is chosen. If it fails to load, the page falls back to the form built from fields. Only .js and .css files at the top of the plugin's folder are served, at /plugins/<name>/, and nothing in tool/.

Adding a comic

On The Web Page, Add comics shows a Scraper picker once a plugin is installed. Choosing a plugin puts away the built-in scraper's options and shows the plugin's settings, which are shared by every row. Each row still has its own folder, archive and address. From a shell:

python run_plugin.py --list
python run_plugin.py --add example Uncompressed/MyComic url=https://example.com/comics/MyComic chapters=5

Run --add from the library's root, as update_comics.py runs comics. --no-cbz keeps no archives.

The comic's metadata holds settings.scraper (the plugin's name) and settings.plugin (its settings). Its Edit dialog shows those settings, and hides what only the built-in scraper has: the start page, numbering, wait time, the chapter list, and working out chapters.

What a run does

1. The plugin's program runs in <comic>/.plugin-staging/, and its output appears under Running now.
2. What it wrote is taken in:
   • Pages are numbered on from the comic's last.
   • A chapter's name goes into each of its pages' names, since every chapter has its own 001.jpg.
   • A chapter the comic already has is not taken again, so a program that resumes by fetching its last chapter again is fine.
   • The very file the comic already has is not taken twice.
   • A different picture under a name the comic already uses is taken in as (2); nothing is ever written over.
3. New chapters are added to the comic's chapter list, and each gets one archive the same way any chaptered comic does. A comic with no chapters keeps a single archive.
4. A ComicInfo.xml the program wrote is kept with its chapter (or with the comic, for pages) and written into the archives (details).
5. Where it got to is remembered for the next command.

A scheduled run stops the program a little before the comic's timeout, so whatever it fetched is still taken in and the next run carries on (exit 13). A run that was killed outright leaves its staging folder behind. The next run takes in whatever of it is whole, and drops an archive that will not open.

Plugin comics are never walked or aligned: their page numbers say which file is which. The safety checks mirror_base makes (exits 8 to 11) do not apply, because the plugin's program decides the order.

What a plugin's program has to do

A plugin works best with a program that does three things:

• It writes where it is run, or where it is told. The program is run in the comic's staging folder, and only what lands there is taken in. A program that writes beside itself needs an option naming the folder, passed as {staging}.
• It uses the container's browser if it drives one. It should take the browser and driver from MIRROR_BROWSER_BINARY and MIRROR_DRIVER_BINARY when those are set. Otherwise Selenium goes looking for a Chrome of its own.
• It exits non-zero when it fails. A program that always exits 0 makes a broken run look like one with nothing new.

If you wrote the program, change it. If you can't, the plugin's plugin.py can make up the difference:

• Its build_command can run a small script of the plugin's own in place of the program.
• That script imports the program, points its output folder at the working folder, and swaps in a browser setup that uses the variables above.
• It then calls the program's main() and exits with a code that reflects what happened.

This depends on the program's internals, so recheck it whenever the program's submodule is updated.

What a plugin's program runs with

The image has Python 3.12 with selenium and requests, plus Chromium and its chromedriver (named by MIRROR_BROWSER_BINARY and MIRROR_DRIVER_BINARY). A program that needs nothing more runs with that, and its plugin needs no install.

A program that needs more Python packages names them in install:

"install": {"requirements": "tool/requirements.txt", "pip": ["somepackage>=1.2"]}

They are installed into a virtual environment in the plugin's own folder, .venv/. It is made with --system-site-packages, so the image's selenium and requests are still seen. The command is run with that environment's Python: {python}, or a plugin.py command that starts with sys.executable, which is swapped for it.

• When: on install and update, and before any run that finds it missing or out of date. Pip's output goes to that run's log.
• A new image: the environment is in the config folder, so it is kept. If the new image brings a newer Python, which a virtual environment cannot outlive, the next run installs the packages again and says why.
• Failure: if they cannot be installed, the run fails (exit 12) with pip's output in its log, and the next run tries again.

Programs other than Python packages, such as ffmpeg, cannot be installed this way: the container does not run as root, and anything installed with apt would be lost with the image. Name them in needs so a missing one is reported clearly.