v0.15.0

Summary of Improvements

In the works for over a year with over 500 commits and 100 issues closed, please welcome the biggest single release in SILE history. This is the big “Rewrite it in Rust”! Caveat lector. In truth this is a big release but it is not a rewrite. All the typesetting internals of SILE are still written in Lua and 100% user modifiable at runtime. From an end user or 3rd party module developer standpoint little has changed. However SILE itself is now a compiled Rust application that includes its own Lua interpreter. The build process can (optionally) embed all the Lua and other resources files that makeup SILE and its dependencies in a single binary. This opens the door for improvements such as being able to leverage Rust libraries (including exposing their functions to Lua), write some parts of core functions in Rust for performance or preference, write modules in languages other than Lua or C, package SILE for platforms where Lua is not easy to get running, and much more.

The language change mostly affects building and packaging SILE itself. Once running, relatively little has changed with the way SILE interacts with documents. That being said quite a number of default settings have been changed. See the Usage section of these release notes and the retrograde module documentation for tips on how to transition smoothly.

For a transition period, the Lua based CLI is still available as sile-lua. This may be useful for scripting environments that generate inputs and/or parse the output of SILE itself. These should be transitioned to the new Rust CLI sile which has a few minor differences in argument handling and output message formatting. The Lua CLI will only be available for a limited number of future releases. Please do report any issues using the new CLI.

Extra thanks to @Omikhleia for lots contributions and input during the development cycle; and also to @ctrlcctrlv for generous sponsorships that enabled me to commit quite a bit more time to development.

Installation: For Anyone Installing From Packages

If you install SILE from your distro's package manager or other packaging, nothing about your process needs to change. Update via your system tools and enjoy.

Installation: For Distro Packagers and Source Installations

If you install from source or package SILE, the build command sequence is the same but there are new prerequisite dependencies. It now requires Rust tooling (cargo, rustc) as well as some more utilities (jq) to be available at build time. No new dependencies are needed at run time. In fact it is now no longer necessary to use a Lua VM available at runtime for the new CLI. SILE brings its own Lua VM along. (The legacy sile-lua CLI of course still uses Lua as before.) Optionally the build process can also be setup to embed all of its runtime dependencies in a single binary. See ./configure --enable-embedded-resources and ./configure --enable-static if you want to pursue the single binary route. Most users should use the defaults that install the Lua files and other assets separately as an easy reference for tinkering with and overriding them.

Another notable change is that the default Lua VM has been switched from whatever the system supplied to to LuaJIT. This has been an option for a while, but the default has been whatever the newest PUC Lua version was on the host system. LuaJIT is roughly equivalent to Lua 5.1 and doesn't have some small niceties from 5.4, but it is much much faster. Some of the differences are papered over since SILE depends on and provides the compat53 library compatibility layer. Of course build time SILE can still be configured to used any version of Lua of your choice. This can be used to match compatibility with 3rd party modules or other system components.

Usage

A number of command behaviours and default settings have been changed. This will likely cause documents to render with a different flow. It will also break support for some 3rd party modules which will need to be updated to match. Many (but not all) of the changes can be temporarily disabled to cause as few changes when rendering old documents as possible. A new module called retrograde can be loaded at run time that will reset defaults and even revert some commands and functions to their previous behavior. A target argument can be passed for the version of SILE your document was designed for. Any default setting changes and as much other functionality as practical that may have changed since that release will be reverted.

$ sile -u 'packages.retrograde[target=v0.14.17]' <INPUTS>

Since the Lua VM version used by default is different, you may need to reinstall 3rd party modules with a matching Lua version. You can first query SILE to understand what version of Lua it is using. With that information you can specify the Lua version you want when you install modules as LuaRocks to match. As a demonstration we'll install a Markdown input module in a project-local directory where SILE will find it without extra path configuration, then use it to render a PDF file:

$ sile -q -e 'print(SILE.lua_version); os.exit()'
5.1
$ luarocks --lua-version 5.1 --tree lua_modules install markdown.sile
[…]
$ echo 'Test *Markdown* rendering.' > test.md
$ sile -u inputters.markdown test.md
[…]

Without further ado, here is the nitty–gritty.

⚠ BREAKING CHANGES

• packages: Lists now respect the input document spacing and normal settings with regard to paragraphs breaks before, after, and inside lists. This is place of overriding the paragraph skip settings to match the list item spacing setting and always forcing paragraph breaks before and after lists.

• classes: Hitherto SILE has cleared the current.parindent setting as soon as it used it at the beginning of a paragraph. With this release, the setting is not being cleared until a paragraph is explicitly ended. This will not have an affect on many documents, but could completely blow up layout code that implicitly relied on the effect. Normal paragraphs (e.g. separated by a blank line in the input) and any use cases that explicitly called far ending a paragraph (e.g. by calling \par) will be unaffected. But anywhere a paragraph break was simulated by adding vertical space, the indentation will not be applied the next start of a line. This means that calling any variant of \skip inline in a paragraph will result in content beginning on a new line without using the parindent setting.

  Fixing this change in behaviour requires either explicitly resetting the current.parindent setting after it is initially used or explicitly ending a paragraph before or after placing a vertical skip.

• inputters: Input documents using the SIL language will now retain whitespace more consistently. Whitespace following environment blocks is no longer swallowed in differently than space following command syntax. Consecutive line breaks in the input will consistently trigger new paragraphs no matter what they follow.

  Note that this change cannot be patched over via the retrograde package settings because by the time your document could specify what packages to load or settings to set, the input document has already been parsed. To achieve the same rendering results where environments could be ended leaving any amount of blank lines and still joined to the following content as part of the same paragraphs, you will need to remove the extraneous whitespace.

• core: Several top level instance creators of various names have been re-organized under SILE.types.

  Specifically SILE.color, SILE.measurement, and SILE.length have the same names, just under SILE.types.. Additionally SILE.nodefactory is now SILE.types.node and SILE.units is not SILE.types.unit.

  This brings a little bit of sanity to the naming schemes so that you can guess how to use something from the name, but it also makes room for 3rd party add ons to more easily extend or replace these functions. It also makes it easier to start substituting Rust bits where desired.

• core: Use SILE.papersize() instead of SILE.paperSizeParser()

• classes: The "center", "raggedleft" and "raggedright" environments formerly reset the margins (left or right skips), meaning they'd take the full frame width. They all cancelled the paragraph indent. The new behaviour honors the fixed part of the parent context's margins, meaning that if you have an environment playing with margins such as an epigraph or an indented quote, those margins are not lost. The raggedleft and raggedright environment also now no longer cancel the paragraph indent.

• classes: The \script function was heavily overloaded to have many different functions at once and more targeted tools were introduced in SILE v0.14.0 To load 3rd party modules designed for use with SILE, use \use[module=...] instead of \script[src=...]. To run arbitrary Lua code inline use \lua{}; Lua code may be provided inline or externally via either a require= option to load a regular (non-SILE) Lua module using the Lua module path or src= option to load a file by file path.

• core: For ... reasons ... the default width of spaces in SILE has been a highly opinionated and non-standard 1.2 spaces. While it can be argued that this makes some fonts and some documents look better, it is a very strange thing to have as a global default. Unfortunately setting it back to a more conventional 1 space is a major change and will cause many/most documents to re–flow.

  The old default can be recovered either in documents with:

  \set[parameter=shaper.spaceenlargementfactor,value=1.2,makedefault=true]
  

  ...or even from the CLI when rendering a document:

  $ sile -e 'SILE.settings:set("shaper.spaceenlargementfactor", 1.2, true)'

• core: The previous default paragraph indent was hard coded with a point size (20pt) that did not adapt will to different fonts or page layouts. The new default uses a relative unit that will adjust based on the leading (1bs). This will cause most documents to re–flow. To keep them the same the setting may be reset to the old default either in the document:

  \set[parameter=document.parindent,value=20pt,makedefault=true]
  

  ...or even from the CLI when rendering a document:

  $ sile -e 'SILE.settings:set("document.parindent", "20pt", true)'

• utilities: For modules that rely on SILE.utilities (SU), and in particular raw content handling functions subContent(), walkContent(), stripContentPos(), hasContent(), and contentToString(), these and similar functions have been moved into SILE.utilities.ast (SU.ast). The subContent() implementation also no longer adds id="stuff" attributes to everything.

• packages: The default rendering of Ruby readings has changed from just using a bold weight to using the OpenType +ruby feature. Fonts that support this should work with no change, but documents rendered in fonts that do not support it will need to set the ruby.opentype feature to false to get the same rendering method as before.

• classes: The former implementation of the "em" command did not support nesting and was just setting the font style to italic. The command now alternates italic and regular when nested.

• packages: The current (pseudo) idempotent behaviour when loading a package potentially clobbers anything that has been modified since the last load. Loading a package, then modifying a function it provides, then loading the same package again will clobber the modification. This is good for idempotency but not very good for user experience when you may not be modifying all aspects of a document render pipeline at once, as in when using templates.

  This change makes the default behaviour to run setting, raw handler, and command registrations only once. An alternative to :loadpackage() called :reloadpackage() can be used to force all these registrations to be rerun when the goal is to make sure of a specific state.

• classes: Remove obsolete/broken native markdown class

• core: The internal package manager that installed stuff to the system from inside SILE was deprecated back in v0.13.2. It is now completely removed. External 3rd party packages are fully supported using LuaRocks and are much more robust.

Features

• build: Make vendoring Lua sources controllable at build time (773245b)
• build: Set default Lua version to prefer LuaJIT (#1873) (8a8b506)
• classes,outputters,packages: New approach to full bleed printing, cropmarks, background (aa538e2)
• classes: Make text alignment commands honor nesting and margins (7cde8a3)
• classes: Provide a "blockquote" environment in the plain class (75a8bdf)
• classes: Support emphasis nesting (0c1c017)
• cli: Add configure time option to embed Lua resources in binary (9f11100)
• cli: Allow more than one input document (Rust edition) (bdfeecd)
• core: Allow multiple runtime SILE_PATH segments (b329923)
• core: Allow multiple runtime SILE_PATH segments in Lua binary as well as Rust (5e05fa9)
• core: Automatically include project-local lua_modules in module loader path (fd940a9)
• core: Make it easier to add project-specific LuaRocks trees as dependencies (796b344)
• docker: Add all default fonts and package management tooling to container (6bd5dc0)
• docker: Automatically adjust Docker user to owner of mounted data (4ee535f)
• docker: Deploy upcoming major version branches to a develop tag on GHCR (bd2c6f3)
• docker: Enable data for all provided tools that support system locales (c01203f)
• docker: Make it easier for users to install stuff into the images (71aa6f2)
• inputters: Add formal grammar specification for SIL files (5f78af3)
• inputters: Output source comments in SIL debug info (672226f)
• math: Support numbering display math equations (d2e348f)
• outputters: Add hook system to outputters (b5422a8)
• outputters: Add support for pdf elements to debug outputter (a33e5c7)
• packages: Add package to reset some defaults similar to previous releases (fa8d401)
• packages: Backport default font change from v0.9.5 to retrograde package (6b5f52d)
• packages: Expand retrograde package to handle behavior shims (a549b44)
• packages: Expand retrograde package to shim paragraph indentation changes (bead177)
• packages: Make most package _init() actions run only once (ed3331a)
• packages: PDF links can now span multiple lines (842cb56)
• packages: Remove arbitrary six-level maximum to list nesting (ee76eab)
• packages: Strikethrough and underline can span multiple lines (3eb1a1f)
• packages: Use OpenType +ruby feature for ruby, toggle with setting (624e3a8)
• settings: Add hook callback system for settings (efade76)
• settings: Allow \set with no parameter as a temporary wrapper (59755c9)
• tooling: Add direnv support (a52e6e5)
• tooling: Add DRAFT flag to allow rendering docs without regenerating TOC (f581b54)
• tooling: Add nix builds with various lua versions (c28cd94)
• tooling: Add nix packages building sile with clang (5b30695)
• tooling: Add tooling to generate Lua API documentation from sources (3995e14)
• types: Add __tostring to colors for easier debugging (e1106a7)
• typesetter,shaper: Add automated italic correction heuristics (540b0cf)
• typesetters: Add base typesetter (multi-)liner support (e645dbd)
• utilities: Add Greek alphabetical (non-arithmetic) numbering (34e2e53)
• utilities: Automatically cast anything passed to SU.debug to a string (da80c56)
• utilities: Make it easier to search breadcrumbs excluding the current command (64e8d3b)
• utilities: Refactor and extend AST-handling utilities (fc7e7fb)

Bug Fixes

• backends: Fix plain text backend to work in LuaJIT/Lua 5.1 (d0a2a1e)
• build: Adjust macfonts module to more robustly import AppKit (c0571be)
• build: Avoid race condition by making sure binary exists before use (d37da35)
• build: Correct configure flag so debug builds are not release mode (d931849)
• build: Fix build configuration for --with-system-libtexpdf (d22ea34)
• build: Move build-time dependency checks out of runtime dep check configure flag (5e587bd)
• build: Stop copying libraries places autotools doesn't normally handle them (68d9008)
• build: Switch from xz to zst compression for official artifacts (685780d)
• classes: Reset current parindent at end of paragraphs, not beginning (84c70fd)
• core: Output makedepends file after class finish and snippets (f59670c)
• core: Use unique function name so static linking doesn't clobber Lua-5.1 (ccdb8a2)
• docker: Use LuaJIT not Lua5.1 for docker builds (7687bf6)
• inputters: Allow empty string as option value in SIL parser (160501c)
• inputters: Correct typo in SIL parser, disallow conflicting commands (923b11e)
• inputters: Make whitespace after environments behave the same as commands (6d51ee3)
• languages: Handle UTF8 properly in Liang hyphenation (b33c11c)
• languages: Load default en language support if none explicitly requested (#2036) (d02b063)
• packages: Color changes shall not affect other content at page breaks (9d6d409)
• packages: Correct bogus mixup of variable names in package inits (8dfe3c3)
• packages: Fix broken font fallback clear function (2b15de5)
• packages: Use current pagebuilder API not deprecatod one internaly (730150f)
• tooling: Add npx to nix' devShell (624137f)
• tooling: Fix Nix build for rustier sile (f53904c)
• tooling: Really ignore all nix symlinks (34b0493)
• typesetters: Debug hbox could show incorrectly offset boxes (c5113b7)
• utilities: Cast empty to default and only ever return a bool from SU.boolean() (ba2e37a)
• utilities: Fix broken breadcrumbs:contains() (f1972bd)

Performance Improvements

• utilities: Leave tostring() casting to after debug invocations, not before (72965ad)

Miscellaneous Chores

• classes: Deprecate \script in favor of more explicit options (b738507)
• classes: Remove obsolete/broken native markdown class (133567e)
• core: Change default paragraph indent to 1 baseline skip (542c869)
• core: Change default space width to 1 space (66cdbad)
• core: Deprecate SILE.paperSizeParser() (4486a19)
• core: Remove deprecated package manager (4836a93)

Code Refactoring

• core: Move assorted classes from oddball names in core to types (8287a0f)
• packages: Stop overriding paragraph handling in and around lists (8b2117c)