Joyful Web
The Joyful Web package generates documents from source files written in Org markup and rendered with Jinja2 templates.
Joyful Web is generic enough to generate arbitrary documents but as the name implies it is mainly intended for generating “web sites” of some form.
One example usage: user writes a topic wiki and a dated blog in Org files. Joyful Web converts these files to HTML organized in a structure of users choosing. Joyful Web accesses metadata in and about the Org files and generates RSS feeds for recently created and recently updated topics and blog entries. The Org document keyword #+TAGS: is used to generate a “tag cloud”.
In the end, the resulting functionality and layout are fully under the user’s control. Joyful Web provides tools for parsing Org files and generating common derived content types. If some battery is not included Joyful Web provides ways to extend functionality by writing your own Python modules.
Joyful Web processing follows a series of stages. At the highest level it is a mapping of N Org documents to M output documents. In some more detail, the processing flow is diagrammed below:
The user provides:
- Org source files.
- Jinja2 templates written with understanding of Joyful Web data structures.
- Configuration parameters.
Joyful Web provides:
- A compile mechanism that runs Emacs (and other external programs) on a single Org file in order to produce a more general JSON file.
- A set of processing modules which produced derived content from the loaded JSON files.
- A render mechanism which applies the resulting content to the user’s Jinja2 template to produce the final output.
Joyful Web’s compiling stage runs Emacs and other processes on the Org file in-place in order to produce derived content that can not conveniently be derived directly in Python code or aggregated through Jinja2 templates. This stage is slow relative to rendering and its results are needed by multiple different renderings. Thus, the compiled result is written to a file. The format chosen is JSON as this is easily produced and consumed.
The JSON file contains a single dictionary which has the following keys:
text- full and original content in Org markup.
plain- document exported to plain text.
body- the
<body>of the HTML export. tree- JSON representation of the file’s
org-elementtree (with circular dependencies broken). name- name of file sans directory path and
.orgextension. root- directory root containing the original Org file.
path- subdirectory path fragment between
orgrootand the file. revs- file revisions in the form of a list of pairs of Git commit hash and timestamp (if the file is not in Git then file state information is used).
This data structure is made available as-is to the template under the org dictionary item. See Template data structure for more.
At the highest level, rendering is a two process:
- The compiled data structure from one or more JSON files are loaded.
- Processing modules are applied sequentially on this structure to augment it.
- A template is applied to the result and a final output file is produced.
Templates must be written with an understanding of the data structure provided to them. This structure depends on the number of JSON input files given and the processing modules applied. Thus it is up to the user to coordinate a consistent application of any given template with the proper JSON input files and processing modules.
At top level, the data structure is a dictionary. The element with the key org contains an ordered array of data structures loaded from the given JSON files. Each application of a processing module adds its results to a new top-level dictionary item with the key that of the named processor from the configuration. Finally, all parameters in the associated render configuration section are placed in a (sub)dictionary held by the key cfg.
A Joyful Web processor is simply a Python function with the following signature:
def process(dat, **kwds):
"Document string explaining what this function does"
newdat = dict(...)
# fill newdat
return newdatThe dat object is the current template data structure. It should be treated as read-only by the processor. The kwds are any configuration items the user has bound to the processor in the configuration file. The returned newdat holds the result of the processing and will be attached to the template data structure with a key as determined by user configuration. More on configuration below.
Joyful Web relies a simple configuration file for the following:
- specify any defaults for
joycommand line arguments. - bind configuration parameters and a template to form a rendering.
The configuration file syntax is simple. It consists of named sections which contain key/value pairs. The values in the configuration file are expanded for both environment variable and user directory tilde (~) references.
Section names are interpreted as follows:
[joy COMMAND]- interpreted by
joy COMMANDto provide defaults for command options as described above. Seejoy COMMAND --helpand discussion above. [render NAME]- name a rendering by binding a template to a number of named processors and any extra parameters to be passed to the template.
[global render]- if given, provides default parameters to all renderings.
[processor NAME]- name a processor by binding a Python module+function and its arguments.
An illustrative example configuration:
While the compilation stage breaks down the Org files into easily used chunks and the processors derive additional chunks it is the templates that build up these parts into a new document of whole cloth. The cloth is woven with Jinja2 templates. Users are completely free to author templates as they wish but with understanding of the template data structure described above.
As described above, the dat dictionary is filled with Org-level content as the org item, processor results in keys named after each processor and render configuration parameters in the cfg item. These keys are made available to the template.
An illustrative example template file:
Modules can be set on the command line. For example:
joy render -m joy.modules.rss rss.html myblogentry.jsonMultiple modules can be specified by repeating -m flags. Modules can also be given in the template-specific section the configuration file:
[template rss.html] module = joy.modules.rss
Joyful Web processing tries to avoid policy lock-in. This means it is up to the user to determine how they want to organize their input Org files, the rendered output files and the N-to-M mapping between them. This user policy is expressed through a Joyful Web configuration file, calls to the command line program joy (see below) and of course in the user-provided Jinja2 templates.
Joyful Web provides a command line interface program called joy. Its basic usage:
joy --help
joy [command] [options] [arguments]The two main commands correspond to the two processing stages:
joy compile -o myblogpost.json myblogpost.org
joy render -o myblogpost.html blogtemplate.html myblogpost.jsonJoyful Web itself and specifically the joy command are configured with a set of named parameters. These parameters may come from three sources. In order of increasing precedence and with some example spelling:
- environment variables. These are named like
JOY_<PARAMETER_NAME>. - configuration file. A
parameter_namevariable in the[joy COMMAND]sections (see more below). - command line options. A
--parameter-namecommand line option. Seejoy COMMAND --help.
The joy command parameters are:
org_path- A list of directories in which Org files may be found, potentially in sub-directories. This is used in
joy compile. The path where the input file was found is stored in the resulting JSON file. template_path- A list of directories in which Jinja2 template files may be found. This is used by
joy renderto locate the named template.
Beside providing defaults for the joy commands, the configuration file may be used to make parameters available to a rendering template.
The configuration file syntax is simple. It consists of a number of named sections which contain a number of key/value pairs. The values in the configuration file are expanded for both environment variable and user directory tilde (~) references.
Section names are interpreted as follows:
[joy COMMAND]- interpreted by
joy COMMANDto provide defaults for command options as described above. Seejoy COMMAND --helpand discussion above. [global]- passed directly to all template files as
cfg.<parameter>. [template XXX]- passed directly to template file
XXX(if used) ascfg.<parameter>. These merge with and override any parameters that may be provided by[global].
Then one might find in the blog.html template file something like:
<title> {{ cfg.title }} </title>Joyful Web provides some other functionality.
Joyful Web implicitly converts Org files to other formats as described above. These and other converters may be called with the joy export command:
joy export [-f FORMAT] -o myfile.FORMAT /path/to/myfile.orgThe FORMAT in the above example can be guessed by the output extension or set explicitly. To see supported formats run:
joy list formatsjson- Dump the
org-elementtree as a JSON file. Note, this is only a fragment of the JSON file produced by thejoy compilecommand. html- The full Org file exported as HTML
body- Just the
<bod>part of the above pdf- The full Org file exported as LaTeX/PDF.
Adding support for a new format is just a matter of adding a joy/org2*.el file.
Note, as with joy compile, the exporter runs in the Org source directory which may cause other files to be deposited.
As the website gets larger it is a good idea to automate the running
of joy with a build tool such as Waf. On example is used in the
joyful authors web site. See the wscript file. The basic idea is to
break out the individual export, revisions and rendering in a
fine-grained way to let Waf work its magic.