publish.org

Overview

This is a self-documenting org-mode publishing script. It is run by executing all the source code blocks herein. The actual publish.el script just loads this file in a buffer and calls org-babel-execute-buffer followed by the function that starts the publishing process.

This is based on David Wilson’s publish.el which itself is based on Pierre Neidhardt’s publish.el. Huge thanks for both.
It requires emacs v27+
I’m using UIkit CSS framework.
Reader comments powered by Utterances.
There’s a little bit of JavaScript, mostly for the tags used to filter and sort on the homepage.
CI is run on GitHub actions where this is built and deployed to Netlify. It is hosted at https://notes.alex-miller.co/. See CI configuration.

Learning resources for org-mode publishing

Usage

emacs -Q --batch -l ./publish.el --funcall my/publish

This creates the HTML files in the public dir.

Development

Assets

Assets like CSS and JS files are stored in a git backed directory site_asstes. The files therein are symlinked in the public directory which is gitignored. ln -s site_assets/js public/js ln -s site_assets/css public/css ln -s site_assets/favicon public/favicon

Serving locally

After building locally to the public directory, I use the emacs-web-server package to serve it locally. Use httpd-serve-directory and point it at the publishing directory. It will serve the site on http://localhost:8080/.

Dependencies

Package repositories

Adds melpa and elpa archives.
Sets the package archives directory so that packages aren’t installed in ~~/.emacs.d/elpa~.

(require 'package)

(setq package-user-dir (expand-file-name "./.packages"))

(setq package-archives '(("melpa" . "https://melpa.org/packages/")
                        ("elpa" . "https://elpa.gnu.org/packages/")))

(package-initialize)
(unless package-archive-contents
  (package-refresh-contents))

use-package

See use-package repo for more on this. It’s a clean way to handle dependencies in emacs.

(unless (package-installed-p 'use-package)
  (package-install 'use-package))
(require 'use-package)

esxml

esxml provides elisp functions to generate HTML markup. Basically this means less reliance on ugly concat and format function calls.

(use-package esxml :ensure t)

htmlize

I don’t really know much about emacs-htmlize and all of its capabilities, but in the context of this script, it provides CSS styling for code syntax highlighting.
I believe the default is to use inline CSS, but it can generate a style sheet based on your emacs theme by calling org-html-htmlize-generate-css. I did that then linked the stylesheet in the HTML document <head>.
Tell it to use a stylesheet over line styles by setting the org-html-htmlize-output-type variable. See below.
Check out Org css for more on this.

(use-package htmlize :ensure t)

ts

ts.el for sanity when formatting and parsing dates.

(use-package ts :ensure t)

s

s.el for sanity when working with strings.

(use-package s :ensure t)

ox-publish

The publishing system for org-mode

(require 'ox-publish)

Variables

Site variables

These get referenced when generating the HTML.

(setq my/site-title   "Alex's Slip-box"
      my/site-tagline "These are my org-mode notes in sort of Zettelkasten style"
      my/sitemap-title "")

Org publish and export variables

I’m not going to bother explaining all these since they’re thoroughly explained with describe-variable

(setq org-publish-use-timestamps-flag t
      org-publish-timestamp-directory "./.org-cache/"
      org-export-with-section-numbers nil
      org-export-use-babel nil
      org-export-with-smart-quotes t
      org-export-with-sub-superscripts nil
      org-export-with-tags 'not-in-toc
      org-export-date-timestamp-format "Y-%m-%d %H:%M %p"
      org-id-locations-file-relative t
      org-id-locations-file "./.org-id-locations"
      org-id-track-globally t)

HTML exporter variables

Tell htmlize to use a CSS stylesheet rather than inline styles.
Use describe-variable to learn about the rest of them.

(setq org-html-metadata-timestamp-format "%Y-%m-%d"
      org-html-checkbox-type 'site-html
      org-html-html5-fancy nil
      org-html-htmlize-output-type 'css
      org-html-self-link-headlines t
      org-html-validation-link nil
      org-html-inline-images t
      org-html-doctype "html5")

Other variables

This is backed by a git repository, so we don’t need backups

(setq make-backup-files nil)

Export document

Site header

This function is called when generating the HTML template below.
- info arg is a plist from which we can get configuration details about the org document. I’m not using it here, but it comes in handy in other functions to get things like the document title, date, etc.
Here I am using esxml to declare the markup in elisp.
- It’s quoted (with `) but we can use ~,~ to selectively evaluate expressions therein. Noice.
  - See Backquote docs for more.
- @ function is for declaring node attributes like class, id or whatever.

(defun my/site-header (info)
  (sxml-to-xml
   `(div (@ (class "header uk-section uk-section-primary"))
         (div (@ (class "heading uk-container"))
              (div (@ (class "site-title-container uk-flex uk-flex-middle"))
                   (h1 (@ (class "site-title uk-h1 uk-heading-medium")) ,my/site-title))
              (div (@ (class "site-tagline uk-text-lead")) ,my/site-tagline))
         (div (@ (class "uk-container"))
              (nav (@ (class "uk-navbar-container uk-navbar-transparent")
                      (uk-navbar))
                   (div (@ (class "uk-navbar-left"))
                        (ul (@ (class "uk-navbar-nav"))
                            (li (a (@ (class "nav-link") (href "/")) "Notes"))
                            (li (a (@ (class "nav-link") (href "https://github.com/apmiller108")) "Github"))
                            (li (a (@ (class "nav-link") (href "https://alex-miller.co")) "alex-miller.co")))))))))

Site footer

This function is called when generating the HTML template below.
creator is Emacs {{version}} (Org mode {{version}})

(defun my/site-footer (info)
  (sxml-to-xml
  `(footer (@ (class "footer uk-section uk-section-secondary"))
            (div (@ (class "uk-container footer-container"))
                 (div (@ (class "footer-links"))
                      (a (@ (href "https://notes.alex-miller.co/")
                            (class "footer-link")
                            (uk-icon "icon: album"))
                            "notes")
                      (a (@ (href "https://github.com/apmiller108")
                            (class "footer-link")
                            (uk-icon "icon: github-alt"))
                            "github")
                      (a (@ (href "https://twitter.com/apmiller108")
                            (class "footer-link")
                            (uk-icon "icon: twitter"))
                         "@apmiller108")
                      (a (@ (href "https://www.reddit.com/user/apmillz")
                            (class "footer-link")
                            (uk-icon "icon: reddit"))
                         "u/apmillz"))
                 (div (@ (class "made-with"))
                      (p "Made with " ,(plist-get info :creator)))))))

The HTML Template

This is the whole page layout. It makes use of the header and footer functions above and injects the org-mode document exported HTML (the contents arg).
I think all of this is pretty self explanatory, but one thing I should call out is the use of :filetags to generate the tag links. I’m not entirely sure I had to do this, but I declared as a custom export option in the derived backend. See below.
Same with the :updated property.
- This is a timestamp this is automatically generated when an org-mode file is saved. See Automatically generate an updated at timestamp when saving an org file for how that works.

(defun my/org-html-template (contents info)
  (concat
  "<!DOCTYPE html>"
  (sxml-to-xml
    `(html (@ (lang "en"))
          (head
            (meta (@ (charset "utf-8")))
            (meta (@ (author "Alex P. Miller")))
            (meta (@ (name "viewport")
                    (content "width=device-width, initial-scale=1, shrink-to-fit=no")))
            (link (@ (rel "apple-touch-icon")
                    (sizes "180x180")
                    (href "/favicon/apple-touch-icon.png?v=1")))
            (link (@ (rel "icon")
                    (type "image/png")
                    (sizes "32x32")
                    (href "/favicon/favicon-32x32.png?v=1")))
            (link (@ (rel "icon")
                    (type "image/png")
                    (sizes "16x16")
                    (href "/favicon/favicon-16x16.png?v=1")))
            (link (@ (rel "manifest")
                    (href "/favicon/manifest.json?v=1")))
            (link (@ (rel "mask-icon")
                    (href "/favicon/safari-pinned-tab.svg?v=1")))
            (link (@ (rel "stylesheet")
                    (href "/css/uikit.min.css")))
            (link (@ (rel "stylesheet")
                    (href "/css/code.css")))
            (link (@ (rel "stylesheet")
                    (href "/css/site.css")))
            (script (@ (src "/js/uikit.min.js")) nil)
            (script (@ (src "/js/uikit-icons.min.js")) nil)
            (script (@ (src "/js/site.js")) nil)
            (script (@ (src "https://www.googletagmanager.com/gtag/js?id=G-YM3EHHB2YQ")) nil)
            (script
            "window.dataLayer = window.dataLayer || [];
              function gtag(){dataLayer.push(arguments);}
              gtag('js', new Date());

              gtag('config', 'G-YM3EHHB2YQ');"
            )
            (title ,(concat (org-export-data (plist-get info :title) info) " - notes.alex-miller.com")))
          (body
            ,(my/site-header info)
            (div (@ (class "main uk-section uk-section-muted"))
                  (div (@ (class "note uk-container"))
                      (div (@ (class "note-content"))
                            (h1 (@ (class "note-title uk-h1"))
                                ,(org-export-data (plist-get info :title) info))
                            (div (@ (class "note-meta"))
                                ,(when (plist-get info :date)
                                    `(p (@ (class "note-created uk-article-meta"))
                                        ,(format "Created on %s" (ts-format "%B %e, %Y" (ts-parse (org-export-data (plist-get info :date) info))))))
                                ,(when (plist-get info :updated)
                                    `(p (@ (class "note-updated uk-article-meta"))
                                        ,(format "Updated on %s" (ts-format "%B %e, %Y" (ts-parse (plist-get info :updated)))))))
                            ,(let ((tags (org-export-data (plist-get info :filetags) info)))
                               (when (and tags (> (length tags) 0))
                                 `(p (@ (class "blog-post-tags"))
                                     "Tags: "
                                     ,(mapconcat (lambda (tag) (format "<a href=\"/?tag=%s\">%s</a>" tag tag))
                                                 (plist-get info :filetags)
                                                 ", "))))
                            ,contents)
                      ,(when (not (string-equal my/sitemap-title (org-export-data (plist-get info :title) info)))
                          '(script (@ (src "https://utteranc.es/client.js")
                                      (repo "apmiller108/slip-box")
                                      (issue-term "title")
                                      (label "comments")
                                      (theme "boxy-light")
                                      (crossorigin "anonymous")
                                      (async))
                                  nil))))
                  ,(my/site-footer info))))))

Element customization

Links and Images

The link paths need to match the actual file paths of the exported files. So for file links, the exported link paths are downcased and without filename extensions. So, this function ensures the link paths match that format. So [[file:my_post.org][My Post]] becomes <a href="my_post">My Post</a> (no .html on the path).
Org-roam uses the ID property for linking notes (ie, no file path). To get around this I do the following:
1. In my my publish.el file, I generate the .org-id-locations file. This file is committed since it is also used on CI where I couldn’t even generate this file as part of the build process.
2. Again in publish.el , set the my/org-id-locations variable to a hashtable generated from the .org-id-locations file.
3. For fuzzy type links, find the path from the hashtable. Oh, somehow the fuzzy type links are the ID links.
4. Seriously, what a pain in the arse.
I have some inline images in my org files. These are file links without a label that point to files with image extensions. Mostly these are plantuml renderings. They get converted to HTML img tags.
For everything else, just render a good old fashion anchor tag.

(defun my/format-path-for-anchor-tag (path)
  (concat "/"
          (downcase
           (file-name-sans-extension
            path))))
(defun my/org-html-link (link contents info)
  "Removes file extension and changes the path into lowercase org file:// links.
  Handles creating inline images with `<img>' tags for png, jpg, and svg files
  when the link doesn't have a label, otherwise just creates a link."
  ;; TODO: refactor this mess
  (if (string= 'fuzzy (org-element-property :type link))
      (let ((path (gethash (s-replace "id:" "" (org-element-property :path link)) my/org-id-locations)))
        (print path)
        (if path
            (org-element-put-property link :path
                                      (my/format-path-for-anchor-tag
                                       (car (last (s-split "/" path))))))))
  (when (and (string= 'file (org-element-property :type link))
            (string= "org" (file-name-extension (org-element-property :path link))))
    (org-element-put-property link :path
                              (my/format-path-for-anchor-tag
                                        (org-element-property :path link))))

  (if (and (string= 'file (org-element-property :type link))
          (file-name-extension (org-element-property :path link))
          (string-match "png\\|jpg\\|svg"
                        (file-name-extension
                          (org-element-property :path link)))
          (equal contents nil))
      (format "<img src=/%s >" (org-element-property :path link))
    (if (and (equal contents nil)
            (or (not (file-name-extension (org-element-property :path link)))
                (and (file-name-extension (org-element-property :path link))
                      (not (string-match "png\\|jpg\\|svg"
                                        (file-name-extension
                                          (org-element-property :path link)))))))
        (format "<a href=\"%s\">%s</a>"
                (org-element-property :raw-link link)
                (org-element-property :raw-link link))
      (format "<a href=\"%s\">%s</a>"
              (org-element-property :path link)
              contents))))

Headings

This part is largely unchanged from David Wilson’s publish.el on which this is based.

Maybe something else already requires subx-r.el, but we make sure we can use thread-last.
This helper function is used when rendering headlines. It kebab cases the cases the headline text for use as the HTML element’s ID.
- Sometimes heading words are fenced with ~~~, so the code tag is removed.

(require 'subr-x)

(defun my/make-heading-anchor-name (headline-text)
  (thread-last headline-text
    (downcase)
    (replace-regexp-in-string " " "-")
    (replace-regexp-in-string "</?code>" "")
    (replace-regexp-in-string "[^[:alnum:]_-]" "")))

Basically, this translates the org-mode headlines to HTML h tags of the corresponding level with anchor tag handles, IDs that can be easily linked to, while respecting export options.

(defun my/org-html-headline (headline contents info)
  (let* ((text (org-export-data (org-element-property :title headline) info))
        (level (org-export-get-relative-level headline info))
        (level (min 7 (when level (1+ level))))
        (anchor-name (my/make-heading-anchor-name text))
        (attributes (org-element-property :ATTR_HTML headline))
        (container (org-element-property :HTML_CONTAINER headline))
        (container-class (and container (org-element-property :HTML_CONTAINER_CLASS headline))))
    (when attributes
      (setq attributes
            (format " %s" (org-html--make-attribute-string
                           (org-export-read-attribute 'attr_html
                                                      `(nil
                                                        (attr_html ,(split-string attributes))))))))
    (concat
     (when (and container (not (string= "" container)))
       (format "<%s%s>" container (if container-class (format " class=\"%s\"" container-class) "")))
     (if (not (org-export-low-level-p headline info))
         (format "<h%d%s><a id=\"%s\" class=\"anchor\" href=\"#%s\"><i># </i></a>%s</h%d>%s"
                level
                (or attributes "")
                anchor-name
                anchor-name
                text
                level
                (or contents ""))
       (concat
        (when (org-export-first-sibling-p headline info) "<ul>")
        (format "<li>%s%s</li>" text (or contents ""))
        (when (org-export-last-sibling-p headline info) "</ul>")))
     (when (and container (not (string= "" container)))
       (format "</%s>" (cl-subseq container 0 (cl-search " " container)))))))

The Sitemap (the home page)

Sitemap Entry

Formats sitemap entry as {date} {title} ({filetags}). Returns a list containing the sitemap entry string and the filetags. A unique list of the filetags is created on the sitemap page from this list, that’s why they’re returned from this function.

(defun my/sitemap-format-entry (entry style project)
  (let* ((filetags (org-publish-find-property entry :filetags project 'site-html))
         (created-at (format-time-string "%Y-%m-%d"
                                         (date-to-time
                                          (format "%s" (car (org-publish-find-property entry :date project))))))
         (entry
          (sxml-to-xml
           `(li (@ (data-date ,created-at)
                   (class ,(mapconcat (lambda (tag) tag) filetags " ")))
                (span (@ (class "sitemap-entry-date")) ,created-at)
                (a (@ (href ,(file-name-sans-extension entry)))
                   ,(org-publish-find-title entry project))

                ,(if filetags
                     `(span (@ (class "sitemap-entry-tags"))
                            ,(concat "("
                                     (mapconcat (lambda (tag) tag) filetags ", ")
                                     ")")))))))
        (list entry filetags)))

Sitemap page

From the function above, the filetags are placed into a flattened list, duplicate values removed and sorted alphabetical ascending. These are turned into tags on the page used for filtering the entries by topic. All of the JS used for filtering is provided by the UIkit CSS framework.

(defun my/sitemap (title list)
  (let* ((unique-tags
          (sort
          (delete-dups
            (flatten-tree
              (mapcar (lambda (item) (cdr (car item)))
                      (cdr list))))
          (lambda (a b) (string< a b)))))
    (concat
    "#+TITLE: " title "\n\n"
    "#+BEGIN_EXPORT html\n\n"
    (sxml-to-xml
     `(div (@ (id "tag-filter-component")
              (uk-filter "target: .js-filter"))
           (div (@ (class "tags uk-subnav uk-subnav-pill"))
                (span (@ (uk-filter-control "group: tag"))
                      (a (@ (href "#")) "ALL"))
                ,(mapconcat (lambda (item)
                              (format "<span id=\"%s\" uk-filter-control=\"filter: .%s; group: tag\"><a href=\"#\">%s</a></span>"
                                      (concat "filter-" item)
                                      item
                                      item))
                            unique-tags
                            "\n"))
           (ul (@ (class "uk-subnav uk-subnav-pill"))
               (li (@ (uk-filter-control "sort: data-date; group: date"))
                   (a (@ (href "#")) "Ascending"))
               (li (@ (uk-filter-control "sort: data-date; order: desc; group: date")
                      (class "uk-active"))
                   (a (@ (href "#")) "Descending")))
           (ul (@ (class "sitemap-entries uk-list uk-list-emphasis js-filter"))
               ,(mapconcat (lambda (item) (car (car item)))
                          (cdr list)
                          "\n"))))
    "\n#+END_EXPORT\n")))

Derived backend

You can derive a custom backend from an existing one and can override certain functions. In this example, my-site-html derives from html and overrides template, link, and headline functions.

The :translate-alist part allows you to map an org element to a function handler.
The :options-alist gives you the ability to define keywords that map to export properties. You can use this for custom export properties or override existing properties.
- These are (KEYWORD OPTION DEFAULT BEHAVIOR). The full description can be read by describing the org-export-options-alist variable.
- For more on this see the following:
  - See Org-mode Export Settings.
  - https://orgmode.org/worg/dev/org-export-reference.html
  - http://doc.endlessparentheses.com/Var/org-export-options-alist.html
  - An emacs.stackexchange question I asked about how to use #+roam_tags when publishing. UPDATE: with org-roam V2, roam_tags where replaced with just org-mode’s filetags

(org-export-define-derived-backend
    'site-html
    'html
  :translate-alist
  '((template . my/org-html-template)
    (link . my/org-html-link)
    (headline . my/org-html-headline))
  :options-alist
  '((:page-type "PAGE-TYPE" nil nil t)
    (:html-use-infojs nil nil nil)
    (:updated "UPDATED" nil nil t)
    (:filetags "FILETAGS" nil nil split)))

Publishing

Output paths

This is a helper function that converts an org-mode file name to a directory of the same name, downcased and without the filename extension. So if the filename is my-post.org, a sub-directory would be created in the publishing directory called my-post/. The sitemap is indented to be at the root of the publishing directory (ie, the homepage). This function is called in the next code block.

(defun get-article-output-path (org-file pub-dir)
  (let ((article-dir (concat pub-dir
                            (downcase
                              (file-name-as-directory
                              (file-name-sans-extension
                                (file-name-nondirectory org-file)))))))
    (if (string-match "\\/sitemap.org$" org-file)
        pub-dir
        (progn
          (unless (file-directory-p article-dir)
            (make-directory article-dir t))
          article-dir))
    ))

The publishing function (and conditional TOCs)

This function does a few things:

It adds the export option to generate a table of contents only if there are more than 3 headlines. Otherwise, I don’t see a point to rendering a TOC.
Next it calls the helper function above to create the output directory and appends index.html to the result. This ends up being the article-path for a post. For example, if the filename is my-post.org, the article path would be /my-post/index.html.
Finally, it calls org-publish-org-to which publishes a file using the selected backend.

(defun my/org-html-publish-to-html (plist filename pub-dir)
  (with-current-buffer (find-file filename)
    (when (> (length (org-map-entries t)) 3)
      (insert "#+OPTIONS: toc:t\n")))
  (let ((article-path (get-article-output-path filename pub-dir)))
    (cl-letf (((symbol-function 'org-export-output-file-name)
              (lambda (extension &optional subtreep pub-dir)
                (concat article-path "index" extension))))
      (org-publish-org-to 'site-html
                          filename
                          (concat "." (or (plist-get plist :html-extension) "html"))
                          plist
                          article-path))))

The project alist

This is the configuration for the publishable projects. Each project can be published independently with org-publish and the project name (eg (org-publish "site")), or all of them with org-publish-all.

(setq org-publish-project-alist
      (list
       (list "notes.alex-miller.co"
             :base-extension "org"
             :base-directory "./"
             :publishing-function '(my/org-html-publish-to-html)
             :publishing-directory "./public"
             :auto-sitemap t
             :sitemap-function 'my/sitemap
             :sitemap-title my/sitemap-title
             :sitemap-format-entry 'my/sitemap-format-entry
             :sitemap-sort-files 'alphabetically
             :with-title nil
             :with-toc nil)
       (list "images"
             :base-extension "png\\|jpg\\|svg"
             :base-directory "./images"
             :publishing-directory "./public/images"
             :publishing-function 'org-publish-attachment)
       (list "site" :components '("notes.alex-miller.co" "images"))))

notes.alex-miller.co

This publishes the org-mode files. I keep them in the root directory. I have a few other folders for other note types that I don’t publish. The HTML output is placed in the ./public directory which is gitignored. The sitemap functions are documented above. TOCs are only generated for notes that have more than 3 headlines.

images

I sometimes link and display images in my org-notes, like plantuml renderings. I put these in the ./images directory. This basically just copies them over to the /images directory of the site. This ensure that links and/or inline images work. (See this emacs.stackexchange answer for where I got the idea).

site

It contains everything needed to build the site.

apmiller108 / slip-box Public

slip-box/publish.org