A wiki/blogging engine with a static file back-end
Jupyter Notebook Hy CSS JavaScript HTML Python Makefile
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
.vscode Merge with private fork Apr 30, 2016
pages Added new entry Mar 19, 2016
sushy merge with production from taoofmac.com Jun 5, 2016
themes fixed CSS references Jun 5, 2016
tools added profiling to Makefile Dec 24, 2014
.gitattributes fix line endings on .tpl files under Cygwin Jan 31, 2016
.gitignore
.python-version bumped runtime version Sep 29, 2015
DOKKU_SCALE DOKKU_SCALE Dec 6, 2015
ENV Merge with private fork Apr 30, 2016
LICENSE Initial commit Sep 23, 2014
Makefile Merge with private fork Apr 30, 2016
Procfile.dokku added procfile variations Apr 30, 2016
Procfile.piku added procfile variations Apr 30, 2016
README.md README Jun 5, 2016
fig.yml Added test Dockerfile and fig config Dec 14, 2014
requirements-dev.txt README and requirements cleanup Jun 5, 2016
requirements.txt README and requirements cleanup Jun 5, 2016
uwsgi.ini first pass at indexing internal links between pages Dec 17, 2014

README.md

Stories in Ready

Sushy

A wiki/blogging engine with a static file back-end.

Demo site: http://taoofmac.com.

Status

Currently working out-of-the box, with full-text indexing and markup support already in place. Deployable now to piku/Dokku-alt/Dokku/Heroku.

Roadmap

  • A little more documentation (there's never enough)
  • Blog archive
  • Docker deployment (currently deploying on piku and Dokku using specific Procfiles)

Done

  • Page aliasing (i.e., multiple URLs for a page)
  • Image thumbnailing
  • Friendlier search results
  • More CSS tweaks
  • Atom feeds
  • piku deployment
  • Blog homepage/prev-next navigation
  • Preliminary support for rendering IPython notebooks
  • Closest-match URLs (i.e., fix typos) (removed for performance concerns on large sites)
  • HTTP caching (Etag, Last-Modified, HEAD support, etc.)
  • Sitemap
  • OpenSearch support (search directly from the omnibar on some browsers)
  • CSS inlining for Atom feeds
  • multiprocessing-based indexer (in feature/multiprocessing, disabled for ease of profiling)
  • SSE (Server-Sent Events) support (in feature/server-events) for notifying visitors a page has changed
  • New Relic Support
  • Internal link tracking (SeeAlso functionality, as seen on Yaki)
  • Multiple theme support (only the one theme for now)
  • Automatic insertion of image sizes in img tags
  • Deployable under Dokku-alt
  • Run under uWSGI using gevent workers
  • Full-text indexing and search
  • Syntax highlighting for inline code samples
  • Ink-based site layout and templates (replaced by a new layout in the feature/blog branch)
  • Baseline markup rendering (Textile, Markdown and ReST)

Stuff that will never happen:

  • Site thumbnailing (for taking screenshots of external links) - moved to a separate app
  • Web-based UI for editing pages (you're supposed to do this out-of-band)
  • Revision history (you're supposed to manage your content with Dropbox or git)
  • Comment support

Principles of Operation

  • All your Textile, Markdown or ReStructured Text content lives in a filesystem tree, with a folder per page
  • Sushy grabs and renders those on demand with fine-tuned HTTP headers (this is independently of whether or not you put Varnish or CloudFlare in front for caching)
  • It also maintains a SQLite database with a full-text index of all your content - updated live as you add/edit content.

Markup Support

Sushy supports plaintext, HTML and Textile for legacy reasons, and Markdown as its preferred format. ReStructured Text is also supported, but since I don't use it for anything (and find it rather a pain to read, let alone write), I can't make any guarantees as to its reliability. Work is ongoing for supporting Jupyter notebooks (which have no metadata/frontmatter conventions).

All markup formats MUST be preceded by "front matter" handled like RFC2822 headers (see the pages folder for examples and test cases). Sushy uses the file extension to determine a suitable renderer, but that can be overriden if you specify a Content-Type header (see config.hy for the mappings).

FAQ

Why?

I've been running a classical, object-oriented Python Wiki (called Yaki) for the better part of a decade. It works, but is comparatively big and has become unwieldy and cumbersome to tweak. So I decided to rewrite it. Again. And again.

And I eventually decided to make it smaller -- my intention is for the core to stop at around 1000 lines of code excluding templates, so this is also an exercise in building tight, readable (and functional) code.

Why Hy?

Because I've been doing a lot of Clojure lately for my other personal projects, and both the LISP syntax and functional programming style are quite natural to me.

I thought long and hard about doing this in Clojure instead (and in fact have been poking at an implementation for almost a year now), but the Java libraries for Markdown and Textile have a bunch of irritating little corner cases and I wanted to make sure all my content would render fine the first time, plus Python has an absolutely fantastic ecosystem that I am deeply into.

Then Hy came along, and I realized I could have my cake and eat it too.

Can this do static sites?

I've used a fair amount of static site generators, and they all come up short on a number of things (namely trivially easy updates that don't involve re-generating hundreds of tiny files and trashing the filesystem) -- which, incidentally, is one of the reasons why Sushy relies on a single SQLite file for temporary data.

But there's no reason why this can't be easily modified to pre-render and save the HTML content after indexing runs -- pull requests to do that are welcome.

Requirements

Thanks to Hy, this should run just as well under Python 2 and Python 3. My target environment is 2.7.8/PyPy, though, so your mileage may vary. Check the requirements.txt file - I've taken pains to make sure dependencies are there for a reason and not just because they're trendy.


Deployment

This repository should be deployable on piku (my featherweight version of Heroku) and Dokku -- where it will instantiate a production-ready uWSGI server (using gevent) and a background indexing worker.

As is (for development) the content ships with the code repo. Changing things to work off a separate mount point (or a shared container volume) is trivial.

Configuration

In accordance with the 12 Factor approach, runtime configuration is taken from environment variables:

  • DEBUG - Enable debug logs
  • PROFILER - Enable cProfile statistics (will slow down things appreciatively)
  • CONTENT_PATH - the folder your documents live in
  • THEME_PATH - path under which static assets (JS/CSS/etc.) and templates/views are stored
  • BIND_ADDRESS - IP address to bind the development server to
  • PORT - TCP port to bind the server to
  • NEW_RELIC_LICENSE_KEY - If set, the New Relic agent is enabled

These are set in the Makefile (which I use for a variety of purposes).


Sample Dokku Configuration

Deploying Sushy in dokku implies setting things up so that the separate processes (web and worker, which are instantiated in separate Docker containers) can share at least one storage volume (two if you need to keep the indexing database and the content tree separate).

The example below shows a finished configuration that mounts the /srv/data/sushy host path inside both containers as /app/data, so that both can access the index and the content tree:

Docker options for exposing host directory as shared volume

$ dokku docker-options staging.no-bolso.com
Deploy options:
    -v /srv/data/sushy:/app/data
Run options:
    -v /srv/data/sushy:/app/data

Application Settings

$ dokku config staging.no-bolso.com
=====> staging.no-bolso.com config vars
CONTENT_PATH:      /app/data/space
DATABASE_PATH:     /app/data/sushy.db
DEBUG:             True
THEME_PATH:        themes/blog

Trying it out

Make sure you have libxml and libxslt headers, as well as the JPEG library - the following is for Ubuntu 14.04:

sudo apt-get install libxml2-dev libxslt1-dev libjpeg-dev
# install dependencies
make deps
# run the indexing daemon (updates upon file changes)
make index-watch &
# run the standalone server (or uwsgi)
make serve