Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP
Package archive builder
Emacs Lisp Makefile
branch: master

This branch is even with nicferrier:master

Fetching latest commit…

Cannot retrieve the latest commit at this time

Failed to load latest commit information.
demo-multifile
elnode-auth
emacs-db
emacs-kv
examples
fake-package-with-elisp
recipes
.gitignore
README.creole
elpakit-run.el
elpakit-tests.el
elpakit.el

README.creole

A package and archive builder

Elpakit is a bunch of tools for authoring Emacs ELPA packages.

  • Elpakit makes package-archives for you
  • Elpakit builds multi-file packages or single file packages
  • Elpakit runs tests on packages in isolated Emacs processes
  • Elpakit runs occur on package symbols for you

An example:

(elpakit 
     "/tmp/shoesoffsaas"    ;; the directory to make the archive in
     '("~/work/elnode-auth" ;; the list of package directories
       "~/work/emacs-db"
       "~/work/shoes-off"
       "~/work/rcirc-ssh"))

Elpakit will make a package archive in "/tmp/shoesoffsaas". It will have an archive-contents formatted correctly. It will have packages made from the listed directories but also any packages that those packages depend on will be downloaded from the rest of your package archives.

Status

This is very early code and it's difficult to test a lot of it. It mostly works. You may find bugs. If you do please tell me.

Packaging Details

To use elpakit your development tree must conform to a particular standard.

Packages are directories with either:

  • a single elisp file in them which is the package file; a single-file package is constructed from these
  • a single elisp file, which is the package file, and some form of README file; a single-file package is constructed from these. Essentially the README is ignored.
  • multiple elisp files where all but one elisp file is of the form: test*.el or *test.el. The package is built from the one elisp file remaining. A test package may be built from the test files.
  • a recipes directory containing a file named for the package (eg: "elnode", it can also be different from the dev dir, eg: "emacs-db" might have a "recipes/db" recipe file). The recipes file broadly follows the MELPA conventions. It mostly specifies the :files in the dev directory which belong to the package.
    • There must be only one elisp file in the recipe for a recipe dev directory to be considered a single-file package.
  • a recipes directory, just as above, but containing many elisp files; these are considered tar packages and are built as such.

Recipe files

The recipe format is based on the MELPA format but extended to include the extra meta-data required by tar packages.

The keys that are present in a recipe file are:

  • name - the name of the package
  • version - the string version of the package
  • doc - a doc string
  • requires - the standard package requires sexp
  • files - a list of files that will be embedded in the package.
  • test - how to test the package

The files component may include directories and the directories are then included in the package tarball.

The test is a sort of recursive recipe, more information below.

Here are some examples.

db is a single file package, but the repository directory includes a tests file and the README:

(db
 :files ("db.el"))

this is the simplest form of recipe. It should not be needed very often because there is enough intelligence in elpakit about what is in a repository directory to infer this recipe.

elpakit can help with testing if you tell it about your test files and dependancies, here's the recipe for web, an HTTP client:

(web
 :files 
  ("web.el")
 :test
  (:requires
    ((fakir "0.0.10")
     (elnode "0.9"))
   :files
    ("web-test.el")))

This is still a single file recipe. The test file and it's requires are not needed for running the package, only testing it. Therefore they do not need to be packaged.

elnode is a tar package and has quite a lot of complexity:

(elnode
 :version "0.9.9.6.1"
 :doc "The Emacs webserver."
 :requires 
 ((web "0.1.4") ;; for rle
  (creole "0.8.14") ;; for wiki
  (db "0.0.1")
  (kv "0.0.9"))
 :files 
 ("elnode.el" 
  "elnode-rle.el"
  "elnode-wiki.el"
  "default-wiki-index.creole"
  "default-webserver-test.html"
  "default-webserver-image.png"
  "README.creole"
  "COPYING")
 :test
 (:requires 
  ((fakir "0.0.14"))
  :files
  ("elnode-tests.el")))

Elnode is like this because it needs to deliver the default wiki pages, these must be packaged so they can be installed at package deploy time.

elmarmalade is an example of packaging directories:

(marmalade-service
 :version "2.0.9"
 :doc "The Marmalade package store service."
 :requires 
 ((dash "1.1.0")
  (s "1.6.0")
  (s-buffer "0.0.4")
  (elnode "0.9.9.6.11")
  (htmlize "1.3.9"))
 :files
 ("marmalade-archive.el"
  "marmalade-service.el"
  "marmalade-customs.el"
  "marmalade-boot.el"
  "front-page.html"
  "login-page.html"
  "upload-page.html"
  "static/style.css") ;; the directory 'static' will be included
 :test
 (:requires
  ((fakir "0.1.1")
   (s "1.4.0"))
  :files
  ("marmalade-tests.el"
   "elnode-0.9.9.6.9.tar")))

Test packages

Test packages can be defined in the recipe or automatically inferred through the presence of files matching the pattern test.*.el or .*test.el.

Test packages are always named package-name-test.

Testing

A useful feature of elpakit is being able to build the kit and then run tests defined for a package (note: only one of the packages in the kit right now).

You can do that like this:

(elpakit-test
 '("~/work/emacs-db"
   "~/work/emacs-db-pg"
   "~/work/pg"
   "~/work/emacs-kv")
 'db-pg-tests
 'db-pg)

The output is stored in the buffer *elpakit*.

Elpakit constructs a temporary archive for the kit and then builds the packages to it and then runs an Emacs instance with your local package-archives plus the temporary one and then installs the package you've specified and then runs the tests you specified with an ERT selector.

Testing interactively

elpakit-test can be run interactively. When used like that it uses the current directory as a guess about a package dir. This means you can only package and test one package right now. Any dependancies must be present in the repositories declared in your local package-archives variable.

Testing with a daemon

You can also start an elpakit in a daemonized Emacs and optionally run tests.

(elpakit-start-server some-elpakit to-install)

will start a server, open the *elpakit-daemon* buffer and setup the to-install there.

To run tests automatically, add the test argument, for example:

(elpakit-start-server elnode-elpakit 'elnode-tests "elnode")

will install the elnode-tests package and then run tests for elnode.* in a server Emacs.

You can kill that server Emacs from the command line or from the *elpakit-daemon* buffer like this:

M-x elpakit-stop-server

Managing test processes

Elpakit includes a management tool for processes it starts. Use:

M-x elpakit-list-processes

to make a process list of elpakit started processes. You can use this to show the log of an elpakit process (key L) or to kill a running elpakit process (key K) even if it's a daemon.

Keys available:

F elpakit-process-open-emacsd Open the emacsd directory in dired
O elpakit-process-open-emacsd
f elpakit-process-open-emacs-init Open the init.el file
K elpakit-process-kill Kill the process or daemon
k elpakit-process-kill
L elpakit-process-show-buffer Show the process buffer

Refactoring Support

Elpakit includes some support for refactoring. When you are dealing with projects with lots of files (which Elpakit makes quite easy) it's common to need to refactor.

Elpakit includes elpakit-multi-occur which can show you every reference to a symbol within your Elpakit project.

elpakit-isearch-hook-jack-in can be added to emacs-lisp-mode-hook to connect elpakit-multi-occur to isearch.

It might also be advisable to connect elpakit-multi-occur to a keybinding in the normal emacs-lisp-mode. This is left up to you but I suggest:

(define-key emacs-lisp-mode-map (kbd "M-o") 'elpakit-multi-occur)

Notes on the differences with MELPA and Cask

I once hoped to merge elpakit with MELPA and Cask but it doesn't seem possible. So I build elpakit purely for my own ends these days.

The MELPA recipe standard has no:

  • version
  • doc
  • requires
  • or test sections

as far as I can see. These have been added to elpakit to make the task of building tar files possible without having to build the -pkg.el file.

MELPA doesn't even need the version since it pulls it from the GIT commit?

A side aim for elpakit is that it makes defining a package partly a function of the package's source control. Just like it normally is with other package systems (dpkg, rpm, pip, etc...).

MELPA specifies the recipes elsewhere than the repository source control. It's obvious why this has happened, they needed to support people MELPA packaging repositories they did not have commit access to.

I don't know how to solve this problem, maybe the MELPA guys could switch their code to autodetect repository provided recipes somehow?

Elpakit covers pretty much the same ground as Cask. The Cask program uses an external binary though while elpakit is just elisp. Elpakit also uses file based package archives quite a lot, while Cask seems to prefer HTTP. This requires an HTTP server which I don't think is a necessary step at all.

Useful tips

There are a bunch of useful things you can do if you're building apps that are collections of elpa packages.

Building a tar package

elpakit has all the logic to build multi-file packages so it made sense to make that available with a command:

M-x elpakit-make-multi

will use the directory you are in as a package directory (if it can be established that it is a package) or ask you to specify a package directory. It will then build that package for you in a temorary directory. It will open the temporary directory with the built package in it. You can use that to upload to Marmalade or some such.

Making a multi-file package branch for MELPA

MELPA requires multi-file packages repositories to be packaged in a particular way. Namely that they should have the pkg.el file present in the repository.

elpakit will automatically build an orphan branch for you in your repository for use by MELPA.

Just set the customization variable elpakit-do-melpa-on-multi-file-package to true and then use elpakit-make-multi on a git repository based multi-file package and elpakit will make the branch for you. Elpakit won't push the branch so you get to review stuff before it goes back to your origin.

Evaling an elpakit

You can build an elpakit entirely inside your emacs. This basically means just evaling all the lisp it finds.

(elpakit-eval   
 '("~/work/elnode-auth"
   "~/work/emacs-db"
   "~/work/shoes-off"
   "~/work/rcirc-ssh"))

Sometimes you have to muck about with the ordering of the kit to make it work because there are dependancies (requires) that don't work unless you eval things in a particular order.

A remote elpakit destination

Elpakit doesn't care what the destination is as long as it looks like a directory to Emacs. That means you can use TRAMP:

(elpakit "/ssh:my-remote-host.example.com/myapp-elpa/"
   '("~/work/elnode-auth"
     "~/work/emacs-db"
     "~/work/shoes-off"))

This is useful for deploying packages to remote locations, for example "live" in an elnode app.

Copyable elpakits

An alternative to the push to remote elpakit is building it locally and having the remote pull it. This is possible too, especially with a bit of elnode magic.

(elpakit "/tmp/my-archive"
   '("~/work/elnode-auth"
     "~/work/emacs-db"
     "~/work/shoes-off"))
(elnode-make-webserver "/tmp/my-archive" :port 8007 :host "0.0.0.0")

and on the remote use:

wget -r -np http://elnode-server:8007

to get the package archive built by elpakit.

Defining kits to be reusable

Just collecting the list of package directories into a list means you can do lots of different things with elpakit:

(defconst shoes-off-elpakit
  '("~/work/shoes-off"
    "~/work/rcirc-ssh"
    "~/work/emacs-db"
    "~/work/esxml"
    "~/work/elnode-auth"
    "~/work/emacs-kv"
    "~/work/shoes-off-aas/talkapp"))

Then you can:

(elpakit "~/my-app-elpa" shoes-off-elpakit)

If you use a defconst then you can re-eval it more easily.

Building kits remotely

If you want to deploy to a remote host without pushing to an official repository you can still do that, just use tramp for the destination:

(elpakit
 "/ssh:nic@livehost.example.com:apps/app1/app-elpa"
 my-elpakit)

Presuming my-elpakit is an elpakit list. Elpakit uses a straight copy so tramp works fine.

Starting a server

Starting a server and running some tests in it:

(setq nic-server
   (elpakit-start-server 
        shoes-off-train 'talkapp :test t))

Server's can be started with extra lisp to initialize them, in this case they do not auto-require the install target.

So here's a quick snippet to start a server with an archive, copy some files in to the package location, then require the main thing:

(setq nic-server
      (elpakit-start-server
       shoes-off-train
       'talkapp
       :extra-lisp
       '(progn
         (shell-command
          (format
           "cp -r ~/work/teamchat/talkapp/*db*.elc %s"
           (file-name-directory
            (find-lisp-object-file-name
             'talkapp-start
             (symbol-function 'talkapp-start)))))
         (require (quote talkapp)))))

Note the clever hack to find the package location based on an autoload specified function.

Here's a more clever version of that wrapped up in an interactive, defun, very useful startup routine:

(defvar talkapp-elpakit-server nil)

;; Start the talkapp in an elpakit server
(defun talkapp-do-elpakit-server ()
  (interactive)
  (condition-case err
      (progn
        (server-eval-at (cdr talkapp-elpakit-server) '1)
        (error
         "talkapp-elpakit-server already running %s"
         (cdr talkapp-elpakit-server)))
    (error (if (string-match "^No such server" (cadr err))
               (setq talkapp-elpakit-server
                     (elpakit-start-server
                      shoes-off-train
                      'talkapp
                      :extra-lisp
                      ;; Copy in the db files from the working dir...
                      '(progn
                        (shell-command
                         (format
                          "cp -r ~/work/teamchat/talkapp/*db*.elc %s"
                          (file-name-directory
                           (find-lisp-object-file-name
                            'talkapp-start
                            (symbol-function 'talkapp-start)))))
                        (require (quote talkapp)))))
               ;; Else rethrow
               (signal (car err) (cdr err))))))

It only allows one at a time atm but that's still better than trying to do everything inside a single emacs.

Making a package from your ELPA

Elpakit will also let you make a package from your currently installed ELPA packages. This is useful if you want to share a lot of dependencies with friends or colleagues.

M-x elpakit-package-list-buf

Makes a buffer with a list of your packages in it.

In the ELPA package list buffer:

Key What it does
k kill the current package
M make a new package file from the remaining package names

The package file that it makes can then be checked into Git (to be made available with MELPA) or uploaded to marmalade-repo.

Glossary

A package is either a single EmacsLisp file with a special header or a collection of EmacsLisp and possibly other files (like an info file or HTML files) in a tarball. A single file EmacsLisp package is called a single file package and a tar package is called a multi-file package. Elpakit deals with either single or multi file packages and can build a multi-file package for you (a non-trivial process) from a collection of source files.

A package dir is a directory in which the source files for a package are kept. Probably this is a checkout of a version control repository but need not be.

A recipe is a file that tells Elpakit how to put together a package. It is found in a package-dir. For single file packages this is almost never necessary. For multi-file packages it is necessary. The recipe is necessary for multi-file packages because it contains information about the package that has no other home and because Elpakit cannot guess what files in your tree you actually want to be present in your resulting package.

An archive is a place you can install packages from. Archives can either be http URLs or a local directory. Elpakit can make archives from collections of package dirs. To actually install from a particular archive you need to add the archive to the Emacs variable package-archives.

An elpakit is a list of package-dir that are going to be built into an archive. Once the archive is built we can do more with it, such as running tests on the packages we've built (which will be clean and not tainted by any local evaluation you have done in your local Emacs).

A local emacs is the Emacs you are writing code and running Elpakit in, as opposed to an Emacs instance you start to do testing or such.

TODO

  • include any dir file and info files in multi-file packages
  • add a walk through constructor process for recipe files?
  • add a mode (other than lisp-mode) for editing recipe files?
  • add the URL to the package in elpakit/make-pkg-lisp
    • should go into define-package's extra-properties
  • add upstream repository list to a recipe
    • so that a package could specify where it's dependant repositorys can be found
    • this is mainly so a package could be installed from a url and specify that depends come from marmalade (for example)
Something went wrong with that request. Please try again.