Skip to content
This repository

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

emacs-lisp version of the marmalade package repository

branch: master
Octocat-spinner-32 marmalade-repo-test add example packages for test reasons June 11, 2013
Octocat-spinner-32 recipes add all the static resources to the package March 11, 2014
Octocat-spinner-32 static bunch of bootstrap updates July 31, 2013
Octocat-spinner-32 .gitignore add the archive caches to ignore June 18, 2013
Octocat-spinner-32 README.creole notes about eventual consistency June 26, 2013
Octocat-spinner-32 deploy.el minor deploy tweaks July 31, 2013
Octocat-spinner-32 dummy-package.el update the version on the dummy March 11, 2014
Octocat-spinner-32 elnode-0.9.9.6.9.tar add auth, package handling and a first test for package handling. April 23, 2013
Octocat-spinner-32 front-page.html a few changes to the css July 31, 2013
Octocat-spinner-32 login-page.html bunch of bootstrap updates July 31, 2013
Octocat-spinner-32 marmalade-archive.el fix the archive reference March 11, 2014
Octocat-spinner-32 marmalade-boot.el changes to get deployment going June 04, 2013
Octocat-spinner-32 marmalade-customs.el add marmalade-archive-dir to store where archive-contents cache files go June 17, 2013
Octocat-spinner-32 marmalade-mongo.el melpa branch created by elpakit April 15, 2013
Octocat-spinner-32 marmalade-service.el add a simple upload check message April 09, 2014
Octocat-spinner-32 marmalade-tests.el fix marmalade/upload test (requires elnode fixes too) April 19, 2014
Octocat-spinner-32 marmalade-users.el fix the location of the marmalade users db so it doesn't overwrite th… April 09, 2014
Octocat-spinner-32 mongo-convert updates to show how to do stuff. May 30, 2013
Octocat-spinner-32 nginx.conf the nginx config we use June 22, 2013
Octocat-spinner-32 todo.org testing elmarmalade--package-list August 15, 2012
Octocat-spinner-32 upload-page.html bunch of bootstrap updates July 31, 2013
README.creole

The Marmalade Service - A package repository in EmacsLisp

This is an EmacsLisp package to reimplement the marmalade-repo with Elnode. Nathan Weizenbaum wrote the original marmalade-repo with node.js and mongodb.

This reimplementation uses an Elnode application dealing with a filesystem of all packages. It's basically a file server app.

This packages includes conversion tools from the mongodb to the filesystem.

These are the things marmalade should do:

  • construct the package archive file from the package directory structure
    • the package archive is an index of all the packages in the repository.
  • allow the archive to be updated with uploads
  • provide the ELPA /package/ HTTP API for downloading packages
  • provide HTML front end to the /package/ HTTP API
  • provide EmacsLisp front end to the /package/ HTTP API
    • this will likely be a separate project
  • provide an HTML front end to the package archives with some discovery

Rollout

Some notes on rollout. Marmalade is going to need more sophistication than I've ever managed to build with elnode. It's basically a collection of microservices, which is always what has been attractive about elnode... it's just I've never managed to put it all together.

Marmalade will require multiple servers because the archive stuff is very expensive, because we're using emacs-db for now and that requires concentrating access into a single place.

  • install the package on the server
    • it needs to know where the package archive is
(customize-set-variable
 'marmalade-package-store-dir
 (file-name-as-directory 
   (expand-file-name "~/marmalade/packages")))
  • have a daemon file for the package
  • nginx script
    • which depends on your port number
  • ssl? optional
  • we need 2 emacs instances
    • 1 to serve archive
    • 1 to serve everything else

Current problems

Handling errors better

Here is the upload function

(let ((package-file-name
       (marmalade/save-package upload-file base-file-name)))
  (elnode-send-redirect httpcon package-file-name 201))

which calls this the save package function

(with-temp-file temp-package-file
  (insert-string
   (substring-no-properties package-data)))
;; Now get the real path
(marmalade/package-path temp-package-file))))

which makes a temp file and puts the package data in it and then uses this

(defun marmalade/package-path (package-file)
  "Turn PACKAGE-FILE into a repository package path."
  (let* ((info (marmalade/package-info package-file))
         (version (elt info 3))

which uses this to get the meta data of the existing (temp) package file.

(defun marmalade/package-info (package-file)
  "Return the package-info on the PACKAGE-FILE.

PACKAGE-FILE is either an ELISP or a TAR file to be uploaded to
the package repository."
  (cond
    ((string-match "\\.el$" package-file)
     (with-temp-buffer
       (insert-file-contents-literally package-file)
       (buffer-string) ; leave it in so it's easy to debug
       (package-buffer-info)))
    ((string-match "\\.tar$" package-file)
     (package-tar-file-info package-file))
    (t (error "Unrecognized extension `%s'"
              (file-name-extension package-file)))))

And that, I think, is where a lot of error checking needs to be done and reported from.

Currently we have this which tests the package save:

(ert-deftest marmalade/save-package ()
  "Test the save package stuff."
  (let ((marmalade-package-store-dir "/tmp/test-marmalade-dir"))
    (marmalade/package-file
     :code
     (let ((expected
            "/tmp/test-marmalade-dir/dummy-package/0.0.1/dummy-package-0.0.1.el"))
       (should
        (equal
         (marmalade/save-package
          package-content-string
          "dummy-package.el")
         expected))
       (should
        (equal
         (fakir-file-path package-file)
         expected))))))

It doesn't test much about it though. We need a lot of invalid packages to be tested.

Eventually consistent

The archive is maintained separately from the filesystem. It is possible that a delete is performed on the filesystem but the archive that a user has just downloaded still shows the package. In this case the actual package retrieval will fail and the user will, presumably, refresh the package archive and the package will be shown removed.

Admin tasks

For reference (and in case I get run over or something) here's a grab bag of marmalade tasks.

To close down node.js/mongo marmalade-repo

We will need to:

  • shut off the node.js app so people can't upload
  • do a full dump of the db
  • run mongo-covert to export
  • rsync the files to the new host
  • turn on new marmalade uploader

Syncing the mongodb dump from Marmalade v1

Here's the cron I run to keep a local copy of marmalade:

rsync --bwlimit=20 -av \
      --files-from=<(ssh $MARMALADE_HOST find seville-marmalade/mongodump -cmin -60) \
      -e ssh $MARMALADE_HOST:. $DESTINATION_DIR

The dump is pretty big so I'm not getting the latest of everything. Just the last hour and I run it twice a day.

Importing a mongodb dump from Marmalade v1

Marmalade v1 dumps it's mongodb to a file every hour. I pull those files to my house and then use them to setup a local copy of the marmalade database that I use to generate the marmalade v2 filesystem.

How to restore the database:

mongorestore $DIRECTORY

the $DIRECTORY should be an individual full-dump directory.

I have a directory full of dumps so this is how I actually do it:

mongorestore ~/seville-marmalade/mongodump/$(ls -t ~/seville-marmalade/mongodump/ | head -n2 | tail -n1)

that chooses the last but one dump all the time.

Turning the mongodb into a file system

A simple bash script uses mongofiles to export the mongo database into a marmalade file system:

bash mongo-convert

This uses ~/marmalade/packages as the package root, by default.

Sync the extracted file system

mongofiles doesn't update files, it just splats over them so you need a smarter rsync to move them to another host:

rsync -catve ssh "$MARMALADE_HOST:marmalade/packages/*" marmalade/packages

It would be really useful to understand the incomming packages on that rsync and to call the internal archive-contents update urls. Some sort of grep on the rsync output could probably achieve that.

This would mean new marmalade could be a mirror of old marmalade for a while, before we switch over.

Deploy marmalade

A deploy script is included in the repository.

This is the sequence of things you need to do to deploy:

$ emacs-local/bin/emacsclient -s /tmp/emacs1000/marmalade -e '(kill-emacs)'
$ rm -rf marmalade-repo
$ emacs-local/bin/emacs --script deploy.el marmalade-repo
$ HOME=$(realpath marmalade-repo) emacs-local/bin/emacs --daemon=marmalade

The steps are:

1. kill any existing daemon

2. remove the existing deployment (not the data)

3. deploy the package and it's dependancies

4. start the daemon

Once the Emacs instance is started you can attach to it like this:

$ emacs-local/bin/emacsclient -s /tmp/emacs1000/marmalade $(realpath marmalade-repo)

The argument $(realpath marmalade-repo only ensures we open some directory. Any directory or file could be given, as ever with emacsclient.

Note on the package archives file: currently the archive file has to be generated manually and it's rather slow. Use M-x marmalade/package-archives to generate it.

Something went wrong with that request. Please try again.