This reimplementation uses an Elnode application dealing with a filesystem of all packages. It's basically a file server app.
This packages includes conversion tools from the mongodb to the filesystem.
These are the things marmalade should do:
- construct the package archive file from the package directory structure
- the package archive is an index of all the packages in the repository.
- allow the archive to be updated with uploads
- provide the ELPA /package/ HTTP API for downloading packages
- provide HTML front end to the /package/ HTTP API
- provide EmacsLisp front end to the /package/ HTTP API
- this will likely be a separate project
- provide an HTML front end to the package archives with some discovery
Some notes on rollout. Marmalade is going to need more sophistication than I've ever managed to build with elnode. It's basically a collection of microservices, which is always what has been attractive about elnode... it's just I've never managed to put it all together.
Marmalade will require multiple servers because the archive stuff is very expensive, because we're using emacs-db for now and that requires concentrating access into a single place.
- install the package on the server
- it needs to know where the package archive is
(customize-set-variable 'marmalade-package-store-dir (file-name-as-directory (expand-file-name "~/marmalade/packages")))
- have a daemon file for the package
- nginx script
- which depends on your port number
- ssl? optional
- we need 2 emacs instances
- 1 to serve archive
- 1 to serve everything else
Here is the upload function
(let ((package-file-name (marmalade/save-package upload-file base-file-name))) (elnode-send-redirect httpcon package-file-name 201))
which calls this the save package function
(with-temp-file temp-package-file (insert-string (substring-no-properties package-data))) ;; Now get the real path (marmalade/package-path temp-package-file))))
which makes a temp file and puts the package data in it and then uses this
(defun marmalade/package-path (package-file) "Turn PACKAGE-FILE into a repository package path." (let* ((info (marmalade/package-info package-file)) (version (elt info 3))
which uses this to get the meta data of the existing (temp) package file.
(defun marmalade/package-info (package-file) "Return the package-info on the PACKAGE-FILE. PACKAGE-FILE is either an ELISP or a TAR file to be uploaded to the package repository." (cond ((string-match "\\.el$" package-file) (with-temp-buffer (insert-file-contents-literally package-file) (buffer-string) ; leave it in so it's easy to debug (package-buffer-info))) ((string-match "\\.tar$" package-file) (package-tar-file-info package-file)) (t (error "Unrecognized extension `%s'" (file-name-extension package-file)))))
And that, I think, is where a lot of error checking needs to be done and reported from.
Currently we have this which tests the package save:
(ert-deftest marmalade/save-package () "Test the save package stuff." (let ((marmalade-package-store-dir "/tmp/test-marmalade-dir")) (marmalade/package-file :code (let ((expected "/tmp/test-marmalade-dir/dummy-package/0.0.1/dummy-package-0.0.1.el")) (should (equal (marmalade/save-package package-content-string "dummy-package.el") expected)) (should (equal (fakir-file-path package-file) expected))))))
It doesn't test much about it though. We need a lot of invalid packages to be tested.
The archive is maintained separately from the filesystem. It is possible that a delete is performed on the filesystem but the archive that a user has just downloaded still shows the package. In this case the actual package retrieval will fail and the user will, presumably, refresh the package archive and the package will be shown removed.
For reference (and in case I get run over or something) here's a grab bag of marmalade tasks.
We will need to:
- shut off the node.js app so people can't upload
- do a full dump of the db
- run mongo-covert to export
- rsync the files to the new host
- turn on new marmalade uploader
Here's the cron I run to keep a local copy of marmalade:
rsync --bwlimit=20 -av \ --files-from=<(ssh $MARMALADE_HOST find seville-marmalade/mongodump -cmin -60) \ -e ssh $MARMALADE_HOST:. $DESTINATION_DIR
The dump is pretty big so I'm not getting the latest of everything. Just the last hour and I run it twice a day.
Marmalade v1 dumps it's mongodb to a file every hour. I pull those files to my house and then use them to setup a local copy of the marmalade database that I use to generate the marmalade v2 filesystem.
How to restore the database:
the $DIRECTORY should be an individual full-dump directory.
I have a directory full of dumps so this is how I actually do it:
mongorestore ~/seville-marmalade/mongodump/$(ls -t ~/seville-marmalade/mongodump/ | head -n2 | tail -n1)
that chooses the last but one dump all the time.
A simple bash script uses mongofiles to export the mongo database into a marmalade file system:
This uses ~/marmalade/packages as the package root, by default.
mongofiles doesn't update files, it just splats over them so you need a smarter rsync to move them to another host:
rsync -catve ssh "$MARMALADE_HOST:marmalade/packages/*" marmalade/packages
It would be really useful to understand the incomming packages on that rsync and to call the internal archive-contents update urls. Some sort of grep on the rsync output could probably achieve that.
This would mean new marmalade could be a mirror of old marmalade for a while, before we switch over.
A deploy script is included in the repository.
This is the sequence of things you need to do to deploy:
$ emacs-local/bin/emacsclient -s /tmp/emacs1000/marmalade -e '(kill-emacs)' $ rm -rf marmalade-repo $ emacs-local/bin/emacs --script deploy.el marmalade-repo $ HOME=$(realpath marmalade-repo) emacs-local/bin/emacs --daemon=marmalade
The steps are:
1. kill any existing daemon
2. remove the existing deployment (not the data)
3. deploy the package and it's dependancies
4. start the daemon
Once the Emacs instance is started you can attach to it like this:
$ emacs-local/bin/emacsclient -s /tmp/emacs1000/marmalade $(realpath marmalade-repo)
The argument $(realpath marmalade-repo only ensures we open some directory. Any directory or file could be given, as ever with emacsclient.
Note on the package archives file: currently the archive file has to be generated manually and it's rather slow. Use M-x marmalade/package-archives to generate it.