From 8d20fef109ef40bfc4192d6fcf10b7d781998025 Mon Sep 17 00:00:00 2001 From: Nicholas Ferrier Date: Mon, 21 Apr 2014 01:45:21 +0100 Subject: [PATCH] add new html page and re-org the doc --- README.creole | 253 ++------------------------------------- admin-tasks.creole | 116 ++++++++++++++++++ register-comingsoon.html | 39 ++++++ 3 files changed, 167 insertions(+), 241 deletions(-) create mode 100644 admin-tasks.creole create mode 100644 register-comingsoon.html diff --git a/README.creole b/README.creole index 7389f5f..b3cd287 100644 --- a/README.creole +++ b/README.creole @@ -22,253 +22,24 @@ These are the things marmalade should do: * provide HTML front end to the {{{/package/}}} HTTP API * provide EmacsLisp front end to the {{{/package/}}} HTTP API ** this will likely be a separate project -* provide an HTML front end to the package archives with some discovery -== Rollout == +==== ETA ==== -Some notes on rollout. Marmalade is going to need more sophistication -than I've ever managed to build with elnode. It's basically a -collection of microservices, which is always what has been attractive -about elnode... it's just I've never managed to put it all together. +Current expected ETA for live elmarmalade: end of May. -Marmalade will require multiple servers because the archive stuff is -very expensive, because we're using emacs-db for now and that requires -concentrating access into a single place. +node.js marmalade has been broken for some time with the web +upload. This is caused by major outdatedness in the node.js that +marmalade was built on. It is very crufty and tedious to update. +So Nic Ferrier is banking on getting this live. Sadly he has a +demanding job at the moment so doesn't get the time to hack on this +that he'd like. -* install the package on the server -** it needs to know where the package archive is -{{{ -(customize-set-variable - 'marmalade-package-store-dir - (file-name-as-directory - (expand-file-name "~/marmalade/packages"))) -}}} +==== Other marmalade things ==== -* have a daemon file for the package -* nginx script -** which depends on your port number -* ssl? optional -* we need 2 emacs instances -** 1 to serve archive -** 1 to serve everything else +It would be nice if marmalade provided better package discovery. -== Current problems == +It would be nice if marmalade stored licences in a clear way and +allowed you to use only safe licences if you wish. -=== Handling errors better === - -Here -is -[[https://github.com/nicferrier/elmarmalade/blob/master/marmalade-service.el#L150|the upload function]] - -{{{ -(let ((package-file-name - (marmalade/save-package upload-file base-file-name))) - (elnode-send-redirect httpcon package-file-name 201)) -}}} - -which calls -this -[[https://github.com/nicferrier/elmarmalade/blob/master/marmalade-service.el#L121|the save package function]] - -{{{ -(with-temp-file temp-package-file - (insert-string - (substring-no-properties package-data))) -;; Now get the real path -(marmalade/package-path temp-package-file)))) -}}} - -which makes a temp file and puts the package data in it and then -uses -[[https://github.com/nicferrier/elmarmalade/blob/master/marmalade-service.el#L121|this]] - -{{{ -(defun marmalade/package-path (package-file) - "Turn PACKAGE-FILE into a repository package path." - (let* ((info (marmalade/package-info package-file)) - (version (elt info 3)) -}}} - -which -uses -[[https://github.com/nicferrier/elmarmalade/blob/master/marmalade-service.el#L67|this]] -to get the meta data of the existing (temp) package file. - - -{{{ -(defun marmalade/package-info (package-file) - "Return the package-info on the PACKAGE-FILE. - -PACKAGE-FILE is either an ELISP or a TAR file to be uploaded to -the package repository." - (cond - ((string-match "\\.el$" package-file) - (with-temp-buffer - (insert-file-contents-literally package-file) - (buffer-string) ; leave it in so it's easy to debug - (package-buffer-info))) - ((string-match "\\.tar$" package-file) - (package-tar-file-info package-file)) - (t (error "Unrecognized extension `%s'" - (file-name-extension package-file))))) -}}} - -And that, I think, is where a lot of error checking needs to be done -and reported from. - -Currently we -have -[[https://github.com/nicferrier/elmarmalade/blob/master/marmalade-tests.el#L200|this]] -which tests the package save: - - -{{{ -(ert-deftest marmalade/save-package () - "Test the save package stuff." - (let ((marmalade-package-store-dir "/tmp/test-marmalade-dir")) - (marmalade/package-file - :code - (let ((expected - "/tmp/test-marmalade-dir/dummy-package/0.0.1/dummy-package-0.0.1.el")) - (should - (equal - (marmalade/save-package - package-content-string - "dummy-package.el") - expected)) - (should - (equal - (fakir-file-path package-file) - expected)))))) -}}} - -It doesn't test much about it though. We need a lot of invalid -packages to be tested. - -=== Eventually consistent === - -The archive is maintained separately from the filesystem. It is -possible that a delete is performed on the filesystem but the archive -that a user has just downloaded still shows the package. In this case -the actual package retrieval will fail and the user will, presumably, -refresh the package archive and the package will be shown removed. - - -== Admin tasks == - -For reference (and in case I get run over or something) here's a grab -bag of marmalade tasks. - - -=== To close down node.js/mongo marmalade-repo === - -We will need to: - -* shut off the node.js app so people can't upload -* do a full dump of the db -* run mongo-covert to export -* rsync the files to the new host -* turn on new marmalade uploader - -=== Syncing the mongodb dump from Marmalade v1 === - -Here's the cron I run to keep a local copy of marmalade: - -{{{ -rsync --bwlimit=20 -av \ - --files-from=<(ssh $MARMALADE_HOST find seville-marmalade/mongodump -cmin -60) \ - -e ssh $MARMALADE_HOST:. $DESTINATION_DIR -}}} - -The dump is pretty big so I'm not getting the latest of -everything. Just the last hour and I run it twice a day. - -=== Importing a mongodb dump from Marmalade v1 === - -Marmalade v1 dumps it's mongodb to a file every hour. I pull those -files to my house and then use them to setup a local copy of the -marmalade database that I use to generate the marmalade v2 filesystem. - -How to restore the database: - -{{{ -mongorestore $DIRECTORY -}}} - -the {{{$DIRECTORY}}} should be an individual full-dump directory. - -I have a directory full of dumps so this is how I actually do it: - -{{{ -mongorestore ~/seville-marmalade/mongodump/$(ls -t ~/seville-marmalade/mongodump/ | head -n2 | tail -n1) -}}} - -that chooses the last but one dump all the time. - - -=== Turning the mongodb into a file system === - -A simple bash script uses {{{mongofiles}}} to export the mongo -database into a marmalade file system: - -{{{ -bash mongo-convert -}}} - -This uses {{{~/marmalade/packages}}} as the package root, by default. - - -=== Sync the extracted file system === - -{{{mongofiles}}} doesn't update files, it just splats over them so you -need a smarter {{{rsync}}} to move them to another host: - -{{{ -rsync -catve ssh "$MARMALADE_HOST:marmalade/packages/*" marmalade/packages -}}} - -//It would be really useful// to understand the incomming packages on -that rsync and to call the internal archive-contents update urls. Some -sort of grep on the rsync output could probably achieve that. - -This would mean new marmalade could be a mirror of old marmalade for a -while, before we switch over. - -=== Deploy marmalade === - -A deploy script is included in the repository. - -This is the sequence of things you need to do to deploy: - -{{{ -$ emacs-local/bin/emacsclient -s /tmp/emacs1000/marmalade -e '(kill-emacs)' -$ rm -rf marmalade-repo -$ emacs-local/bin/emacs --script deploy.el marmalade-repo -$ HOME=$(realpath marmalade-repo) emacs-local/bin/emacs --daemon=marmalade -}}} - -The steps are: - -1. kill any existing daemon - -2. remove the existing deployment (//not// the data) - -3. deploy the package and it's dependancies - -4. start the daemon - -Once the Emacs instance is started you can attach to it like this: - -{{{ -$ emacs-local/bin/emacsclient -s /tmp/emacs1000/marmalade $(realpath marmalade-repo) -}}} - -The argument {{{$(realpath marmalade-repo}}} only ensures we open some -directory. Any directory or file could be given, as ever with -{{{emacsclient}}}. - -//Note on the package archives file: currently the archive file has to -be generated manually and it's rather slow. Use M-x -marmalade/package-archives to generate it.// diff --git a/admin-tasks.creole b/admin-tasks.creole new file mode 100644 index 0000000..6a28c3d --- /dev/null +++ b/admin-tasks.creole @@ -0,0 +1,116 @@ +== Admin tasks == + +For reference (and in case I get run over or something) here's a grab +bag of marmalade tasks. + + +=== To close down node.js/mongo marmalade-repo === + +We will need to: + +* shut off the node.js app so people can't upload +* do a full dump of the db +* run mongo-covert to export +* rsync the files to the new host +* turn on new marmalade uploader + +=== Syncing the mongodb dump from Marmalade v1 === + +Here's the cron I run to keep a local copy of marmalade: + +{{{ +rsync --bwlimit=20 -av \ + --files-from=<(ssh $MARMALADE_HOST find seville-marmalade/mongodump -cmin -60) \ + -e ssh $MARMALADE_HOST:. $DESTINATION_DIR +}}} + +The dump is pretty big so I'm not getting the latest of +everything. Just the last hour and I run it twice a day. + +=== Importing a mongodb dump from Marmalade v1 === + +Marmalade v1 dumps it's mongodb to a file every hour. I pull those +files to my house and then use them to setup a local copy of the +marmalade database that I use to generate the marmalade v2 filesystem. + +How to restore the database: + +{{{ +mongorestore $DIRECTORY +}}} + +the {{{$DIRECTORY}}} should be an individual full-dump directory. + +I have a directory full of dumps so this is how I actually do it: + +{{{ +mongorestore ~/seville-marmalade/mongodump/$(ls -t ~/seville-marmalade/mongodump/ | head -n2 | tail -n1) +}}} + +that chooses the last but one dump all the time. + + +=== Turning the mongodb into a file system === + +A simple bash script uses {{{mongofiles}}} to export the mongo +database into a marmalade file system: + +{{{ +bash mongo-convert +}}} + +This uses {{{~/marmalade/packages}}} as the package root, by default. + + +=== Sync the extracted file system === + +{{{mongofiles}}} doesn't update files, it just splats over them so you +need a smarter {{{rsync}}} to move them to another host: + +{{{ +rsync -catve ssh "$MARMALADE_HOST:marmalade/packages/*" marmalade/packages +}}} + +//It would be really useful// to understand the incomming packages on +that rsync and to call the internal archive-contents update urls. Some +sort of grep on the rsync output could probably achieve that. + +This would mean new marmalade could be a mirror of old marmalade for a +while, before we switch over. + +=== Deploy marmalade === + +A deploy script is included in the repository. + +This is the sequence of things you need to do to deploy: + +{{{ +$ emacs-local/bin/emacsclient -s /tmp/emacs1000/marmalade -e '(kill-emacs)' +$ rm -rf marmalade-repo +$ emacs-local/bin/emacs --script deploy.el marmalade-repo +$ HOME=$(realpath marmalade-repo) emacs-local/bin/emacs --daemon=marmalade +}}} + +The steps are: + +1. kill any existing daemon + +2. remove the existing deployment (//not// the data) + +3. deploy the package and it's dependancies + +4. start the daemon + +Once the Emacs instance is started you can attach to it like this: + +{{{ +$ emacs-local/bin/emacsclient -s /tmp/emacs1000/marmalade $(realpath marmalade-repo) +}}} + +The argument {{{$(realpath marmalade-repo}}} only ensures we open some +directory. Any directory or file could be given, as ever with +{{{emacsclient}}}. + +//Note on the package archives file: currently the archive file has to +be generated manually and it's rather slow. Use M-x +marmalade/package-archives to generate it.// diff --git a/register-comingsoon.html b/register-comingsoon.html new file mode 100644 index 0000000..05e50ee --- /dev/null +++ b/register-comingsoon.html @@ -0,0 +1,39 @@ + + + + + +marmalade-repo - for all your EmacsLisp needs + + +
+
+ +
+
+
+
+
+

sorry... you can't register for marmalade-repo ...

+

We just haven't got a working registration yet. But it's coming very soon.

+

In the mean time, if you need an account, please add an issue on the + github.

+
+
+
+ + +