add new html page and re-org the doc

README.creole

@@ -22,253 +22,24 @@ These are the things marmalade should do:
 * provide HTML front end to the {{{/package/}}} HTTP API
 * provide EmacsLisp front end to the {{{/package/}}} HTTP API
 ** this will likely be a separate project
-* provide an HTML front end to the package archives with some discovery
 
-== Rollout ==
+==== ETA ====
 
-Some notes on rollout. Marmalade is going to need more sophistication
-than I've ever managed to build with elnode. It's basically a
-collection of microservices, which is always what has been attractive
-about elnode... it's just I've never managed to put it all together.
+Current expected ETA for live elmarmalade: end of May.
 
-Marmalade will require multiple servers because the archive stuff is
-very expensive, because we're using emacs-db for now and that requires
-concentrating access into a single place.
+node.js marmalade has been broken for some time with the web
+upload. This is caused by major outdatedness in the node.js that
+marmalade was built on. It is very crufty and tedious to update.
 
+So Nic Ferrier is banking on getting this live. Sadly he has a
+demanding job at the moment so doesn't get the time to hack on this
+that he'd like.
 
-* install the package on the server
-** it needs to know where the package archive is
 
-{{{
-(customize-set-variable
- 'marmalade-package-store-dir
- (file-name-as-directory 
-   (expand-file-name "~/marmalade/packages")))
-}}}
+==== Other marmalade things ====
 
-* have a daemon file for the package
-* nginx script
-** which depends on your port number
-* ssl? optional
-* we need 2 emacs instances
-** 1 to serve archive
-** 1 to serve everything else
+It would be nice if marmalade provided better package discovery.
 
-== Current problems ==
+It would be nice if marmalade stored licences in a clear way and
+allowed you to use only safe licences if you wish.
 
-=== Handling errors better ===
-
-Here
-is
-[[https://github.com/nicferrier/elmarmalade/blob/master/marmalade-service.el#L150|the upload function]]
-
-{{{
-(let ((package-file-name
-       (marmalade/save-package upload-file base-file-name)))
-  (elnode-send-redirect httpcon package-file-name 201))
-}}}
-
-which calls
-this
-[[https://github.com/nicferrier/elmarmalade/blob/master/marmalade-service.el#L121|the save package function]]
-
-{{{
-(with-temp-file temp-package-file
-  (insert-string
-   (substring-no-properties package-data)))
-;; Now get the real path
-(marmalade/package-path temp-package-file))))
-}}}
-
-which makes a temp file and puts the package data in it and then
-uses
-[[https://github.com/nicferrier/elmarmalade/blob/master/marmalade-service.el#L121|this]]
-
-{{{
-(defun marmalade/package-path (package-file)
-  "Turn PACKAGE-FILE into a repository package path."
-  (let* ((info (marmalade/package-info package-file))
-         (version (elt info 3))
-}}}
-
-which
-uses
-[[https://github.com/nicferrier/elmarmalade/blob/master/marmalade-service.el#L67|this]]
-to get the meta data of the existing (temp) package file.
-
-
-{{{
-(defun marmalade/package-info (package-file)
-  "Return the package-info on the PACKAGE-FILE.
-
-PACKAGE-FILE is either an ELISP or a TAR file to be uploaded to
-the package repository."
-  (cond
-    ((string-match "\\.el$" package-file)
-     (with-temp-buffer
-       (insert-file-contents-literally package-file)
-       (buffer-string) ; leave it in so it's easy to debug
-       (package-buffer-info)))
-    ((string-match "\\.tar$" package-file)
-     (package-tar-file-info package-file))
-    (t (error "Unrecognized extension `%s'"
-              (file-name-extension package-file)))))
-}}}
-
-And that, I think, is where a lot of error checking needs to be done
-and reported from.
-
-Currently we
-have
-[[https://github.com/nicferrier/elmarmalade/blob/master/marmalade-tests.el#L200|this]]
-which tests the package save:
-
-
-{{{
-(ert-deftest marmalade/save-package ()
-  "Test the save package stuff."
-  (let ((marmalade-package-store-dir "/tmp/test-marmalade-dir"))
-    (marmalade/package-file
-     :code
-     (let ((expected
-            "/tmp/test-marmalade-dir/dummy-package/0.0.1/dummy-package-0.0.1.el"))
-       (should
-        (equal
-         (marmalade/save-package
-          package-content-string
-          "dummy-package.el")
-         expected))
-       (should
-        (equal
-         (fakir-file-path package-file)
-         expected))))))
-}}}
-
-It doesn't test much about it though. We need a lot of invalid
-packages to be tested.
-
-=== Eventually consistent ===
-
-The archive is maintained separately from the filesystem. It is
-possible that a delete is performed on the filesystem but the archive
-that a user has just downloaded still shows the package. In this case
-the actual package retrieval will fail and the user will, presumably,
-refresh the package archive and the package will be shown removed.
-
-
-== Admin tasks ==
-
-For reference (and in case I get run over or something) here's a grab
-bag of marmalade tasks.
-
-
-=== To close down node.js/mongo marmalade-repo ===
-
-We will need to:
-
-* shut off the node.js app so people can't upload
-* do a full dump of the db
-* run mongo-covert to export
-* rsync the files to the new host
-* turn on new marmalade uploader
-
-=== Syncing the mongodb dump from Marmalade v1 ===
-
-Here's the cron I run to keep a local copy of marmalade:
-
-{{{
-rsync --bwlimit=20 -av \
-      --files-from=<(ssh $MARMALADE_HOST find seville-marmalade/mongodump -cmin -60) \
-      -e ssh $MARMALADE_HOST:. $DESTINATION_DIR
-}}}
-
-The dump is pretty big so I'm not getting the latest of
-everything. Just the last hour and I run it twice a day.
-
-=== Importing a mongodb dump from Marmalade v1 ===
-
-Marmalade v1 dumps it's mongodb to a file every hour. I pull those
-files to my house and then use them to setup a local copy of the
-marmalade database that I use to generate the marmalade v2 filesystem.
-
-How to restore the database:
-
-{{{
-mongorestore $DIRECTORY
-}}}
-
-the {{{$DIRECTORY}}} should be an individual full-dump directory.
-
-I have a directory full of dumps so this is how I actually do it:
-
-{{{
-mongorestore ~/seville-marmalade/mongodump/$(ls -t ~/seville-marmalade/mongodump/ | head -n2 | tail -n1)
-}}}
-
-that chooses the last but one dump all the time.
-
-
-=== Turning the mongodb into a file system ===
-
-A simple bash script uses {{{mongofiles}}} to export the mongo
-database into a marmalade file system:
-
-{{{
-bash mongo-convert
-}}}
-
-This uses {{{~/marmalade/packages}}} as the package root, by default.
-
-
-=== Sync the extracted file system ===
-
-{{{mongofiles}}} doesn't update files, it just splats over them so you
-need a smarter {{{rsync}}} to move them to another host:
-
-{{{
-rsync -catve ssh "$MARMALADE_HOST:marmalade/packages/*" marmalade/packages
-}}}
-
-//It would be really useful// to understand the incomming packages on
-that rsync and to call the internal archive-contents update urls. Some
-sort of grep on the rsync output could probably achieve that.
-
-This would mean new marmalade could be a mirror of old marmalade for a
-while, before we switch over.
-
-=== Deploy marmalade ===
-
-A deploy script is included in the repository.
-
-This is the sequence of things you need to do to deploy:
-
-{{{
-$ emacs-local/bin/emacsclient -s /tmp/emacs1000/marmalade -e '(kill-emacs)'
-$ rm -rf marmalade-repo
-$ emacs-local/bin/emacs --script deploy.el marmalade-repo
-$ HOME=$(realpath marmalade-repo) emacs-local/bin/emacs --daemon=marmalade
-}}}
-
-The steps are:
-
-1. kill any existing daemon
-
-2. remove the existing deployment (//not// the data)
-
-3. deploy the package and it's dependancies
-
-4. start the daemon
-
-Once the Emacs instance is started you can attach to it like this:
-
-{{{
-$ emacs-local/bin/emacsclient -s /tmp/emacs1000/marmalade $(realpath marmalade-repo)
-}}}
-
-The argument {{{$(realpath marmalade-repo}}} only ensures we open some
-directory. Any directory or file could be given, as ever with
-{{{emacsclient}}}.
-
-//Note on the package archives file: currently the archive file has to
-be generated manually and it's rather slow. Use M-x
-marmalade/package-archives to generate it.//

admin-tasks.creole

@@ -0,0 +1,116 @@
+== Admin tasks ==
+
+For reference (and in case I get run over or something) here's a grab
+bag of marmalade tasks.
+
+
+=== To close down node.js/mongo marmalade-repo ===
+
+We will need to:
+
+* shut off the node.js app so people can't upload
+* do a full dump of the db
+* run mongo-covert to export
+* rsync the files to the new host
+* turn on new marmalade uploader
+
+=== Syncing the mongodb dump from Marmalade v1 ===
+
+Here's the cron I run to keep a local copy of marmalade:
+
+{{{
+rsync --bwlimit=20 -av \
+      --files-from=<(ssh $MARMALADE_HOST find seville-marmalade/mongodump -cmin -60) \
+      -e ssh $MARMALADE_HOST:. $DESTINATION_DIR
+}}}
+
+The dump is pretty big so I'm not getting the latest of
+everything. Just the last hour and I run it twice a day.
+
+=== Importing a mongodb dump from Marmalade v1 ===
+
+Marmalade v1 dumps it's mongodb to a file every hour. I pull those
+files to my house and then use them to setup a local copy of the
+marmalade database that I use to generate the marmalade v2 filesystem.
+
+How to restore the database:
+
+{{{
+mongorestore $DIRECTORY
+}}}
+
+the {{{$DIRECTORY}}} should be an individual full-dump directory.
+
+I have a directory full of dumps so this is how I actually do it:
+
+{{{
+mongorestore ~/seville-marmalade/mongodump/$(ls -t ~/seville-marmalade/mongodump/ | head -n2 | tail -n1)
+}}}
+
+that chooses the last but one dump all the time.
+
+
+=== Turning the mongodb into a file system ===
+
+A simple bash script uses {{{mongofiles}}} to export the mongo
+database into a marmalade file system:
+
+{{{
+bash mongo-convert
+}}}
+
+This uses {{{~/marmalade/packages}}} as the package root, by default.
+
+
+=== Sync the extracted file system ===
+
+{{{mongofiles}}} doesn't update files, it just splats over them so you
+need a smarter {{{rsync}}} to move them to another host:
+
+{{{
+rsync -catve ssh "$MARMALADE_HOST:marmalade/packages/*" marmalade/packages
+}}}
+
+//It would be really useful// to understand the incomming packages on
+that rsync and to call the internal archive-contents update urls. Some
+sort of grep on the rsync output could probably achieve that.
+
+This would mean new marmalade could be a mirror of old marmalade for a
+while, before we switch over.
+
+=== Deploy marmalade ===
+
+A deploy script is included in the repository.
+
+This is the sequence of things you need to do to deploy:
+
+{{{
+$ emacs-local/bin/emacsclient -s /tmp/emacs1000/marmalade -e '(kill-emacs)'
+$ rm -rf marmalade-repo
+$ emacs-local/bin/emacs --script deploy.el marmalade-repo
+$ HOME=$(realpath marmalade-repo) emacs-local/bin/emacs --daemon=marmalade
+}}}
+
+The steps are:
+
+1. kill any existing daemon
+
+2. remove the existing deployment (//not// the data)
+
+3. deploy the package and it's dependancies
+
+4. start the daemon
+
+Once the Emacs instance is started you can attach to it like this:
+
+{{{
+$ emacs-local/bin/emacsclient -s /tmp/emacs1000/marmalade $(realpath marmalade-repo)
+}}}
+
+The argument {{{$(realpath marmalade-repo}}} only ensures we open some
+directory. Any directory or file could be given, as ever with
+{{{emacsclient}}}.
+
+//Note on the package archives file: currently the archive file has to
+be generated manually and it's rather slow. Use M-x
+marmalade/package-archives to generate it.//