Two more fetchers `targz` and `raw` #57

wants to merge 11 commits into
@@ -77,15 +77,15 @@ Packages are specified by files in the `recipes` directory. You can
contribute a new package by adding a new file under `recipes` using
the following form,
- :fetcher [git|github|bzr|hg|darcs|svn|wiki]
+ :fetcher [git|github|bzr|hg|darcs|svn|wiki|targz|raw]
[:url "<repo url>"]
[:repo "github-user/repo-name"]
[:files ("<file1>", ...)])
: a lisp symbol that has the same name as the package being specified.
@@ -108,6 +108,9 @@ the package name. Note that the `:url` property is not needed for the
differs from the package name being built. In the case of the `github`
fetcher, use `:repo` instead of `:url`; the git URL will then be
+The `targz` fetcher can be used to build packages from a
+downloaded tarball `:url`.
+The `raw` fetcher expects a `:url` of an emacs lisp file.
: optional property specifying the explicit files used to build the
@@ -116,6 +119,9 @@ root of the repository. This is necessary when there are multiple
`.el` files in the repository but the package should only be built
from a subset.
+: optional property specifying the package version to report as present on this elisp archive. Every fetcher type determines automatically this version number from the date-of-retrieval depending on the method, however in some cases you would like this melpa archive to report it has the always-newest-version of a package, this could be handy when you have a custom melpa archive and want some elisp packages to always install from it, even if other archives have more really-up-to-date versions. A way to acomplish this would be to use a bigger natural-order version on your custom melpa archive, that way that particular package will always be taken from your custom melpa archive.
@@ -124,6 +130,100 @@ from a subset.
+### Recipe Examples
+#### wiki
+To create a package for an elisp maintained on the emacs wiki, for example
+if you wanted to create a recipe for [wn-mode.el]( you just need to create a recipe that looks like this:
+(wn-mode :fetcher wiki)
+Other packages can consist of many files like [Icicles](
+(icicles :fetcher wiki :files
+ ("icicles.el" "icicles-chg.el" "icicles-cmd1.el" "icicles-cmd2.el" "icicles-doc1.el" "icicles-doc2.el" "icicles-face.el" "icicles-fn.el" "icicles-mac.el" "icicles-mcmd.el" "icicles-mode.el" "icicles-opt.el" "icicles-var.el"))
+#### git
+Most recently many elisp libraries are being developed using git, to create a recipe for one of them, say [Evil]( you do:
+(evil :url "git://" :fetcher git)
+An optional `:commit` keyword can be used to specify the commit to checkout.
+#### github
+This fetcher is a shortcut for packages fetched from github repos.
+(helm :repo "emacs-helm/helm" :fetcher github)
+#### targz
+Retrieving a tarball from a github download or gist tar.
+The difference between using a `github` or `targz` fetcher
+is that the later just downloads a tarball and doesnt keep a local
+clone of the github repo. The `:url` can point to any
+http-exposed tarball not only github downloads and gist tars.
+Note that because you're using github's tarball urls, you can
+replace `master` with any tag or commit number.
+ :fetcher targz
+ :url "")
+#### raw
+The `raw` fetcher can be used to obtain elisp libraries from http-exposed
+plain files.
+For example, say you found [Edward O'Connor elisp files]( and wanted to create a package for his `color-theme-hober2`
+ :fetcher raw
+ :url "")
+Or you want to create a package for his OS-X hacks
+ :fetcher raw
+ :url ""
+ :files ("growl.el" "osx-plist.el"))
+Another use case for the `raw` is where you want to create a package from a github hosted file without having to clone the entire repository, take
+for example the [Fancy language](, the <code>fancy-mode</code> is maintained as part of fancy's repo, but having to
+clone the whole repo just to obtain the elisp would be a bit too much, for this case you can take advantage of the fact that github allows you to retrieve
+a raw file content for any branch, tag or commit and create a package like this to have an always updated-with-master fancy-mode:
+ :fetcher raw
+ :url "")
+Note that the `:url` keyword can take any kind of URLs that
+can be fetched by emacs, so you can use <code>http://</code>, <code>ftp://</code>, <code>file://</code>, etc.
+For the `raw` fetcher the `:url` keyword can also be a list of different source URLs:
+ :fetcher raw
+ :url ("file:///my/secrets.el" "ftp://my.corp/dev/env.el"))
### Single File Repository
@@ -134,7 +234,7 @@ from a subset.
Since there is only one `.el` file, this package only needs the `:url` and `:fetcher` specified,
:url ""
:fetcher git)
@@ -147,7 +247,7 @@ The
contains the *starter-kit* package along with extra packages in the
`modules` directory; *starter-kit-bindings*, *starter-kit-lisp*, etc.
:url ""
:fetcher git)
@@ -95,9 +95,13 @@ In turn, this function uses the :fetcher option in the config to
choose a source-specific fetcher function, which it calls with
the same arguments."
(let ((repo-type (plist-get config :fetcher)))
+ (when (get-buffer "*package-build-checkout*")
+ (kill-buffer "*package-build-checkout*"))
(print repo-type)
- (funcall (intern (format "pb/checkout-%s" repo-type))
- name config cwd)))
+ (let ((checkedout-version
+ (funcall (intern (format "pb/checkout-%s" repo-type))
+ name config cwd)))
+ (or (plist-get config :version) checkedout-version))))
(defvar pb/last-wiki-fetch-time 0
"The time at which an emacswiki URL was last requested.
@@ -143,6 +147,36 @@ seconds; the server cuts off after 10 requests in 20 seconds.")
(default-directory dir))
(car (nreverse (sort (mapcar 'pb/grab-wiki-file files) 'string-lessp))))))
+(defun pb/grab-raw (elisp-url)
+ "Retrieve URL and save it under `default-directory`"
+ (with-current-buffer (url-retrieve-synchronously elisp-url)
+ (setq buffer-file-name
+ (expand-file-name (file-name-nondirectory elisp-url) default-directory))
+ (let ((version-control 'never))
+ (save-buffer)
+ (condition-case nil
+ (aref (package-buffer-info) 3) ;; package version
+ (error "0")))))
+(defun pb/checkout-raw (name config dir)
+ "Checkout package NAME with config CONFIG by downloading each raw file into DIR"
+ (with-current-buffer (get-buffer-create "*package-build-checkout*")
+ (message dir)
+ (unless (file-exists-p dir)
+ (make-directory dir))
+ (let ((download-url (plist-get config :url))
+ (files (plist-get config :files))
+ (default-directory dir)
+ urls)
+ (cond
+ ((and download-url (not files))
+ (setq urls (list download-url)))
+ ((and files (not download-url))
+ (setq urls files))
+ (t
+ (setq urls (mapcar (lambda (file) (concat download-url "/" file)) files))))
+ (car (nreverse (sort (mapcar 'pb/grab-raw urls) 'string-lessp))))))
(defun pb/darcs-repo (dir)
"Get the current darcs repo for DIR."
@@ -286,6 +320,86 @@ seconds; the server cuts off after 10 requests in 20 seconds.")
"\\([0-9]\\{4\\}-[0-9]\\{2\\}-[0-9]\\{2\\} [0-9]\\{2\\}:[0-9]\\{2\\}\\)"))))
+(defun pb/grab-tarball (&optional tarext)
+ "Retrieve and expand a tarball using TAREXT auto-compression.
+ Returns a list of formatted date for each tar entry."
+ (message download-url)
+ (require 'jka-compr)
+ (require 'tar-mode)
+ (with-auto-compression-mode
+ (let* ((temp-file (jka-compr-make-temp-name))
+ (info (and tarext (jka-compr-get-compression-info tarext)))
+ (compress-program (and info (jka-compr-info-uncompress-program info)))
+ (compress-message (and info (jka-compr-info-uncompress-message info)))
+ (compress-args (and info (jka-compr-info-uncompress-args info))))
+ (if (not info)
+ (url-retrieve-synchronously download-url)
+ (url-copy-file download-url temp-file t)
+ (jka-compr-call-process compress-program
+ compress-message
+ temp-file
+ (current-buffer)
+ nil
+ compress-args)
+ (delete-file temp-file))
+ (tar-mode)
+ (pb/untar-buffer (plist-get config :files-under)))))
+(defun pb/untar-buffer (&optional files-under)
+ "Extract all archive members in the tar-file into the current directory.
+ If the tar seems like a git generated package, only extract the contents
+ of the first folder and take the the git commit as the package version.
+ This function derived from tar-untar-buffer from emacs' tar-mode.el
+ and is licenced under GPL"
+ (interactive)
+ ;; FIXME: make it work even if we're not in tar-mode.
+ (let ((descriptors tar-parse-info))
+ ; Read the var in its buffer.
+ (with-current-buffer
+ (if (tar-data-swapped-p) tar-data-buffer (current-buffer))
+ (set-buffer-multibyte nil) ;Hopefully, a no-op.
+ ;; A git package
+ (when (string= "pax_global_header" (tar-header-name (car descriptors)))
+ ;; extract files from the first directory.
+ (setq files-under (tar-header-name (cadr descriptors))))
+ (mapcar (lambda (descriptor)
+ (let* ((name (tar-header-name descriptor))
+ (dir (if (eq (tar-header-link-type descriptor) 5)
+ name
+ (file-name-directory name)))
+ (start (tar-header-data-start descriptor))
+ (end (+ start (tar-header-size descriptor))))
+ (when (and (not (file-directory-p name))
+ (or (not files-under)
+ (and (not (string-equal files-under name))
+ (string-prefix-p files-under name)
+ (setq name (substring name (length files-under))
+ dir (substring dir (length files-under))))))
+ (message "Extracting %s" name)
+ (if (and dir (not (file-exists-p dir)))
+ (make-directory dir t))
+ (let ((coding-system-for-write 'no-conversion))
+ (write-region start end name))
+ (set-file-modes name (tar-header-mode descriptor))))
+ (format-time-string "%Y%m%d" (tar-header-date descriptor)))
+ descriptors))))
+(defun pb/checkout-targz (name config dir)
+ "Checkout package NAME with config CONFIG by downloading and extracting a tar.gz url into DIR"
+ (with-current-buffer (get-buffer-create "*package-build-checkout*")
+ (message dir)
+ (unless (file-exists-p dir)
+ (make-directory dir))
+ (let ((download-url (plist-get config :url))
+ (default-directory dir))
+ (car (nreverse (sort (pb/grab-tarball "tar.gz") 'string-lessp))))))
(defun pb/dump (data file)
"Write DATA to FILE as a pretty-printed Lisp sexp."
(write-region (concat (pp-to-string data) "\n") nil file))