There is no good way to install and upgrade use-package

[marcinant]

Problem is that use-package is available as ELPA package and this is probably easiest way to install and upgrade.
Unfortunately use-package should be initialized before Emacs initializes ELPA packages.

So, when this macro is installed as ELPA package there is no way to (require 'use-package) because Emacs doesn't know where use-package is.

The only way I see is to add full path to bind-key and use-package as second argument to require.
It works however manual update is required every time ELPA package is upgraded.

Any other ideas to solve this problem?

[thomasf]

I'm not sure that I follow.. When use-package is installed, package.el will find it and upgrade it after invoking list-packages

I have additionaly set up so that I don't need to package-initialize packages by saving the load path in a byte compiled load-path.el.

This is from the top of my init.el

(when load-file-name
  (load (expand-file-name
         "load-path" (file-name-directory load-file-name)) nil t))

(setq
 package-enable-at-startup nil
 package-archives
 '(("melpa-stable" . "http://stable.melpa.org/packages/")
   ("melpa" . "http://melpa.org/packages/")
   ("marmalade"   . "http://marmalade-repo.org/packages/")
   ("org"         . "http://orgmode.org/elpa/")
   ("gnu"         . "http://elpa.gnu.org/packages/")
   ("sc"   . "http://joseito.republika.pl/sunrise-commander/")))

(eval-when-compile
  (require 'package)
  (package-initialize t))
(unless (boundp 'package-pinned-packages)
  (setq package-pinned-packages ()))

(defun require-package (package &optional min-version no-refresh)
  "Install given PACKAGE, optionally requiring MIN-VERSION.
If NO-REFRESH is non-nil, the available package lists will not be
re-downloaded in order to locate PACKAGE."
  (if (package-installed-p package min-version)
      t
    (if (or (assoc package package-archive-contents) no-refresh)
        (package-install package)
      (progn
        (package-refresh-contents)
        (require-package package min-version t)))))

(eval-when-compile
  (require-package 'use-package)
  (require 'use-package))

This is the load-path.el located besides my init.el

In addition to byte compiling the effecive load-path load-path.el will recompile itself if one of the compiled paths are missing at start up which takes care of the most usual case when packages are upgraded and obsolete ones removed.

(I have not cared a great deal about style in this particular file...)

;;; load-path.el

(defconst user-emacs-directory
  (if (eq system-type 'ms-dos)
      ;; MS-DOS cannot have initial dot.
      "~/_emacs.d/"
    "~/.emacs.d/")
  "Directory beneath which additional per-user Emacs-specific files are placed.
Various programs in Emacs store information in this directory.
Note that this should end with a directory separator.
See also `locate-user-emacs-file'.")

(defconst user-data-directory
  (file-truename "~/.config/emacs-user-data"))
(defconst user-cache-directory
  (file-truename "~/.cache/emacs-user-cache"))
(defconst user-lisp-directory
  (expand-file-name "lisp" user-emacs-directory))

(defun load-path--take (n list)
  "Returns a new list of the first N items in LIST, or all items if there are fewer than N.
This is just a copy of the fully expanded macro from dash."
  (let (result)
    (let
        ((num n)
         (it 0))
      (while
          (< it num)
        (when list
          (setq result
                (cons
                 (car list)
                 result))
          (setq list
                (cdr list)))
        (setq it
              (1+ it))))
    (nreverse result)))


(defconst user-site-lisp-directory
  (expand-file-name "site-lisp/shared" user-emacs-directory))
(defconst user-themes-directory
  (expand-file-name "themes" user-emacs-directory))
(defconst user-notes-directory
  (file-truename "~/notes"))

;; These should always exist
(make-directory user-data-directory t)
(make-directory user-cache-directory t)

;; emacs23 compat
(if (boundp 'custom-theme-load-path)
    (add-to-list 'custom-theme-load-path user-themes-directory)
  (add-to-list 'load-path user-themes-directory))

(defun add-to-load-path (path &optional dir)
  (setq load-path
        (cons (expand-file-name path (or dir user-emacs-directory)) load-path)))

(defun load-path-load-path ()
  (let ((load-path load-path))
    ;; Add top-level lisp directories, in case they were not setup by the
    ;; environment.
    (require 'package)
    (package-initialize)
    (dolist (dir (nreverse
                  (list user-lisp-directory
                        user-site-lisp-directory)))
      (dolist (entry (nreverse (directory-files-and-attributes dir)))
        (and
         (cadr entry)
         (not (string= (car entry) ".."))
         (add-to-load-path (car entry) dir))))


    (mapc #'add-to-load-path
          (nreverse
           (list
            (expand-file-name "~/.config-private/emacs")
            (expand-file-name "~/.opt/extempore/extras")
            (expand-file-name "/usr/local/opt/extempore/extras")
            (concat user-site-lisp-directory "/emms/lisp")
            "/usr/local/share/emacs/site-lisp/"
            "/usr/local/share/emacs/site-lisp/mu4e/"
            "/opt/local/share/emacs/site-lisp/"
            "/usr/share/emacs/site-lisp/SuperCollider/"
            "/usr/share/emacs/site-lisp/supercollider/"
            "/var/lib/gems/1.9.1/gems/trogdoro-el4r-1.0.10/data/emacs/site-lisp/")))

    (let ((cl-p load-path))
      (while cl-p
        (setcar cl-p (file-name-as-directory
                      (expand-file-name (car cl-p))))
        (setq cl-p (cdr cl-p))))

    (when
        (or (not (boundp 'emacs-version))
           (string< emacs-version "24.3"))
      (add-to-load-path
       (expand-file-name "site-lisp/cl-lib" user-emacs-directory)))


    (delete-dups
     (delq nil (mapcar #'(lambda (x)
                         (if (file-directory-p x)
                             x
                           nil))
                     load-path)))))


(defmacro load-path-set-load-path ()
  `(progn
     (setq load-path ',(load-path-load-path))
     (let ((failed nil))
       (mapc #'(lambda (x)
                 (unless failed
                   (setq failed (not (file-directory-p x)))))
             load-path)
       (when failed
         (require 'bytecomp)
         (let ((byte-compile-verbose nil)
               (byte-compile-warnings nil)
               (use-package-verbose nil)
               (ad-redefinition-action 'accept))
           (byte-recompile-file "~/.emacs.d/load-path.el" t 0 t))))))

(load-path-set-load-path)

(eval-after-load "info"
  #'(progn
      (when (fboundp 'info-initialize)
        (info-initialize)
        (defun add-to-info-path (path &optional dir)
          (setq Info-directory-list
                (cons (expand-file-name path (or dir user-emacs-directory)) Info-directory-list)))
        (mapc #'add-to-info-path
              (nreverse
               (list
                (expand-file-name "~/.refdoc/info")))))))

(when (bound-and-true-p x-bitmap-file-path)
  (add-to-list 'x-bitmap-file-path
               (concat user-emacs-directory "/icons")))

(require 'cus-load nil t)

(provide 'load-path)

;;; load-path.el ends here

[xuchunyang]

Unfortunately use-package should be initialized before Emacs initializes ELPA packages.

Why? I think there is no need to initialize use-package, of course you have to install use-package before using it, you can install use-package from ELPA like this way:

;; Setup package.el
(require 'package)
(setq package-enable-at-startup nil)
(add-to-list 'package-archives '("melpa" . "http://melpa.org/packages/"))
(package-initialize)

;; Bootstrap `use-package'
(unless (package-installed-p 'use-package)
  (package-refresh-contents)
  (package-install 'use-package))

;; Use use-package now

[marcinant]

Running package-initialize on top of emacs is plain stupid.

You are right. If I add package-initialize before I try to require use-package then it's going to run properly.

However and idea behind package-initialize is to run this command AFTER .emacs or init.el and run all configuration on after-init-hook.

[thomasf]

@marcinant My solution only requires/uses package.el inside eval-when-compile in init.el or when load-path.el is recompiled..

[thomasf]

My emacs starts in about 1-1.1seconds on my 2009 i7 desktop computer (provided files are loaded into disk cache)..
Thats with 778 package dirs in elpa/ and a ~10kloc init.el with 518 (use-package forms and 389 uses of :ensure. (some of the use-package instances are probably :disabled but not many)

IIRC, the package-initialize otherwise needed to set up load paths of packages and their dependencies adds 0.3-0.5s to the init time which becomes way more noticable than closer to 1s.

[thomasf]

I just also remembered my load-path.elonly recompiles itself if it already is byte compiled, It also does some non load path related things and I have not taken care of other variables like the theme path..
So my solarized-theme setup looks like this

(use-package solarized-theme
  :ensure t
  :if window-system
  :init
  (progn
    (setq solarized-use-less-bold t
          solarized-use-more-italic t
          solarized-emphasize-indicators nil
          solarized-distinct-fringe-background nil
          solarized-high-contrast-mode-line nil))
  :config
  (progn
    (load "solarized-theme-autoloads" nil t)
    (setq theme-dark 'solarized-dark
          theme-bright 'solarized-light)))

[npostavs]

Running package-initialize on top of emacs is plain stupid.

?

Artur Malabarba wrote:

On the "Customizable modes..." thread I suggested we run
(package-initialize) sooner than the way it's currently done. Right
now, it's called after loading the init file. Which means any user who
tries to customize an installed package by pasting some code into his
init file will be confronted with errors.
This happens A LOT.

Stefan kindly explains why it can't just be done before loading the init file:

[...] the user may need/want to run some Elisp
code of his own choosing before package-initialize is called.
E.g. [...] set package-load-list and package-directory-list

But I'm asking that we try a little harder to find a better solution.
This package.el-induced "cannot find load file" error is the most
predominant issue I see people run into in the wild. Some people file
issues for this stuff on github (and waste developer time), [...]

Option 2) Instead of us manually telling users to add
`(package-initialize)' to their init-files, we have Emacs do that
automatically. [...]

[marcinant]

Running package-initialize on top of emacs is plain stupid.

?

Just because startup.el contains this code:

       (package-initialize))

  (setq after-init-time (current-time))
  (run-hooks 'after-init-hook)

So, package-initialize is part of normal startup procedure and there is no reason to disturb this order and run package-initialize twice.

Artur Malabarba wrote:

On the "Customizable modes..." thread I suggested we run
(package-initialize) sooner than the way it's currently done. Right
now, it's called after loading the init file. Which means any user who
tries to customize an installed package by pasting some code into his
init file will be confronted with errors.
This happens A LOT.

It's their problem. They should use after-init-hook and that's it.

Some people file
issues for this stuff on github (and waste developer time), [...]

Developer should just provide documentation how to configure package on after-init-hook.

[thomasf]

@marcinant it runs twice unless you set (setq package-enable-at-startup nil)

[thomasf]

I used strace to guide my decision of removing package-initialize on normal emacs startup.. The combination of emacs synchronized io and lot's of file descriptor calls just makes it slow.

Since use-pakcage :commands generates autoloads there is little package.el does at start up except set the load path.

[marcinant]

Hmm... is your .emacs available somewhere?

Anyway I decided to use this dirty piece of code:

(when package-enable-at-startup
  (eval-when-compile
    (let ((default-directory package-user-dir))
      (require 'bind-key (car (file-expand-wildcards "bind-key*/bind-key.elc" t)))
      (require 'use-package (car (file-expand-wildcards "use-package*/use-package.elc" t))))))

Now I need to switch my .emacs to use package and test it.
I don't care about load time that much as I use emacsclient and run emacs on startup just once.

[thomasf]

yeah https://github.com/thomasf/dotfiles-thomasf-emacs I have one emacs per "workspace group" of desktops which more or less means one emacs per project.. I also have a default "home" emacs for all general emacsclient needs. Startup time is more important when working like that.

[marcinant]

After further code analysis I have to admit that I don't understand the idea behind use-package.

Package.el is intended to load after user init file.
Unfortunately use-package doesn't want to work while packages are not initialized.

IMHO this is just plain wrong. Could someone (an author) enlighten me and tell why is it designed in this way?

[thomasf]

It's not designed specifically for package.el.. John (the author) doesnt even use package.el. It's more or less up to the use-package user to do the integration part.

[marcinant]

Ok. Now I can see what the problem was. All my packages are not in load-path until package.el is initialized. Sigh....

[marcinant]

Well, you may close this bug. However problem is still alive.

Although use-package is available on ELPA it should not be installed as package.

[thomasf]

I disagree that this is an use-package issue, after all it's elisp were talking about and it's as easy as invoking (package-initialize) when it's needed.. Also the emacs-devel discussion link posted by npostavs in this issue suggests that the default behavior might actually change into doing it before user init by default.
Having said that, the documentation could probably be updated to explain this situation.

[marcinant]

It's undocumented. It should be mentioned in readme that one has to invoke (package-initialize) or provide load-path manually.

When someone will install this macro as ELPA package and initialize in the way described in readme then it's not going to work.

[thomasf]

btw. emacs-25, ie. the master branch now does this to the top of the user init.el

;; Added by Package.el.  This must come before configurations of
;; installed packages.  Don't delete this line.  If you don't want it,
;; just comment it out by adding a semicolon to the start of the line.
;; You may delete these explanatory comments.
 (package-initialize)

[marcinant]

Well it's just wierd. Default policy is: use init.el to set up config with after-init-hooks, then 'package-initialize' is issued automatically by 'startup.el' right after 'init.el' is loaded.

There is no good reason to run 'package-initialize' from 'init.el'.

[thomasf]

Weird or not, the default policy seems to be changing.

I find it a bit less weird with (package-initialize) at the top than having to wrap most of init.el inside hook functions when the only reason is to wait for package.el. The new policy will probably be a lot less confusing for many Emacs newbies as well.

[errge]

I made a pull request to implement something similar to thomasf's setup, but a simpler way: #487

[raxod502]

Default policy is: use init.el to set up config with after-init-hooks, then 'package-initialize' is issued automatically by 'startup.el' right after 'init.el' is loaded.

Sorry, but I disagree. There is simply no practical disadvantage to running package-initialize in the init-file, and it makes things so much simpler. Putting your entire configuration in after-init-hook? That's a needless layer of indirection.

In any case, Emacs now initializes the package system before loading the init file, thus rendering this entire debate superfluous. The feature will be released in Emacs 27.

[jwiegley]

In any case, Emacs now initializes the package system before loading the init file, thus rendering this entire debate superfluous. The feature will be released in Emacs 27.

As long as this can be disabled, right?

[raxod502]

this can be disabled, right?

In ~/.emacs.d/early-init.el, place:

(setq package-enable-at-startup nil)