Failed to download ‘melpa’ archive (again)

[jcburley]

This issue appears to be back again. Given this in my GNU Emacs init.el:

(require 'package)
(toggle-debug-on-error)
(add-to-list 'package-archives
             '("marmalade" . "https://marmalade-repo.org/packages/") t)
(add-to-list 'package-archives
             '("tromey" . "https://tromey.com/elpa/") t)
(let* ((no-ssl (and (memq system-type '(windows-nt ms-dos))
                    (not (gnutls-available-p))))
       (url (concat (if no-ssl "http" "https") "://melpa.org/packages/")))
  (add-to-list 'package-archives
               (cons "melpa" url) t))
(when (< emacs-major-version 24)
  ;; For important compatibility libraries like cl-lib
  (add-to-list 'package-archives '("gnu" . "http://elpa.gnu.org/packages/")))

A newly started session produces this in *Messages*:

Debug on Error enabled globally
Importing package-keyring.gpg...done
Contacting host: elpa.gnu.org:80
Contacting host: marmalade-repo.org:443 [2 times]
Contacting host: melpa.org:443
Entering debugger...

The debug window shows:

Debugger entered--Lisp error: (error "package.el is not yet initialized!")
  signal(error ("package.el is not yet initialized!"))
  error("package.el is not yet initialized!")
  package-installed-p(0blayout (20161008 607))
  package--mapc(package--add-to-compatibility-table)
  package--build-compatibility-table()
  package--update-downloads-in-progress(("melpa" . "https://melpa.org/packages/"))
  #[0 "\301\300!\207" [("melpa" . "https://melpa.org/packages/") package--update-downloads-in-progress] 2]()
  package--check-signature("https://melpa.org/packages/" "archive-contents" "\n(1 (zzz-to-char . [(20180101 619) ((emacs (24 4)) (cl-lib (0 5)) (avy (0 3 0))) [...way to much on one line here...]
  package--download-one-archive(("melpa" . "https://melpa.org/packages/") "archive-contents" nil)
  package--download-and-read-archives(nil)
  package-refresh-contents()
  (if (or package-archive-contents (file-contents-= package-stamp package-stamp-file)) nil (package-refresh-contents) (write-file-contents package-stamp package-stamp-file))
  eval-buffer(#<buffer  *load*-987823> nil "/home/craig/.unixhome/components/emacs/craig/packages.el" nil t)  ; Reading at buffer position 1822
  load-with-code-conversion("/home/craig/.unixhome/components/emacs/craig/packages.el" "/home/craig/.unixhome/components/emacs/craig/packages.el" nil nil)
  load("/home/craig/.unixhome/components/emacs/craig/packages.el")
  (if (or (not (getenv "UNIXHOME")) (< emacs-major-version 24)) nil (load (concat (getenv "UNIXHOME") "/components/emacs/craig/packages.el")))
  eval-buffer(#<buffer  *load*-854216> nil "/home/craig/.unixhome/components/emacs/init.el" nil t)  ; Reading at buffer position 3146
  load-with-code-conversion("/home/craig/.unixhome/components/emacs/init.el" "/home/craig/.unixhome/components/emacs/init.el" nil nil)
  load("/home/craig/.unixhome/components/emacs/init.el" nil nil t)
  load-file("~/.unixhome/components/emacs/init.el")
  eval-buffer(#<buffer  *load*> nil "/home/craig/.emacs.d/init.el" nil t)  ; Reading at buffer position 522
  load-with-code-conversion("/home/craig/.emacs.d/init.el" "/home/craig/.emacs.d/init.el" t t)
  load("/home/craig/.emacs.d/init" t t)
  #[0 "�\205\266�	\306=\203��\307�\310Q\202?�	\311=\204��\307�\312Q\202?�\313\307\314\315#\203*�\316\202?�\313\307\314\317#\203>�\320\321\322!D\nB�\323\202?�\316  [...same here...]
  command-line()
  normal-top-level()

And emacs-version shows:

GNU Emacs 25.3.2 (x86_64-pc-linux-gnu, GTK+ Version 3.18.9) of 2017-09-12

It also happens on this machine:

GNU Emacs 25.3.1 (x86_64-apple-darwin13.4.0, NS appkit-1265.21 Version 10.9.5 (Build 13F1911)) of 2017-09-12

[purcell]

But the error is telling you exactly what the problem is, no? You need to call package-initialize before package-refresh-contents.

[jcburley]

Hmm, that's not how I'm reading that message -- looks to me like it's just complaining package initialize has failed because melpa refresh failed.

If I do move (package-initialize) to before (package-refresh-contents) in my packages.el file, I don't get the error, but none of my packages get loaded.

To further clarify what I'm doing (and I definitely do not know what I'm doing!!), here's how my init.el starts out:

;(package-initialize)  ;; this is to avoid initializing packages too early
...
(unless (or (not (getenv "UNIXHOME"))  ; Perhaps running under sudo?
            (< emacs-major-version 24))
  (load (concat (getenv "UNIXHOME") "/components/emacs/craig/packages.el")))

And the referenced packages.el starts out:

(require 'package)
(if (equal (or (getenv "EMACS_DEBUG_ON_ERROR") "false") "true") (toggle-debug-on-error))
(add-to-list 'package-archives
             '("marmalade" . "https://marmalade-repo.org/packages/") t)
(add-to-list 'package-archives
             '("tromey" . "https://tromey.com/elpa/") t)
(let* ((no-ssl (and (memq system-type '(windows-nt ms-dos))
                    (not (gnutls-available-p))))
       (url (concat (if no-ssl "http" "https") "://melpa.org/packages/")))
  (add-to-list 'package-archives
               (cons "melpa" url) t))
(when (< emacs-major-version 24)
  ;; For important compatibility libraries like cl-lib
  (add-to-list 'package-archives '("gnu" . "http://elpa.gnu.org/packages/")))

(defvar package-stamp
  (concat
   (format-time-string "%Y-%m-%d\n")
   (prin1-to-string package-archives))
  "String containing today's date on first line followed by stringized var package-archives.")

(defvar package-stamp-file
  (concat session-specific-scratch-dir "/package-refresh.STAMP")
  "Name of file in which package stamp is written -- local to each machine.")

(defun read-file-contents (f)
  "Read contents of a file and return the result as a string."
  (with-temp-buffer
    (insert-file-contents f)
    (buffer-string)))

(defun write-file-contents (s f)
  "Write given string to a specified file."
  (write-region s nil f))

(defun file-contents-= (s f)
  "Return t if file exists and contains exactly the string, nil otherwise."
  (and (file-exists-p f)
       (string= s (read-file-contents f))))

;; Download the ELPA archive description if needed.
;; This informs Emacs about the latest versions of all packages, and
;; makes them available for download.
(unless (or package-archive-contents
            (file-contents-= package-stamp package-stamp-file))
  (package-refresh-contents)
  (write-file-contents package-stamp package-stamp-file))

(package-initialize)

($UNIXHOME is ~/.unixhome, which comes from my github repo.)

This is how I attempt to balance keeping Emacs startup time low in most cases, but refresh the package contents once per day (on each of my four development machines).

Note that if I move the uncommented (package-initialize) to before the (package-refresh-contents) block (and remove the .STAMP file), none of the packages get refreshed. So that seems to be "clearly wrong", although further research into this issue suggests there's anything but "clarity" surrounding this topic!

But for now I'm under the impression that refreshing package contents must come before package initialization, otherwise it won't really accomplish much, since the packages themselves will have already been initialized from their previous contents.

[jcburley]

Note that, while trying various incarnations of this over the past 30 minutes or so, I could have sworn melpa actually was loading successfully for me (when packages were getting refreshed). But it's failing again now.

Also note that I do seem to be following the advice at the top of this page:

https://melpa.org/#/getting-started

That doesn't mean it's correct, or that I'm following it correctly.

Also note that I've added (setq package-enable-at-startup nil) after this in my init.el, but it seems to have no effect on the situation at hand:

;;(package-initialize)
(setq package-enable-at-startup nil)  ; (New.)

[jcburley]

Looking at the diagnostic more closely, I wonder whether there's a recursive dependency of some sort:

  error("package.el is not yet initialized!")
  package-installed-p(0blayout (20161008 607))
  package--mapc(package--add-to-compatibility-table)
  package--build-compatibility-table()
  package--update-downloads-in-progress(("melpa" . "https://melpa.org/packages/"))

Seems like the invocation of package-installed-p by package--mapc fails because package.el has not finished initializing -- which it can't, since it's loading melpa...?

Not sure why this isn't happening for the other archives being refreshed (daily at least) though....

[jcburley]

Anyway, when debug-on-error isn't toggled, startup "works" except melpa fails to load, which suggests a problem specific to melpa (compared to elpa, marmalade, and tromey).

I don't know what that problem is, of course...or whether I'm the only one having it, or just the only one noticing it, or what....

[milkypostman]

Just doing a quick look to debug. It says that package.el is not initialized. Can you look in package.el and see where and why that error would occur? I’m wondering if there is a package you installed that is misbehaving. I would personally blow away my packages directories and start fresh.

…

On Sun, Oct 14, 2018 at 08:04 James Craig Burley ***@***.***> wrote: Anyway, when debug-on-error isn't toggled, startup "works" except melpa fails to load, which suggests a problem specific to melpa (compared to elpa, marmalade, and tromey). I don't know what that problem is, of course...or whether I'm the only one *having* it, or just the only one *noticing* it, or what.... — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#5755 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AACUsoCq7Q8EYG5vvOQDzoJr0dYWlCN8ks5ukzZJgaJpZM4Xa9st> .

[jcburley]

Sorry, I'm having trouble finding the source for package.el and the packages directories on my system at the moment, so will have to work on this later -- today or tomorrow. (I'm sure there are Emacs variables identifying their locations; I just don't know what they are....)

[jcburley]

(Also note that I'm having the same problem on each of my development machines, not just one; although one of them has some SSL issues that prevents it from reaching any of the https URLs, which I have yet to debug.)

[jcburley]

Looks like package.el is built-in (according to C-h P package), and there's no pointer to where the source code might live on my system (I tried find ... -name package.el on various directories without success):

package is a built-in package.

     Status: Built-In.
    Version: 1.1.0
    Summary: Simple package system for Emacs
Other versions: 1.0.1 (marmalade), 0.9 (tromey).

I also tried renaming ~/.emacs.d/elpa/archives to ~/.emacs.d/elpa/archives.old, but that didn't solve anything -- the new ~/.emacs.d/elpa/archives/melpa/archivecontents still ends up a 2-line, 1288296-byte file.

Not sure what else to do here, other than perhaps disable melpa loading entirely in my setup -- I might not need it anyway. as I don't think it has ever really worked properly for me (though for awhile that was because of the http:// instead of https:// prefix on the URL).

[milkypostman]

You have to install the source package if you’re on Linux. Did you try removing all packages?

…

On Sun, Oct 14, 2018 at 12:50 James Craig Burley ***@***.***> wrote: Looks like package.el is built-in (according to C-h P package), and there's no pointer to where the source code might live on my system (I tried find ... -name package.el on various directories without success): package is a built-in package. Status: Built-In. Version: 1.1.0 Summary: Simple package system for Emacs Other versions: 1.0.1 (marmalade), 0.9 (tromey). I also tried renaming ~/.emacs.d/elpa/archives to ~/.emacs.d/elpa/archives.old, but that didn't solve anything -- the new ~/.emacs.d/elpa/archives/melpa/archivecontents still ends up a 2-line, 1288296-byte file. Not sure what else to do here, other than perhaps disable melpa loading entirely in my setup -- I might not need it anyway. as I don't think it has ever really worked properly for me (though for awhile that was because of the http:// instead of https:// prefix on the URL). — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#5755 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AACUsjlpodxVreweZlef-FBlgMjZEU7Oks5uk3ldgaJpZM4Xa9st> .

[jcburley]

Looking at an online version of package.el (so I'm not sure this is the same version my Emacs installation is actually using), I see this comment and code in the definition of package-initialize:

  ;; This uses `package--mapc' so it must be called after
  ;; `package--initialized' is t.
  (package--build-compatibility-table))

That might indicate a bug here, because package--initialized is nil on an Emacs session that has hit the error I'm seeing and is in debug-on-error mode:

(defun package--build-compatibility-table ()
  "Build `package--compatibility-table' with `package--mapc'."
  ;; Initialize the list of built-ins.
  (require 'finder-inf nil t)
  ;; Build compat table.
  (setq package--compatibility-table (make-hash-table :test 'eq))
  (package--mapc #'package--add-to-compatibility-table))

package-installed-p, called by package--mapc, rejects some cases when package--initialized is nil.

So, this looks like a bug in package.el to me, but I'm not sure why we're even getting here in the first place.

[jcburley]

What exactly do you mean by "removing all packages"? I renamed that archives directory away, as I mentioned.

[jcburley]

So I tried removing loading of melpa entirely from my init.el, and also renamed away archives/ again, and now I get the same error, but for tromey.

It's pretty clear now that the toggle-debug-on-error-triggered error I've been describing is actually a bug in GNU Emacs, not in melpa.

Further, even without toggle-debug-on-error, and restoring melpa loading to an earlier place in the package-archives list, whatever comes last in that list -- not necessarily melpa -- hits the problem, according to *Messages*:

Loading /home/craig/.unixhome/components/emacs/init.el (source)...
Loading /home/craig/.unixhome/components/emacs/craig/packages.el (source)...
/home/craig/.unixhome/components/emacs/craig/packages.el might call package-refresh-contents for: 2018-10-14
(("gnu" . "http://elpa.gnu.org/packages/") ("marmalade" . "https://marmalade-repo.org/packages/") ("melpa" . "https://melpa.org/packages/") ("tromey" . "https://tromey.com/elpa/"))
Importing package-keyring.gpg...done
Contacting host: elpa.gnu.org:80
Contacting host: marmalade-repo.org:443
Contacting host: melpa.org:443 [2 times]
Failed to download ‘tromey’ archive.
Wrote /home/craig/.emacs.d/sessions/craig@doe/package-refresh.STAMP
/home/craig/.unixhome/components/emacs/craig/packages.el about to call package-initialize!
/home/craig/.unixhome/components/emacs/craig/packages.el after call to package-initialize!
Loading /home/craig/.unixhome/components/emacs/craig/packages.el (source)...done
Loading /home/craig/.unixhome/components/emacs/craig/clojure.el (source)...done
Loading /home/craig/.unixhome/components/emacs/craig/git.el (source)...done
Loading /home/craig/.unixhome/components/emacs/init.el (source)...done
For information about GNU Emacs and the GNU system, type C-h C-a.

So this is almost certainly not a melpa problem after all!

I'll leave it re-enabled for now and see how things work. It might well be safe to ignore that Failed to download... message...?

Sorry for all the trouble.

[jcburley]

Closing as this does not appear to be a bug in melpa, but rather one (or more) in GNU Emacs itself (package.el, perhaps).

[jcburley]

FWIW, it looks like package--update-downloads-in-progress sees the package--downloads-in-progress list become nil when it removes the final entry on behalf of the second dolist in package--download-and-read-archives. As a result, it calls package--build-compatibility-table, which calls package--mapc (as described in my earlier comment).

But, in that context, package--initialized remains nil, so an exception results.

A possible fix might be to not call package--build-compatibility-table if package--initialized is nil, letting a subsequent call to package-initialize (or similar) take care of that.

[purcell]

I'm having trouble finding the source for package.el

M-x find-library

[purcell]

You should also note that our recommended config snippet here includes an explicit call to package-initialize and some handling of when the host Emacs doesn't support HTTPS. Your pasted config snippet above is not the one we suggest.

Also, you really shouldn't need the tromey archive any more.

[jcburley]

Thanks for find-library -- mine says it can't find, but that's presumably because I'm using a package rather than downloading and building from source!

I'll drop the tromey archive. I've changed the other code somewhat; it's still not the same as what your page suggests, but it should accomplish the same thing.

Anyway, I still think package.el itself has a bug (perhaps easy to fix), but am not sure it's worth pursuing....

[jcburley]

BTW, I confimed that https://elpa.gnu.org/packages/ (i.e. using SSL) works, so perhaps that page should be changed to reflect that (e.g. concat the proto var as done for melpa).

[purcell]

Ah yes, I see that the config snippet suggested at the link I included above has diverged from that on our "Getting Started" page. The former already does as you suggest, and is the one to use.

[purcell]

Ah yes, I see that the config snippet suggested at the link I included above has diverged from that on our "Getting Started" page. The former already does as you suggest, and is the one to use.

Sorry, ignore me: they're both the same.

[jcburley]

Actually, this doesn't look like it can work:

(add-to-list 'package-archives '("gnu" . (concat proto "://elpa.gnu.org/packages/")))

The use of apostrophe before the second ( quotes that entire form, including concat, which I think would result in a non-string in the cdr position.

The melpa one looks correct:

(add-to-list 'package-archives (cons "melpa" (concat proto "://melpa.org/packages/")) t)

That doesn't quote the form at all, instead explicitly cons-ing the keyword string to the value string, which results from evaluating (concat ...).

[jcburley]

FWIW, here's my latest corresponding snippet:

(let* ((no-ssl (and (memq system-type '(windows-nt ms-dos))
                    (not (gnutls-available-p))))
       (prefix (concat (if no-ssl "http://" "https://"))))
  (add-to-list 'package-archives
               (cons "marmalade" (concat prefix "marmalade-repo.org/packages/")) t)
  (add-to-list 'package-archives
               (cons "melpa" (concat prefix "melpa.org/packages/")) t)
  (when (< emacs-major-version 24)
    ;; For important compatibility libraries like cl-lib
    (add-to-list 'package-archives
                 (cons "gnu" (concat prefix "elpa.gnu.org/packages/")))))

[purcell]

Yep, and you'll probably want to call (package-initialize) explicitly - that's in our snippet, and missing from yours. If you don't include it, then Emacs will decide for itself when to call it, which tends to be unpredictable. (Fun fact: it used to be the case that Emacs would look at your init file and if it didn't look like you were calling it, it would call it for you. It might even still work like that.)

[purcell]

Actually, this doesn't look like it can work:

True, fixed, thanks.

[jcburley]

The file does call (package-initialize), but much later after that snippet, which I scoped down to focus on just the building of package-archives. I.e. that (and calling (package-refresh-contents)) was out of scope of the snippet.

(Fun fact: it used to be the case that Emacs would look at your init file and if it didn't look like you were calling it, it would call it for you. It might even still work like that.)

Indeed it still does, as *Messages* shows on a fresh startup (after I edited my init.el to introduce a typo in ;;(package-initialize) along with additional (message ...) calls added the other day that show when it happens):

/home/craig/.unixhome/components/emacs/craig/packages.el about to call package-initialize!
Saving file /home/craig/.emacs.d/init.el...
Wrote /home/craig/.emacs.d/init.el [2 times]
/home/craig/.unixhome/components/emacs/craig/packages.el after call to package-initialize!
Loading /home/craig/.unixhome/components/emacs/craig/packages.el (source)...done
Loading /home/craig/.unixhome/components/emacs/craig/clojure.el (source)...done
Loading /home/craig/.unixhome/components/emacs/craig/git.el (source)...done
Loading /home/craig/.unixhome/components/emacs/init.el (source)...done
For information about GNU Emacs and the GNU system, type C-h C-a.
GNU Emacs 25.3.2 (x86_64-pc-linux-gnu, GTK+ Version 3.18.9) of 2017-09-12