[question] how do I tell spacemacs to look for a local elpa mirror?

[ninrod]

for example, say that I git cloned this repo: https://github.com/syl20bnr/spacemacs-elpa-mirror

And I want to use that as my elpa mirror. How do I say to spacemacs to stop trying to connect to melpa/elpa whatever, and fetch the packages in my local git cloned folder instead?

That would be super useful for me because I'm behind a corporate firewall that prevents emacs from fetching elpa packages, but allows github access.

[d12frosted]

AFAIK, you can't just change the value of package-archives as it will be change by Spacemacs in configuration-layer/initialize. But you can set desired archives by changing the value of configuration-layer--elpa-archives. For example, put following in dotspacemacs/init.

(setq configuration-layer--elpa-archives
      '(("melpa" . "https://raw.githubusercontent.com/d12frosted/elpa-mirror/master/melpa/")
        ("org"   . "https://raw.githubusercontent.com/d12frosted/elpa-mirror/master/org/")
        ("gnu"   . "https://raw.githubusercontent.com/d12frosted/elpa-mirror/master/gnu/")))

The only problem - currently configuration-layer//resolve-package-archives doesn't respect URLs starting with file://, so if you wish to use other than http[s] protocol, you need to modify that function.

[d12frosted]

Ok, just pushed 8578b40. It allows you to use local archive. For example, put following in your dotspacemacs/init function (just don't forget to update path).

(setq configuration-layer--elpa-archives
        '(("melpa" . "/Users/d12frosted/Developer/d12frosted/elpa-mirror/melpa")
          ("org"   . "/Users/d12frosted/Developer/d12frosted/elpa-mirror/org/")
          ("gnu"   . "/Users/d12frosted/Developer/d12frosted/elpa-mirror/gnu/")))

[ninrod]

Ok, will try that ASAP.

[ninrod]

Strange. It kind of worked. It errors out on me saying that it cannot find the spacemacs-theme- package.

[d12frosted]

Oh welp, never really tried using local archive. It works with remote archive, but when I try to bootstrap using local archive I experience pretty similar problem. Gonna investigate that. 😸

Funky, over the internet I see people using local archives, so it might be something wrong in our setup. I thought that the only change required is the url resolver function. But seems that it's not enough 💃

I'll keep you updated. 😈

[d12frosted]

@ninrod Oh I just realised why it doesn't work. It turns out that configuration-layer/retrieve-package-archives doesn't work with /path/to/mirror URLs. As a (temporary) workaround you can redefine configuration-layer/retrieve-package-archives to following function. Meanwhile I am gonna make sure that local urls work with #6828.

(defun configuration-layer/retrieve-package-archives (&optional quiet force)
  "Retrieve all archives declared in current `package-archives'.

This function first performs a simple GET request with a timeout in order to
fix very long refresh time when an archive is not reachable.

Note that this simple GET is a heuristic to determine the availability
likelihood of an archive, so it can gives false positive if the archive
page is served but the archive is not.

If QUIET is non nil then the function does not print message in the Spacemacs
home buffer.

If FORCE is non nil then refresh the archives even if they have been already
refreshed during the current session."
  (unless (and configuration-layer--package-archives-refreshed
               (not force))
    (setq configuration-layer--package-archives-refreshed t)
    (let ((count (length package-archives))
          (i 1))
      (dolist (archive package-archives)
        (unless quiet
          (spacemacs-buffer/replace-last-line
           (format "--> refreshing package archive: %s... [%s/%s]"
                   (car archive) i count) t))
        (spacemacs//redisplay)
        (setq i (1+ i))
        (unless (eq 'error
                    (with-timeout
                        (dotspacemacs-elpa-timeout
                         (progn
                           (display-warning
                            'spacemacs
                            (format
                             "\nError connection time out for %s repository!"
                             (car archive)) :warning)
                           'error))
                      (condition-case err
                          (unless (string-prefix-p "/" (cdr archive))
                            (url-retrieve-synchronously (cdr archive)))
                        ('error
                         (display-warning 'spacemacs
                          (format
                           "\nError while contacting %s repository!"
                           (car archive)) :warning)
                         'error))))
          (let ((package-archives (list archive)))
            (package-refresh-contents))))
      (package-read-all-archive-contents)
      (unless quiet (spacemacs-buffer/append "\n")))))

[ninrod]

nice work @d12frosted! I will try that ASAP.

[d12frosted]

Also keep in mind that not all packages configured in Spacemacs are installed from MELPA, GNU ELPA or Org package archive. Some of them are installed from GitHub. Not many of them, but still.

[d12frosted]

Using local archive urls is possible after #6828 is merged.

[CeleritasCelery]

@d12frosted Is there a way to make this hack work with relative paths from home directory (i.e. ~/some/elpa/location/) as opposed to absolute paths (/path/from/root/to/elpa/location/) as it is setup now?

[ninrod]

@PrgrmAtCeleritas, I don't know exactly what spacemacs is doing differently, but I'm rolling my own config and this snippet is working beautifully for me, using @d12frosted elpa-mirror repo:

(require 'package)
(setq package-enable-at-startup nil)
(setq package-archives '(("melpa" . "~/.emacs.d/mirror-elpa/melpa/")
                         ("org"   . "~/.emacs.d/mirror-elpa/org/")
                         ("gnu"   . "~/.emacs.d/mirror-elpa/gnu/")))

[CeleritasCelery]

@ninrod I tried something similar in the "Spacemacs" flavor but it won't work unless I give it the absolute path. Maybe I can bypass the the Spacemacs sugar and set it up like you did since it is not a problem with Emacs apparently.

[ninrod]

@PrgrmAtCeleritas, Are you using #6828? @d12frosted implemented quite a few hacks in that to make it work properly in spacemacs.

[CeleritasCelery]

I am not. I am using the code snippet provided above. #6828 seemed to deal with Melpa being down but I could give it a shot.

[d12frosted]

Oh sorry, missed this issue.

@ninrod thanks for answers 😸

@PrgrmAtCeleritas in order to use this 'trick' in Spacemacs you have to add following snipped to your dotspacemacs/user-init function.

  (setq configuration-layer--elpa-archives
        '(;; ("melpa" . "/Users/d12frosted/Developer/d12frosted/elpa-mirror/melpa/")
          ;; ("org"   . "/Users/d12frosted/Developer/d12frosted/elpa-mirror/org/")
          ;; ("gnu"   . "/Users/d12frosted/Developer/d12frosted/elpa-mirror/gnu/")
          ("melpa" . "https://raw.githubusercontent.com/d12frosted/elpa-mirror/master/melpa/")
          ("org"   . "https://raw.githubusercontent.com/d12frosted/elpa-mirror/master/org/")
          ("gnu"   . "https://raw.githubusercontent.com/d12frosted/elpa-mirror/master/gnu/")))

This is exactly what I use (mostly for testing purposes or when I need to quickly reinstall all packages).

P. S. Note that using configuration-layer--elpa-archives is a hack rather than documented feature. If at one point it stops working - you have to find another way 😸

[d12frosted]

Ah, almost forgot, #6828 allows usage of local archive (like /Users/d12frosted/Developer/d12frosted/elpa-mirror/melpa/).

[CeleritasCelery]

Thanks @d12frosted. I may not have made my question as clear as I should have. I currently set configuration-layer--elpa-archives to point to a local mirror in my dotspacemacs and it works great. The problem I am having is that my workstations are all remote and and I often change machines. Therefore the absolute path changes as well. I have a symlink through my home directory to them, but the configuration-layer--elpa-archives won't accept relative paths.

for example if I use

(setq configuration-layer--elpa-archives
    '(("melpa"        . "~/nfs/xx/disks/xx0123/personal/emacs/elpa-mirror/spacemacs-elpa-mirror-latest/melpa/")
      ("org"          . "~/nfs/xx/disks/xx0123/personal/emacs/elpa-mirror/spacemacs-elpa-mirror-latest/org/")
      ("gnu"          . "~/nfs/xx/disks/xx0123/personal/emacs/elpa-mirror/spacemacs-elpa-mirror-latest/gnu/")))

it will work just fine. But if I try to use

(setq configuration-layer--elpa-archives
    '(("melpa"        . "~/personal/emacs/elpa-mirror/spacemacs-elpa-mirror-latest/melpa/")
      ("org"          . "~/personal/emacs/elpa-mirror/spacemacs-elpa-mirror-latest/org/")
      ("gnu"          . "~/personal/emacs/elpa-mirror/spacemacs-elpa-mirror-latest/gnu/")))

it will not be able to find it. This causes me to have to change the path whenever I am using a different Network File System.

Is there a way to make it correctly process the second example so that I can use the ~ for a relative path from my home directory?

[ninrod]

@PrgrmAtCeleritas, you could set up absolute path symlinks in all your machines that point to the correct location.

for example, in machine A:

/opt/emacs/melpa-dir -> path/to/melpa/dir-in-machine-A
/opt/emacs/elpa-dir -> path/to/elpa/dir-in-machine-A
...
so on and so forth

[CeleritasCelery]

I don't have admin privileges. So I can only write to NFS directories, which are different for each machine. So the only consistent thing across all machines is ~ points to the home directory for that particular machine.

[ninrod]

Ouch. that sucks.

[d12frosted]

@PrgrmAtCeleritas try following snippet:

(setq configuration-layer--elpa-archives
             `(("melpa"        . ,(expand-file-name "~/Developer/d12frosted/elpa-mirror/melpa"))
               ("org"          . ,(expand-file-name "~/Developer/d12frosted/elpa-mirror/org"))
               ("gnu"          . ,(expand-file-name "~/Developer/d12frosted/elpa-mirror/gnu"))))

But don't forget to adapt paths 😸

[ninrod]

I get lost with all those backquotes and commas. I don't really know what that means. I've just got used to single quotes '.

[d12frosted]

@ninrod 😸

Might be useful to read about Quoting and Backquote. In short - backquote allows to evaluate some code in a quoted expression. And , indicates what to evaluate.

[CeleritasCelery]

That worked @d12frosted. Thank you!

[d12frosted]

@ninrod @PrgrmAtCeleritas also you can avoid using backquotes and duplicating expand-file-name by using following snippet.

(setq configuration-layer--elpa-archives
      (mapcar
       (lambda (pair)
         (cons (car pair)
               (expand-file-name (cdr pair))))
       '(("melpa" . "~/personal/emacs/elpa-mirror/spacemacs-elpa-mirror-latest/melpa/")
         ("org"   . "~/personal/emacs/elpa-mirror/spacemacs-elpa-mirror-latest/org/")
         ("gnu"   . "~/personal/emacs/elpa-mirror/spacemacs-elpa-mirror-latest/gnu/"))))

[ninrod]

@d12frosted thanks. what was your path to learn elisp? did you read the reference manual?

[d12frosted]

Well, to be honest I didn't read it all. Only some parts that I was interested in. I am planning to read it at some point 😸 I began to learn Emacs Lisp when started to make my own configs for the first time. But it was more copy-paste than actual coding. When I started to use and contribute to Spacemacs I really started to 'learn by practice'. Sylvain, Eivind and bmag are the guys who explained me several tricks and topics 😸 So my way is practice + theoretical parts from reference manual.

[ninrod]

@d12frosted nice. You learn a lot from open source.

Now just one minor question. What is the meaning of that . dot like in here:

("gnu"   . "~/personal/emacs/elpa-mirror/spacemacs-elpa-mirror-latest/gnu/")

Why can't I just write

'("gnu"  "~/personal/emacs/elpa-mirror/spacemacs-elpa-mirror-latest/gnu/")

which is a list of 2 strings?

[d12frosted]

@ninrod Checkout documentation about dotted pair notation.

As example:

ELISP> (cdr '(1 . 2))
2 (#o2, #x2, ?\C-b)
ELISP> (cdr '(1 2))
(2)

As you can see, without dot, cdr is list, but usage of configuration-layer--elpa-archives requires it to be a string instead of list.

[ninrod]

@d12frosted thanks I'll have to read the manual. Good stuff and seems well written.

[mfehlhaber]

I am using the following to my full contentment with spacemacs 0.200.9 and thus suggesting to close this issue.

(defun dotspacemacs/init ()
  (setq configuration-layer--elpa-archives
        '(("melpa" . "~/.emacs.d/elpa-mirror/")
          ("org"   . "~/.emacs.d/elpa-mirror/")
          ("gnu"   . "~/.emacs.d/elpa-mirror/")))
...
)

[ninrod]

Hum. was #6828 merged?

[bhipple]

It looks like it hasn't been merged yet, and this doesn't work in develop without those commits, as far as I can tell. This would be a really nice feature to have; I'm running into a lot of issues where my plugin versions are slightly different between my various computers and I see slightly different behavior, and I'd like to be able to pin everything to a distribution I control.

[CeleritasCelery]

You can add packages to the dotspacemacs-frozen-packages to keep certain packages from being automatically updated. You can also change the code yourself to enable this feature.

[ninrod]

I don't see why this can't be merged in. @d12frosted do you know the status on this?

I'm using this for 1 year in my own config and it has been fantastic, super stable and fast.

[CeleritasCelery]

I have been using it too

[d12frosted]

@nickanderson unfortunately, no I don't know.

But using local mirror works on develop. At least, I am using it without any 'special' commits to Spacemacs.

I just have following line inside of dotspacemacs/init function of .spacemacs file

  (setq configuration-layer--elpa-archives
        `(("melpa" . ,(concat d12-path-elpa-mirror-home "melpa/"))
          ("org"   . ,(concat d12-path-elpa-mirror-home "org/"))
          ("gnu"   . ,(concat d12-path-elpa-mirror-home "gnu/"))
          ("melpa" . "https://raw.githubusercontent.com/d12frosted/elpa-mirror/master/melpa/")
          ("org"   . "https://raw.githubusercontent.com/d12frosted/elpa-mirror/master/org/")
          ("gnu"   . "https://raw.githubusercontent.com/d12frosted/elpa-mirror/master/gnu/")))

In case it doesn't work for you, let me know and I'll look into it.

[d12frosted]

@CeleritasCelery my previous comment is addressed to you as well 😸

[chenzhaobo]

in windows ,you can set like this

(defun dotspacemacs/init ()
(setq configuration-layer--elpa-archives
'(("melpa-cn" . "//?/f:/elpa-mirror/melpa/")
("org-cn" . "//?/f:/elpa-mirror/org/")
("gnu-cn" . "//?/f:/elpa-mirror/gnu/")))
...
)

windows path in emacs use like this:

long UNC or UNCW, such as ?\C:\File or ?\UNC\Server\Volume\File

[AleXoundOS]

@d12frosted, it's probably better to put this in dotspacemacs/user-init, because dotspacemacs/init is not recommended for adding new options as I understood. Works for me.

[d12frosted]

Closing this issue as the initial question was answered and it's possible to use local ELPA archive for a long time already 😸 Let me know if you disagree with this action of treason!

[LoganBarnett]

I had trouble getting the examples here to work, but it looks like the configuration-layer--elpa-archives has been renamed to configuration-layer-elpa-archives. Using that name instead makes everything seem to work for me.