elpa straight mirror updates hang #1049

minad · 2023-02-06T08:15:43Z

The ELPA mirror lags behind. Recently it didn't update for three weeks. Is the automatic synchronization broken? The problem is that this breaks updates. I've observed this for the Compat package and it's downstream users.

progfolio · 2023-02-07T02:03:03Z

The mirror is intended to update daily.
It looks like @raxod502 recently committed some changes: radian-software/gnu-elpa-mirror@ac3bc55

Also appears as if the last time it was updated was three days ago, so perhaps there's still an issue there.

progfolio · 2023-02-07T02:08:31Z

The problem is that this breaks updates. I've observed this for the Compat package and it's downstream users.

As a workaround, anyone relying on Compat could install it by pointing the recipe to its github repostiory.

minad · 2023-02-07T13:01:54Z

@progfolio This is not really feasible to expect from the users since many packages depend on Compat now.

progfolio · 2023-02-07T13:53:06Z

It's a matter of pasting the following before any packages which require compat:

(straight-use-package '(compat :host github :repo "emacs-compat/compat"))

Personally, I try to minimize my reliance on mirrors, and I'm sure there are other users who would appreciate the simple solution. Feel free to link them to this comment.

An alternative solution which would be transparent to users while protecting them from future mirror failures would be to add compat to another ELPA such as MELPA.

Im not familiar enough with the gnu-elpa-mirror code base to diagnose and fix the problem myself, so those solutions are the best I can offer until it is fixed.

minad · 2023-02-07T14:14:59Z

It's a matter of pasting the following before any packages which require compat:

Of course, it is not a big deal in principle. The problem however is that breakage occurs for downstream packages like Magit, where the users are not even aware that Compat is used. I want to avoid such breakage in the first place. On the other hand most Straight users are experienced and as such they should be able to figure out updated-related issues caused by Straight themselves.

An alternative solution which would be transparent to users while protecting them from future mirror failures would be to add compat to another ELPA such as MELPA.

That's not such a great option, since it could lead to more breakage down the road except if all the development happens in a separate development branch. Note that the problem is neither Compat nor ELPA. It is the Straight mirror, which needs fixing.

minad · 2023-02-07T14:37:44Z

@progfolio How do you solve the mirror problem with ELPA in Elpaca? For some reason Straight doesn't want to use the ELPA repositories directly (reliability, weird branches for packages, ...).

progfolio · 2023-02-07T15:14:26Z

How do you solve the mirror problem with ELPA in Elpaca?

Currently, I don't. Elpaca uses gnu-elpa-mirror. It's on my TODO list to support GNU ELPA directly.

For some reason Straight doesn't want to use the ELPA repositories directly (reliability, weird branches for packages, ...).

I can only speak for myself here, but I've experienced intermittent reliability/speed issues when cloning repos hosted on savannah. The package-per-branch model is a pain, too.

I've not used it myself, but straight has an option to bypass the mirror. See the straight-recipes-gnu-elpa-use-mirror option. It would have to be set to nil prior to straight's bootstrapping snippet. Looking at the source for it, I'm not sure it will work, though.

minad · 2023-02-07T15:22:14Z

Currently, I don't. Elpaca uses gnu-elpa-mirror. It's on my TODO list to support GNU ELPA directly.

Okay, I would appreciate this! Is the long term plan to deprecate Straight or to merge Elpaca back?

I can only speak for myself here, but I've experienced intermittent reliability/speed issues when cloning repos hosted on savannah.

Makes sense, in particular if many users access it via Straight.

The package-per-branch model is a pain, too.

Agree. But the elpa mirror on github also caused a great amount of pain, so the introduction of the mirror seems to be a bit of an overreaction. It would be better to support the ELPA repositories better in Straight directly and handle failures gracefully.

I've not used it myself, but straight has an option to bypass the mirror. See the straight-recipes-gnu-elpa-use-mirror option. It would have to be set to nil prior to straight's bootstrapping snippet.

Would it make sense to set this option for now by default until the Straight mirror issues have been sorted out? Probably not since most users won't update Straight in the meantime? Hopefully @raxod502 can soon take another look at the mirror scripts.

progfolio · 2023-02-07T16:22:18Z

Is the long term plan to deprecate Straight or to merge Elpaca back?

I don't plan on merging Elpaca back.
I've made several design decisions (e.g. only supporting git VCS) which are incompatible with straight.el's goals. I don't necessarily think straight.el needs to be deprecated either, due to these differences.

It would be better to support the ELPA repositories better in Straight directly and handle failures gracefully.

I agree there's room for improvement here.

Would it make sense to set this option for now by default until the Straight mirror issues have been sorted out? Probably not since most users won't update Straight in the meantime?

I don't think so. Just looking at the source for direct GNU ELPA support, it doesn't look like it will work properly to me. It assumes the packages directory is already populated, which is not the case after a fresh clone.

straight.el/straight.el

Lines 3328 to 3360 in 3eca39d

    
           ;;;;;;; GNU ELPA source 
        
           (defcustom straight-recipes-gnu-elpa-url 
        
             "https://git.savannah.gnu.org/git/emacs/elpa.git" 
        
             "URL of the Git repository for the GNU ELPA package repository." 
        
             :type 'string) 
        
           (defun straight-recipes-gnu-elpa-retrieve (package) 
        
             "Look up a PACKAGE recipe in GNU ELPA. 
        
           PACKAGE should be a symbol. If the package is maintained in GNU 
        
           ELPA (and should be retrieved from there, which isn't the case if 
        
           the package is built in to Emacs), return a MELPA-style recipe. 
        
           Otherwise, return nil." 
        
             (unless (memq package straight-recipes-gnu-elpa-ignored-packages) 
        
               (when (file-exists-p (expand-file-name (symbol-name package) "packages/")) 
        
                 ;; All the packages in GNU ELPA are just subdirectories of the 
        
                 ;; same repository. 
        
                 `(,package :type git 
        
                            :repo ,straight-recipes-gnu-elpa-url 
        
                            :files (,(format "packages/%s/*.el" 
        
                                             (symbol-name package))) 
        
                            :local-repo "elpa")))) 
        
           (defun straight-recipes-gnu-elpa-list () 
        
             "Return a list of recipe names available in GNU ELPA, as a list of strings." 
        
             (cl-remove-if 
        
              (lambda (package) 
        
                (memq (intern package) straight-recipes-gnu-elpa-ignored-packages)) 
        
              (straight--directory-files "packages/"))) 
        
           (defun straight-recipes-gnu-elpa-version () 
        
             "Return the current version of the GNU ELPA retriever." 
        
             2)

The following test (unless I'm missing something) confirms as much:

Test Case

(straight-bug-report
  :user-dir "straight.elpa-source"
  :preserve t
  :pre-bootstrap 
  (setq straight-recipes-gnu-elpa-use-mirror nil)
  :post-bootstrap 
  (straight-use-package
   '(compat :source gnu-elpa)))

Test run at: 2023-02-07 10:23:46
system-type: gnu/linux
straight-version: prerelease (HEAD -> develop, origin/develop) 56a8b6c 2023-02-04
emacs-version: GNU Emacs 30.0.50 (build 1, x86_64-pc-linux-gnu, GTK+ Version 3.24.36, cairo version 1.17.8) of 2023-02-06

Output

Bootstrapping straight.el...
Bootstrapping straight.el...done
Looking for gnu-elpa recipe → Cloning melpa...
Looking for gnu-elpa recipe → Cloning melpa...done
Looking for nongnu-elpa recipe → Cloning elpa (for gnu-elpa)...
Looking for nongnu-elpa recipe → Cloning elpa (for gnu-elpa)...done
Looking for emacsmirror-mirror recipe → Cloning nongnu-elpa...
Looking for emacsmirror-mirror recipe → Cloning nongnu-elpa...done
Looking for emacsmirror-mirror recipe → Cloning el-get...
Looking for emacsmirror-mirror recipe → Cloning el-get...done
Looking for straight recipe → Cloning emacsmirror-mirror...
Looking for straight recipe → Cloning emacsmirror-mirror...done
Building straight...
Building straight...done

Test run with version: prerelease (HEAD -> develop, origin/develop) 56a8b6c 2023-02-04
Packages:
"straight"                n/a                  develop 56a8b6c 2023-02-04
"org-elpa"                n/a                  n/a
"melpa"                   n/a                  master d0a5234d 2023-02-06
"gnu-elpa"                n/a                  main 5c276f056e 2023-01-31
"nongnu-elpa"             n/a                  main 85c7a060c6 2023-01-26
"el-get"                  melpa                master 17db7640 2023-01-28
"emacsmirror-mirror"      n/a                  master 03db6f7 2023-02-03

Error: (wrong-type-argument stringp nil)

I'll have to experiment to see what the best option is.

raxod502 · 2023-02-16T03:08:09Z

Yeah, using GNU ELPA directly is a non-starter IMO. You're very welcome to try it, but I found it to be an utter disaster. The mirror has maintenance problems but at least I am the one getting alerted for failures rather than all the weird git submodule edge cases and build script errors of GNU ELPA happening on users' laptops.

The main reason for the recent outages being so long is simply that I am the only person maintaining the mirror at present, and even though I dedicated most/all of my available development time to fixing the mirror issue each week, it took almost a month to get through it all.

Anyway, as mentioned in the linked thread on GNU ELPA Mirror, I believe I've fixed the problem.

minad added the bug label Feb 6, 2023

minad closed this as completed Feb 18, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

elpa straight mirror updates hang #1049

elpa straight mirror updates hang #1049

minad commented Feb 6, 2023 •

edited

Loading

progfolio commented Feb 7, 2023

progfolio commented Feb 7, 2023 •

edited

Loading

minad commented Feb 7, 2023

progfolio commented Feb 7, 2023 •

edited

Loading

minad commented Feb 7, 2023 •

edited

Loading

minad commented Feb 7, 2023

progfolio commented Feb 7, 2023 •

edited

Loading

minad commented Feb 7, 2023 •

edited

Loading

progfolio commented Feb 7, 2023

raxod502 commented Feb 16, 2023

elpa straight mirror updates hang #1049

elpa straight mirror updates hang #1049

Comments

minad commented Feb 6, 2023 • edited Loading

progfolio commented Feb 7, 2023

progfolio commented Feb 7, 2023 • edited Loading

minad commented Feb 7, 2023

progfolio commented Feb 7, 2023 • edited Loading

minad commented Feb 7, 2023 • edited Loading

minad commented Feb 7, 2023

progfolio commented Feb 7, 2023 • edited Loading

minad commented Feb 7, 2023 • edited Loading

progfolio commented Feb 7, 2023

raxod502 commented Feb 16, 2023

minad commented Feb 6, 2023 •

edited

Loading

progfolio commented Feb 7, 2023 •

edited

Loading

progfolio commented Feb 7, 2023 •

edited

Loading

minad commented Feb 7, 2023 •

edited

Loading

progfolio commented Feb 7, 2023 •

edited

Loading

minad commented Feb 7, 2023 •

edited

Loading