Consider using straight as the package manager #374

rgrinberg · 2018-01-26T05:25:12Z

I'm currently a user of spacemacs, but spacemacs is quite heavy, buggy, and contains lots of cruft I will never use. The layer system in particular is a bit of a nightmare.

But my biggest complaint with spacemacs is that it doesn't use https://github.com/raxod502/straight.el/. I'd really like to use straight as my package manager because I want to use lock files to secure my configuration against random breakage. I'd like a convenient workflow for contributing upstream. And all that other git goodness that comes with having access to the full git history of every package. If doom were to use straight, I'd definitely switch from spacemacs.

hlissner · 2018-01-26T06:17:58Z

I hear that. I already have straight.el in my crosshairs, as well as a local branch where I'm experimenting with straight.el integration. However, I'm waiting on some key features promised for the v1.0 release (of straight.el) before I can move forward with it. I can't predict when that'll be, but I'll keep this issue open and keep you posted.

rgrinberg · 2018-04-11T09:30:39Z

Would it be possible to list which features are still missing in straight? I'd like to make sure that there's progress on those and perhaps work on some of them myself.

hlissner · 2018-04-11T22:59:26Z

I haven't looked at straight for a while, but at the time this issue was created, what Doom needed most (from straight) was a non-interactive workflow for updating packages (straight-pull-packages minus the prompt for each and every package) and GNU ELPA support.

I had a couple other misgivings about poor error handling, confusing pull/push prompts when you run straight-freeze-versions, and unimplemented support for VCSes other than git, but I'm not sure if these issues have been addressed yet.

This week is really busy for me. After the 16th I'll revisit straight and see where we stand.

rgrinberg · 2018-04-22T03:10:20Z

All those issues are straight forward except for:

and unimplemented support for VCSes other than git

Is this really a blocker though? I don't think there are any issues serious packages out there that aren't hosted on git.

hlissner · 2018-04-22T03:40:56Z

You're right, it's not really an issue. I was under the impression that there were (relatively common) non-git/svn dependencies out there, which straight would presumably be unable to fetch, much less manage (and so, Doom wouldn't be able to either). Not sure where I got that idea, I'll have to dig around straight's issue tracker.

hlissner · 2018-07-31T21:04:14Z

Adopt straight.el as our package manager

There are a few kinks to iron out, but for the most part it's done. Doom Emacs, powered by straight. Goodbye gnutls and elpa/quelpa issues. This update doesn't come with rollback or lockfile support yet, but I will eventually include one with Doom, and packages will be (by default, anyway) updated in sync with Doom. Relevant threads: #1577 #1566 #1473

hlissner · 2019-07-22T03:35:46Z

After a long wait, it's done! I published the straight branch a short while ago, which replaces package.el and quelpa with straight.el. This sidesteps all the recent gnutls and gnu elpa issues (#1582 and #1566).

I'll be spending the next week ironing out the kinks before merging it in. It'd be a huge help if you could give a try. Here are a few things you should know going in:

Make a backup of your old .emacs.d! You have to reinstall all your packages. Simply switch branches, delete ~/.emacs.d/.local, and doom refresh away!
Two naming conventions for functions/variables have changed
- Hooks were doom|hook and are now doom-hook-h
- Advice functions were doom*advice and are now doom-advice-a
doom quickstart is now doom install (though quickstart is still
around, it is deprecated).
doom compile :plugins is now doom build -f
doom env will now do what doom env refresh did. doom env clear still exists too, but doom env {auto,enable} are gone (doom refresh will always regenerate your envvar file if it exists).
The package management system is young. Occasionally, Doom may throw weird errors shortly after startup following package updates. If that happens, try doom autoloads followed by doom rebuild all. It doesn't happen often, fortunately.
I haven't employed lockfiles or pinned any packages yet. That comes after documentation.
If you have package! declarations with a :recipe, the recipe format has changed ever so slightly. You'll find more about it on straight's project page, but it usually won't amount to any more than changing :fetcher to :host.
def-package! is now use-package!. This is to make it absolutely clear that this macro is simply a thin wrapper around use-package!.

For Doom users that have been with us for a while, this may cause some breakages. Sorry for the trouble, but the good news is, these are all the breaking changes I have in store.

Let me know if it gives you any trouble. I will keep this thread open until straight makes it to develop.

credmp · 2019-07-22T12:30:11Z

Don't know if this is the right place for this, but I tested this branch on Ubuntu 18.04 LTS with a clean Doom Emacs install. It works correctly, the gnutls.el errors (invalid request) are gone.

ydewit · 2019-07-22T16:14:50Z

I just switched to straight, rm -rf .local/ and ran ./bin/doom refresh but got the following message: ../../../autoloads.el:1410:1:Error: Symbol’s value as variable is void: next`

> Checking core autoloads file
  > Regenerating core autoloads file
  ✓ Scanned 66 file(s)
  ✓ Generated new autoloads.el
  ✓ Expanded module autoload paths
  ✓ Generated autodefs
  ✓ Clean up autoloads
  > Byte-compiling .local/autoloads.el...


In toplevel form:
../../../autoloads.el:1410:1:Error: Symbol’s value as variable is void: next
> Installing & building packages...

hlissner · 2019-07-22T16:42:14Z

@ydewit Are you try this out with a pre-existing private config, or a blank one?

Also try bin/doom --debug refresh to see if it will produce a backtrace from the error.

And change TARGETS to PACKAGE, which is a more descriptive name. Mentioned in #374

hlissner · 2019-07-22T16:59:00Z

@ydewit Nevermind, I found the cause and fixed the issue in 457b542

ydewit · 2019-07-22T17:31:49Z

@hlissner From a blank one.

I just tried the same using the latest and it worked without issues. Thank you.

HaoZeke · 2019-07-23T08:41:43Z

So quick question, do we now need to switch to using the use-package integration? So we don't need to separately declare the package and configure it in different files?

hlissner · 2019-07-23T16:45:13Z

@HaoZeke No, it's business as usual. Your package declarations go in DOOMDIR/packages.el, and your configuration goes into DOOMDIR/config.el. Doom's package management will always follow this declarative paradigm.

However, this is just Doom's (strong) suggestion. If you really don't like this paradigm, you could forget about packages.el (delete it if you'd like) and simply load straight (or package.el) right away, in config.el, and install your packages with the :straight or :ensure properties that use-package provides. That way you won't have to consult the bin/doom utility nearly as often (it would only be used to manage the built-in packages, then yours will be auto-downloaded/installed on startup), but you'd lose out on a few startup optimizations that Doom does, under the assumption that you will do things its way.

madhat2r · 2019-07-23T18:54:12Z

Switched to straight branch and this is what I get when opening emacs now

Error in an autoloads file: autoloads.pkg.el, (void-variable sly-contribs)

Full trace:

Debugger entered--Lisp error: (void-variable sly-contribs)
  add-to-list(sly-contribs sly-macrostep append)
  byte-code("\300\301\302\303#\210\304\305!\203\021\0\305\306\307\"\210\304\305!\203\034\0\305\310\311\"\210\300\207" [add-to-list sly-contribs sly-macrostep append fboundp register-definition-prefixes "sly-macrostep" ("sly-macrostep") "sly-macrostep-tests" ("sly-macrostep-")] 4)
  load("/home/micah/.emacs.d.doom/.local/autoloads.pkg" noerror nomessage)
  (let (command-switch-alist) (load (if noninteractive file (file-name-sans-extension file)) (quote noerror) (quote nomessage)))
  (condition-case e (let (command-switch-alist) (load (if noninteractive file (file-name-sans-extension file)) (quote noerror) (quote nomessage))) ((debug error) (if noninteractive (message "Autoload file warning: %s -> %s" (car e) (error-message-string e)) (signal (quote doom-autoload-error) (list (file-name-nondirectory file) e)))))
  doom-load-autoloads-file("/home/micah/.emacs.d.doom/.local/autoloads.pkg.el")
  (let ((core-autoloads-p (doom-load-autoloads-file doom-autoload-file)) (pkg-autoloads-p (doom-load-autoloads-file doom-package-autoload-file))) (if (and core-autoloads-p (not force-p)) (progn (after! (:or package straight) (require (quote core-packages))) (after! straight (doom-initialize-packages))) (let ((default-directory doom-core-dir)) (let ((--dolist-tail-- (file-expand-wildcards "autoload/*.el"))) (while --dolist-tail-- (let ((file ...)) (load file t t) (setq --dolist-tail-- (cdr --dolist-tail--)))))) (let ((--dolist-tail-- (list doom-local-dir doom-etc-dir doom-cache-dir doom-elpa-dir))) (while --dolist-tail-- (let ((dir (car --dolist-tail--))) (if (file-directory-p dir) nil (make-directory dir (quote parents))) (setq --dolist-tail-- (cdr --dolist-tail--))))) (require (quote core-packages)) (doom-ensure-straight) (doom-initialize-packages force-p)) (if (or (and core-autoloads-p pkg-autoloads-p) force-p noninteractive) nil (if core-autoloads-p nil (message "Your Doom core autoloads file is missing")) (if pkg-autoloads-p nil (message "Your package autoloads file is missing")) (user-error "Run `bin/doom refresh' to generate them")))
  (progn (setq doom-init-p t) (setq exec-path doom--initial-exec-path load-path doom--initial-load-path process-environment doom--initial-process-environment) (require (quote core-lib)) (require (quote core-modules)) (let ((core-autoloads-p (doom-load-autoloads-file doom-autoload-file)) (pkg-autoloads-p (doom-load-autoloads-file doom-package-autoload-file))) (if (and core-autoloads-p (not force-p)) (progn (after! (:or package straight) (require (quote core-packages))) (after! straight (doom-initialize-packages))) (let ((default-directory doom-core-dir)) (let ((--dolist-tail-- (file-expand-wildcards "autoload/*.el"))) (while --dolist-tail-- (let (...) (load file t t) (setq --dolist-tail-- ...))))) (let ((--dolist-tail-- (list doom-local-dir doom-etc-dir doom-cache-dir doom-elpa-dir))) (while --dolist-tail-- (let ((dir ...)) (if (file-directory-p dir) nil (make-directory dir ...)) (setq --dolist-tail-- (cdr --dolist-tail--))))) (require (quote core-packages)) (doom-ensure-straight) (doom-initialize-packages force-p)) (if (or (and core-autoloads-p pkg-autoloads-p) force-p noninteractive) nil (if core-autoloads-p nil (message "Your Doom core autoloads file is missing")) (if pkg-autoloads-p nil (message "Your package autoloads file is missing")) (user-error "Run `bin/doom refresh' to generate them"))) (if noninteractive nil (doom-load-envvars-file doom-env-file (quote noerror))))
  (if (or force-p (not doom-init-p)) (progn (setq doom-init-p t) (setq exec-path doom--initial-exec-path load-path doom--initial-load-path process-environment doom--initial-process-environment) (require (quote core-lib)) (require (quote core-modules)) (let ((core-autoloads-p (doom-load-autoloads-file doom-autoload-file)) (pkg-autoloads-p (doom-load-autoloads-file doom-package-autoload-file))) (if (and core-autoloads-p (not force-p)) (progn (after! (:or package straight) (require (quote core-packages))) (after! straight (doom-initialize-packages))) (let ((default-directory doom-core-dir)) (let ((--dolist-tail-- ...)) (while --dolist-tail-- (let ... ... ...)))) (let ((--dolist-tail-- (list doom-local-dir doom-etc-dir doom-cache-dir doom-elpa-dir))) (while --dolist-tail-- (let (...) (if ... nil ...) (setq --dolist-tail-- ...)))) (require (quote core-packages)) (doom-ensure-straight) (doom-initialize-packages force-p)) (if (or (and core-autoloads-p pkg-autoloads-p) force-p noninteractive) nil (if core-autoloads-p nil (message "Your Doom core autoloads file is missing")) (if pkg-autoloads-p nil (message "Your package autoloads file is missing")) (user-error "Run `bin/doom refresh' to generate them"))) (if noninteractive nil (doom-load-envvars-file doom-env-file (quote noerror)))))
  doom-initialize(nil)
  eval-buffer(#<buffer  *load*-786279> nil "/home/micah/.emacs.d.doom/core/core.el" nil t)  ; Reading at buffer position 23013
  load-with-code-conversion("/home/micah/.emacs.d.doom/core/core.el" "/home/micah/.emacs.d.doom/core/core.el" nil t)
  require(core "/home/micah/.emacs.d.doom/core/core")
  eval-buffer(#<buffer  *load*-595926> nil "/home/micah/.emacs.d.doom/init.el" nil t)  ; Reading at buffer position 2680
  load-with-code-conversion("/home/micah/.emacs.d.doom/init.el" "/home/micah/.emacs.d.doom/init.el" nil nil)
  load("/home/micah/.emacs.d.doom/init.el")
  (let* ((emacs-directory (file-name-as-directory (chemacs-emacs-profile-key (quote user-emacs-directory)))) (init-file (expand-file-name "init.el" emacs-directory)) (custom-file- (chemacs-emacs-profile-key (quote custom-file) init-file)) (server-name- (chemacs-emacs-profile-key (quote server-name)))) (setq user-emacs-directory emacs-directory) (if server-name- (progn (setq server-name server-name-))) (mapcar (function (lambda (env) (setenv (car env) (cdr env)))) (chemacs-emacs-profile-key (quote env))) (load init-file) (if (not custom-file) (progn (setq custom-file custom-file-) (load custom-file))))
  chemacs-load-profile("doom")
  (if args (let ((s (split-string (car args) "="))) (cond ((equal (car args) "--with-profile") (add-to-list (quote command-switch-alist) (quote ("--with-profile" lambda (_) (pop command-line-args-left)))) (chemacs-load-profile (car (cdr args)))) ((equal (car s) "--with-profile") (add-to-list (quote command-switch-alist) (cons (car args) (quote (lambda ...)))) (chemacs-load-profile (mapconcat (quote identity) (cdr s) "="))) (t (chemacs-check-command-line-args (cdr args))))) (chemacs-load-profile (chemacs-detect-default-profile)))
  chemacs-check-command-line-args(nil)
  (cond ((equal (car args) "--with-profile") (add-to-list (quote command-switch-alist) (quote ("--with-profile" lambda (_) (pop command-line-args-left)))) (chemacs-load-profile (car (cdr args)))) ((equal (car s) "--with-profile") (add-to-list (quote command-switch-alist) (cons (car args) (quote (lambda (_))))) (chemacs-load-profile (mapconcat (quote identity) (cdr s) "="))) (t (chemacs-check-command-line-args (cdr args))))
  (let ((s (split-string (car args) "="))) (cond ((equal (car args) "--with-profile") (add-to-list (quote command-switch-alist) (quote ("--with-profile" lambda (_) (pop command-line-args-left)))) (chemacs-load-profile (car (cdr args)))) ((equal (car s) "--with-profile") (add-to-list (quote command-switch-alist) (cons (car args) (quote (lambda (_))))) (chemacs-load-profile (mapconcat (quote identity) (cdr s) "="))) (t (chemacs-check-command-line-args (cdr args)))))
  (if args (let ((s (split-string (car args) "="))) (cond ((equal (car args) "--with-profile") (add-to-list (quote command-switch-alist) (quote ("--with-profile" lambda (_) (pop command-line-args-left)))) (chemacs-load-profile (car (cdr args)))) ((equal (car s) "--with-profile") (add-to-list (quote command-switch-alist) (cons (car args) (quote (lambda ...)))) (chemacs-load-profile (mapconcat (quote identity) (cdr s) "="))) (t (chemacs-check-command-line-args (cdr args))))) (chemacs-load-profile (chemacs-detect-default-profile)))
  chemacs-check-command-line-args(("emacs"))
  eval-buffer(#<buffer  *load*> nil "/home/micah/.emacs" nil t)  ; Reading at buffer position 6290
  load-with-code-conversion("/home/micah/.emacs" "/home/micah/.emacs" t t)
  load("~/.emacs" t t)
  #f(compiled-function () #<bytecode 0x1df989>)()
  command-line()
  normal-top-level()

Before this update, the autoloads files were collected in lexicographical order (by traversing straight's build directory). By using straight--build-cache's keys (which are entered in the order they were registered) we avoid issues like

ckruse · 2019-07-29T12:36:04Z

I tried out this branch this morning. There seems to be something wrong with package management: not all packages get loaded. For example I did the following:

(package! move-text)
(use-package! move-text
  :commands (move-text-up move-text-down)
  :bind (([M-up] . move-text-up)
         ([M-down] . move-text-down)))

In the development branch it works like a charm; in the straight branch if have to eval-region the use-package! move-text to get it working. Same with for example this block:

(package! flycheck-mix)
(use-package! flycheck-mix
  :commands flycheck-mix-setup
  :init
  (defvar elixir-enable-compilation-checking nil)

  (add-to-list 'safe-local-variable-values
               (cons 'elixir-enable-compilation-checking nil))
  (add-to-list 'safe-local-variable-values
               (cons 'elixir-enable-compilation-checking t))

  (add-hook 'hack-local-variables-hook
            (lambda ()
              (when (and (or elixir-enable-compilation-checking) (string-equal major-mode "elixir-mode"))
                (flycheck-mix-setup)
                ;; enable credo only if there are no compilation errors
                (when (not (member '(warning . elixir-credo) (flycheck-checker-get 'elixir-mix 'next-checkers)))
                  (flycheck-add-next-checker 'elixir-mix '(warning . elixir-credo))))))

  (add-hook 'flycheck-before-syntax-check-hook (lambda ()
                                                 (when (string-equal major-mode "elixir-mode")
                                                   (setenv "MIX_ENV" "test")))))

This was working when I was using the develop branch, but it doesn't with the straight branch: the flycheck-mix package doesn't get loaded and the elixir-enable-compilation-checking isn't marked as a safe directory-local variable.

Am I missing something?

Edit: it seems as if packages.el doesn't get loaded upon startup. A simple (message "called!") doesn't generate any messages in the *Messages* buffer when starting up?

hlissner · 2019-07-29T13:00:03Z

@ckruse I'm not sure if you combined them for brevity, but just to make sure:

(package! move-text) should go in ~/.doom.d/packages.el
And your use-package! block should go in ~/.doom.d/config.el`

Doom's package management system is declarative. You do not put them in the same file. Your packages.el is not loaded at startup (for interactive sessions) on either branch (until you start using the package management API).

However, in both branches, bin/doom loads them immediately -- which is where the bulk of package management in Doom takes place.

ckruse · 2019-07-29T14:11:03Z

@hlissner with the changes you suggested everything works again, thanks.

Interestingly I've been using this setup on the develop branch for months :-) I believe you when you say that packages.el isn't read on startup on either branch, but something in my configuration must've caused Doom to load the file nonetheless.

I did this because I thought this is how it was meant to do. Sorry for causing unnecessary noise.

hlissner · 2019-07-29T15:05:32Z

@ckruse No sweat! It's possible you're eagerly loading package.el (which Doom spends some effort avoiding), either by requireing it directly or using it's autoloaded API (e.g. the package-installed-p or package-install functions), or you're eager loading a package that does (like magit, for instance).

Once package.el is loaded, Doom pulls in all its package management machinery (and reads all your module packages.el files). There is usually a better way of going about these things, but it's really easy to miss, what with how abysmal the documentation situation in Doom is at the moment. I'm working on it!

hlissner · 2019-08-27T06:32:03Z

The straight branch has now been merged into develop, so I'll consider this request fulfilled.

I recommend current develop users do the following for the smoothest upgrade:

cd ~/.emacs.d
rm -rf .local
git pull --rebase
bin/doom refresh

And current straight users will need to switch to the develop branch before runing doom upgrade:

cd ~/.emacs.d
git checkout develop
bin/doom upgrade

ckruse · 2019-08-27T06:53:08Z

Thank you for your work!

hlissner added the is:feature Adds or requests new features, or extends existing ones label Jan 26, 2018

hlissner added re:packages Pertains to package & dependency management status:backlog labels Apr 2, 2019

bricewge mentioned this issue Jun 5, 2019

can't use a fork of doom-themes #1466

Closed

hlissner removed the status:backlog label Jul 22, 2019

This was referenced Jul 22, 2019

[BUG] Errors after recent update #1566

Closed

A Bug on fresh installation #1582

Closed

hlissner added a commit that referenced this issue Jul 22, 2019

Fix void-variable next error

457b542

And change TARGETS to PACKAGE, which is a more descriptive name. Mentioned in #374

Br1ght0ne mentioned this issue Jul 27, 2019

[BUG] After the last upgrade #1605

Closed

This was referenced Jul 28, 2019

[BUG] Ctrl-K in Ivy history kills buffer #1598

Closed

install emacs-snippets error #812

Closed

This was referenced Aug 5, 2019

[BUG] Failed to load ’org-agenda’ package incrementally, because: (void-function doom-shut-up-a) #1634

Closed

[BUG] Failed to download ‘gnu’ archive. #1651

Closed

hlissner closed this as completed Aug 27, 2019

hlissner added the status:resolved Issue was addressed internally label Aug 27, 2019

github-actions bot locked as resolved and limited conversation to collaborators Feb 1, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Consider using straight as the package manager #374

Consider using straight as the package manager #374

rgrinberg commented Jan 26, 2018

hlissner commented Jan 26, 2018 •

edited

rgrinberg commented Apr 11, 2018

hlissner commented Apr 11, 2018 •

edited

rgrinberg commented Apr 22, 2018

hlissner commented Apr 22, 2018

hlissner commented Jul 31, 2018

hlissner commented Jul 22, 2019 •

edited

credmp commented Jul 22, 2019

ydewit commented Jul 22, 2019

hlissner commented Jul 22, 2019

hlissner commented Jul 22, 2019

ydewit commented Jul 22, 2019

HaoZeke commented Jul 23, 2019

hlissner commented Jul 23, 2019 •

edited

madhat2r commented Jul 23, 2019 •

edited

ckruse commented Jul 29, 2019 •

edited

hlissner commented Jul 29, 2019

ckruse commented Jul 29, 2019

hlissner commented Jul 29, 2019

hlissner commented Aug 27, 2019

ckruse commented Aug 27, 2019

Consider using straight as the package manager #374

Consider using straight as the package manager #374

Comments

rgrinberg commented Jan 26, 2018

hlissner commented Jan 26, 2018 • edited

rgrinberg commented Apr 11, 2018

hlissner commented Apr 11, 2018 • edited

rgrinberg commented Apr 22, 2018

hlissner commented Apr 22, 2018

hlissner commented Jul 31, 2018

hlissner commented Jul 22, 2019 • edited

credmp commented Jul 22, 2019

ydewit commented Jul 22, 2019

hlissner commented Jul 22, 2019

hlissner commented Jul 22, 2019

ydewit commented Jul 22, 2019

HaoZeke commented Jul 23, 2019

hlissner commented Jul 23, 2019 • edited

madhat2r commented Jul 23, 2019 • edited

ckruse commented Jul 29, 2019 • edited

hlissner commented Jul 29, 2019

ckruse commented Jul 29, 2019

hlissner commented Jul 29, 2019

hlissner commented Aug 27, 2019

ckruse commented Aug 27, 2019

hlissner commented Jan 26, 2018 •

edited

hlissner commented Apr 11, 2018 •

edited

hlissner commented Jul 22, 2019 •

edited

hlissner commented Jul 23, 2019 •

edited

madhat2r commented Jul 23, 2019 •

edited

ckruse commented Jul 29, 2019 •

edited