Can't edit packages, doom removes .git history.

[danielkrajnik]

I confirm that...

• I have searched the issue tracker, documentation, FAQ, Discourse, and Google, in case this issue has already been reported/resolved.
• I have read "How to Debug Issues", and will use it to provide as much information about this issue as possible.
• The issue can be reproduced on the latest available commit of Doom.
• The issue can be reproduced on a stable release of Emacs, such as 27, 28, or 29. (Unstable versions end in .50, .60, or .9x)

Expected behavior

Consistently be able to merge upstream updates with your local changes.

Current behavior

At some point local changes to htmlz-mode package break doom upgrade and the package can't be merged anymore. Inspecting git log shows that the previous commits just "disappear", everything except for your local commit. When you then try to git pull:

hint: You have divergent branches and need to specify how to reconcile them.
hint: You can do so by running one of the following commands sometime before
hint: your next pull:
hint:
hint:   git config pull.rebase false  # merge
hint:   git config pull.rebase true   # rebase
hint:   git config pull.ff only       # fast-forward only
hint:
hint: You can replace "git config" with "git config --global" to set a default
hint: preference for all repositories. You can also pass --rebase, --no-rebase,
hint: or --ff-only on the command line to override the configured default per
hint: invocation.
fatal: Need to specify how to reconcile divergent branches.

and doom upgrade :

  > Updating packages (this may take a while)...
    > In repository "htmlz-mode", HEAD on default branch "master" has diverged from "origin/master"

         1) Abort
         2) Merge "origin/master" into "master"
         3) Rebase "master" onto "origin/master"
         4) Reset "master" to "origin/master"
         5) Magit log "master...origin/master" and open recursive edit

      How to proceed? (1, 2, 3, 4, 5)

None of these options works though.

Steps to reproduce

1. add to your packages.el:

elisp
(package! htmlz-mode
  :recipe (:host github :repo "0xekez/htmlz-mode"))

2. doom sync
3. cd $HOME/.config/emacs/.local/straight/repos/htmlz-mode
4. edit htmlz.el
5. add to line 109 ;;;###autoload
6. save and close
7. run in the same directory git commit -am "fix htmlz-mode not showing up on start"
8. run git log to confirm the old upstream and new local commits are saved
9. go about your day. At some point (maybe today, next day, next week) doom upgrade will fail with the above error and you git log will only have your local commit.

System Information

https://pastebin.com/mmuSYT1N

[hlissner]

Please do not ignore the template in the future. I've restored the template in your post. Please fill out the ???s, including steps to reproduce the issue, the output of M-x doom/info (preferably in a gist, rather than dumped directly into the issue), and an explanation of what you're trying to do and how you've tried to pull it off.

Generally, code in .local/straight/repos should not be updated directly, unless you are pinning the package yourself, otherwise a Doom update will overwrite it when their pins are changed upstream -- it is by design that the package manager prioritizes package! statements.

The local package installs are build artifacts, and will become more immutable as Doom nears a proper v3 release. I recommend cloning the package locally and registering that as your working copy (e.g. (package! package :recipe (:local-repo "path/to/local/repo" :build (:not compile))). There will be workflows to make this easier to do/automate this step in the future, but in any case, Doom will never touch packages that live outside of .local/straight.

[danielkrajnik]

Thank you for looking into this (and for keep developing this great project), I've updated the issue template.

should not be updated directly, unless you are pinning the package

There are packages that haven't been touched for a while until one day the maintainer merges the changes. org-wild-notifier has recently added long-standing patch to repeated tasks that I would have never learned about if it wasn't for merge conflict in doom upgrade . Why prevent the user from playing with the upstream packages? It's educational to see a package break after you pull its updates. I thought git was meant to facilitate this type of decentralization.

will become more immutable as Doom nears a proper v3 release

I'd prefer to be able to tinker with some packages and still be able to merge upstream changes. Is this dictated by something from straight.el project? Or is it specific to Doom?

I recommend cloning the package locally and registering that as your working copy

But then you won't be able to receive upstream changes?

[danielkrajnik]

I wonder if it helps if you fetch more git commits (I assume by default these are "shallow copies".

git fetch --depth 20

[hlissner]

I'd prefer to be able to tinker with some packages and still be able to merge upstream changes. Is this dictated by something from straight.el project? Or is it specific to Doom?

Doom's package manager (by design) sees local inconsistencies as something to be corrected so it can ensure a predictable outcome. That they are git repos (or that Doom uses straight) is an implementation detail to that end. Getting it to skirt around local modifications will severely complicate that process. This is Doom's stipulation, not straight's, which has different goals.

That said, though .local/straight will tend toward immutable in v3, there are ways to accommodate that in-place tinkering you're asking for, but as an automated version of what I've already suggested (re-cloning to a staging area and pointing Doom to it). For example: Doom could prompt you to stage a package before writing any changes to files under .local/straight, then silently switch you to the new clone. Add to that M-x doom/package-stage and bin/doom packages stage [PATH|NAME] commands, or a setting to auto-stage packages silently, and the process could be made mostly invisible to the end-user.

As for updates, if all this is systematized, staged packages can at least receive updates (unless otherwise specified) provided there are no merge conflicts. Post-v3 doom upgrade will be atomic, so it will roll back at the first sign of critical failure (like a merge conflict).

In any case, these are v3.0-v3.2 plans, so it must be handled manually for the time being.

But then you won't be able to receive upstream changes?

With (package! package :recipe (:local-repo "path/to/local/repo" :build (:not compile)), no. You'd have to pull changes yourself with git (or magit). If it's not in .local/straight, it is outside of Doom's jurisdiction.

(I assume by default these are "shallow copies".

They are shallow clones by design (for savings on download time and disk space, mostly). You can disable this across the board with:

;; Add to $DOOMDIR/init.el
(setq straight-vc-git-default-clone-depth 'full)

Or selectively unshallow a repo with:

;; Add to $DOOMDIR/packages.el
(package! X :recipe (:depth 'full))

Or by running git fetch --unshallow in the repo.

Just remember to avoid running doom purge or doom sync -p, as those will regraft + compact repos under .local/straight/repos (essentially re-shallowing them).

[danielkrajnik]

(package! X :recipe (:depth 'full))

(package! htmlz-mode
  :recipe (:host github :repo "0xekez/htmlz-mode" :depth 'full))

Sorry, but the above command doesn't work?

doom sync

~ ❯ doom sync
                     15:04:15
> Synchronizing "default" profile...
  > Regenerating envvars file
    ✓ Generated ~/.config/emacs/.local/env
  x There was an unexpected runtime error
    Message: Wrong type argument
    Details: (stringp nil)
    Backtrace:
      (string-match "^.*/\\(.+\\)\\.git$" nil)
      (if (string-match regexp repo) (progn (match-string 1 repo)))
      (let ((regexp "^.*/\\(.+\\)\\.git$")) (if (string-match regexp repo) (progn (match-string 1 repo))))
      (if host (replace-regexp-in-string "^.+/" "" repo) (let ((regexp "^.*/\\(.+\\)\\.git$")) (if (string-match regexp repo) (pr...
      (let* ((--cl-rest-- recipe) (repo (car (cdr (plist-member --cl-rest-- ':repo)))) (host (car (cdr (plist-member --cl-rest-- ...
      (straight-vc-git-local-repo-name (:depth 'full :package "htmlz-mode" :type git))
      (apply straight-vc-git-local-repo-name (:depth 'full :package "htmlz-mode" :type git))
      (let ((func (intern (format "straight-vc-%S-%S" type method)))) (if (fboundp func) nil (let ((regexp (format "^straight-vc-...
      (straight-vc local-repo-name git (:depth 'full :package "htmlz-mode" :type git))
      (let* ((--cl-rest-- recipe) (type (car (cdr (plist-member --cl-rest-- ':type))))) (straight-vc 'local-repo-name type recipe))
      (straight-vc-local-repo-name (:depth 'full :package "htmlz-mode" :type git))
      (or (straight-vc-local-repo-name plist) package)
    ! Wrote extended backtrace to ~/.config/emacs/.local/state/logs/cli.doom.230914150526.4100145.error
  ! Script was abruptly aborted, leaving Doom in an incomplete state!
  - Run 'doom sync' to repair it.

[danielkrajnik]

Doom's package manager (by design) sees local inconsistencies as something to be corrected

How about allowing commits with a pre-agreed comment/tag "doom-keep" that would be preserved by doom and reset everything else? Sorry, but I think that it's really important to find a solution that would work for both the maintainers and the users. I think that users should be allowed to modify the packages, which are often poorly maintained where patches may be merged in a week or years time.

I've just run another doom upgrade to confirm that doom indeed nukes all the git log in htmlz-mode package even if you run git fetch --depth 20 prior.

Doom could prompt you to stage a package before writing any changes to files under .local/straight, then silently switch you to the new clone.

Sorry, but this sounds really complicated and if there is a possibility to just agree on a specific tag/comment prefix (e.g. "doom-keep") I would 1000 times prefer that.

[hpfr]

Based on what you’ve said so far, it seems like you really want to modify things in place for some reason, but IMO life is a lot simpler when you let straight.el manage itself and use :local-repo with an absolute path as Henrik suggested.

But then you won't be able to receive upstream changes?

I don’t really get this. If you’re making local modifications to a package, you should want pulling to be manual.

Why prevent the user from playing with the upstream packages? It's educational to see a package break after you pull its updates.

You’re not prevented. Doom makes it extremely ergonomic to override a package with a local repo and to add it as a project. Magit makes it extremely ergonomic to clone, fetch, and pull. From there, you can rebase or merge and educate yourself to your heart’s content!

I thought git was meant to facilitate this type of decentralization.

git does facilitate this type of decentralization. It facilitates it so well that its default workflow is for every developer to have their own clone of the project, and each clone is on equal footing with every other clone. In this case, straight.el is another developer. Why would you want to share a clone with them? They’re kinda cantankerous. You could even add their clone as a remote to your local clone if you wanted to keep track of the different states, although while you have :local-repo enabled straight.el won’t be updating its clone. But it could be nice for flipping back and forth between upstream and local.

Basically, I would really recommend rethinking why you care so much about modifying the straight.el checkout, because it creates headaches that are easy to avoid, for dubious benefit.

[danielkrajnik]

it seems like you really want to modify things in place for some reason

I was just thinking about this issue when I couldn't again call htmlz-mode. No kidding, I was just about to do some web development when I remembered that doom removed the change I've made for the tenth time. How convenient you mention it just now.

To be clear, straight.el doesn't prevent you from adding local changes. This is something doom added without warning the users.

Doom makes it extremely ergonomic to override a package with a local repo and to add it as a project

Is this documented anywhere? I'd be willing to give it a try if there was a guide. I assume that if you know how to do it and work full-time as a developer it's obvious, but as a regular user I don't see why is this supposed to be easier than just changing existing packages.

because it creates headaches that are easy to avoid

Sorry, but it seem to me that you are just offloading the headache onto the users. I realize that a project like doom is a huge burden to maintain, but let's call things by their name.

[hpfr]

To be clear, straight.el doesn't prevent you from adding local changes. This is something doom added without warning the users.

Well yeah, it’s a starter kit that’s supposed to make upgrading packages in bulk easy for the majority of users. If you personally want to use raw straight.el, why not just do that?

Is this documented anywhere? I'd be willing to give it a try if there was a guide. I assume that if you know how to do it and work full-time as a developer it's obvious, but as a regular user I don't see why is this supposed to be easier than just changing existing packages.

It’s easier because you don’t have to compete with doom upgrade. This is straying away from a GitHub issue and towards a Discourse or Discord help request, but sure

1. magit-clone htmlz-mode. SPC g C if you use Evil
2. When the clone finishes and the Magit status buffer appears, projectile-add-known-project (SPC p a)
3. Make your modifications and magit-commit
4. Add a (package! package :recipe (:local-repo "/path/to/my/htmlz-mode" :build (:not compile)) directive to your packages.el
5. doom sync -u
6. Restart Emacs

You now have a local checkout of htmlz-mode that won’t be touched by straight. When you want to update it:

1. projectile-switch-project (SPC p p) to your checkout
2. Pull up the magit status buffer and fetch. You can figure it out :) M-x where-is, any command with help in the name, and M-x info will guide you
3. Use magit to merge or rebase

You can then restart emacs or just eval-buffer in htmlz.el to reload the package.

I’m sorry, but I can’t provide further clarification of these instructions here. You can see how this approach won’t scale to all of Doom’s users. And it goes without saying that there are multiple ways to accomplish this, I just picked one. Emacs is complicated; it will take some time to learn. You can do it!

Sorry, but it seem to me that you are just offloading the headache onto the users. I realize that a project like doom is a huge burden to maintain, but let's call things by their name.

I’m sorry it isn’t working out for you. There are tradeoffs to every approach, and biasing the Doom CLI towards seamlessly upgrading packages with package! for overrides makes sense to me at least. If you take some more time with it, it may become more clear.

Also, if it’s just an autoload, you can just (use-package! htmlz-mode :commands my-command). Take some time to read the help for use-package, it will pay off

[danielkrajnik]

Thank you very much, that is very helpful. This step-by-step guides are the best and I wish we had more of them.

One more question regarding this process - where do you usually save them? Do you have a separate folder in .config/emacs?

if it’s just an autoload, you can just (use-package! htmlz-mode :commands my-command)

If that would fix lack of an ;;;autoload header that would be great, but sadly adding (use-package! htmlz-mode :commands htmlz-mode) resulted in

Cannot open load file: No such file or directory, htmlz-mode

error. Opening the htmlz.el file though, selecting everything and running eval-region makes it work. Really not sure what to make out of this. Do you think use-pacakage! can fix this? This package would be really really useful for web-development tasks.

I’m sorry, but I can’t provide further clarification of these instructions here.

Not a problem, at all. These instructions clarify it really well. Nonetheless, this still adds many more steps compared to just:

1. adding changes to an already cloned package in .config/emacs
2. running git commit -am "change added".

no need to worry about new cron jobs in this scenario. If merge conflicts happen they will appear in doom upgrade (or at least I thought they would appear).

You can see how this approach won’t scale to all of Doom’s users

That's why I thought that you can just edit packages in-place. That would be more scaleable I think although I don't know how much more burden it would add to the maintenance of the project. I really hope that blocking users from editing packages was worth it. If you can't easily edit them, then you won't bother to do it, unless something really really important comes up.

I will definitely try it for this one package, because it's really important, but most emacs problems are not really important, mostly they are just annoying nuisances and compared to the effort required of maintaining separate forks for each small change they are usually not worth the effort.

If you personally want to use raw straight.el, why not just do that?

I very much want to stop using doom, but migrating everything will take at least a week. Don't mean to say that it isn't a fantastic project, but the amount of times doom did things in ways I didn't expect led to a huge amount of frustration.