Chezmoi "add" adds/deletes subdirectories corresponding to submodules

[detly]

Describe the bug

Given a subdirectory that's a git submodule prefixed with external_, chezmoi apply will clone it into the correct place, but running chezmoi add on the parent (target) directory will attempt to (a) add the cloned submodule without the external_ attribute and (b) delete the submodule.

I was surprised by this because the directory corresponding to the submodule was created by Chezmoi, and no changes were made to it. I'd expect apply immediately followed by add to (generally) not have anything to do.

To reproduce

• create a submodule within Chezmoi's source directory, prefixed with external_
• run chezmoi apply
• run chezmoi add <target directory housing the submodule>

Expected behavior

No change in source state.

Output of command with the --verbose flag

$ git status # In Chezmoi source dir
HEAD detached at 7392844
Changes not staged for commit:
  (use "git add/rm <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
	deleted:    normal-dir/external_repro-submodule
Untracked files:
  (use "git add <file>..." to include in what will be committed)
	normal-dir/repro-submodule/
no changes added to commit (use "git add" and/or "git commit -a")

Output of chezmoi doctor

$ chezmoi doctor
RESULT    CHECK                       MESSAGE
warning   version                     v2.42.2, built at 2023-12-06T22:39:17Z, built by Alpine Linux
warning   latest-version              v2.46.0
ok        os-arch                     linux/amd64 (Alpine Linux)
ok        uname                       Linux runner-zxwgkjap-project-54387395-concurrent-0 5.4.109+ #1 SMP Wed Jun 16 20:00:10 PDT 2021 x86_64 Linux
ok        go-version                  go1.21.4 (gc)
ok        executable                  /usr/bin/chezmoi
ok        upgrade-method              upgrade-package
ok        config-file                 no config file found
warning   source-dir                  /builds/detly/chezmoi-externals-repro is a git working tree (dirty)
ok        suspicious-entries          no suspicious entries
warning   working-tree                /builds/detly/chezmoi-externals-repro is a git working tree (dirty)
ok        dest-dir                    ~ is a directory
ok        umask                       022
ok        cd-command                  found /bin/ash
ok        cd-args                     /bin/ash
info      diff-command                not set
ok        edit-command                found /usr/bin/vi
ok        edit-args                   /usr/bin/vi
ok        git-command                 found /usr/bin/git, version 2.43.0
warning   merge-command               vimdiff not found in $PATH
ok        shell-command               found /bin/ash
ok        shell-args                  /bin/ash
info      age-command                 age not found in $PATH
info      gpg-command                 gpg not found in $PATH
info      pinentry-command            not set
info      1password-command           op not found in $PATH
info      bitwarden-command           bw not found in $PATH
info      bitwarden-secrets-command   bws not found in $PATH
info      dashlane-command            dcli not found in $PATH
info      doppler-command             doppler not found in $PATH
info      gopass-command              gopass not found in $PATH
info      keepassxc-command           keepassxc-cli not found in $PATH
info      keepassxc-db                not set
info      keeper-command              keeper not found in $PATH
info      lastpass-command            lpass not found in $PATH
info      pass-command                pass not found in $PATH
info      passhole-command            ph not found in $PATH
info      rbw-command                 rbw not found in $PATH
info      vault-command               vault not found in $PATH
info      vlt-command                 vlt not found in $PATH
info      secret-command              not set

Additional context

There is a git repo that reproduces this issue here, with a CI job to run the steps and show the output.

[twpayne]

Thank you very much for reporting this and particularly for your work in creating a test case that perfectly reproduces the problem. Very much appreciated!

The root cause of this is similar to #1574 and #3559, namely that chezmoi add is currently buggy when handling adding entries that come from externals.

#3581 should fix this.

[twpayne]

Note to self for future: I don't think chezmoi can reliably add entries in external_ directories. Consider the following cases:

$ chezmoi add normal-dir/repro-submodule/.gitignore

Should this overwrite the submodule's .gitignore?

$ chezmoi add normal-dir/repro-submodule/.git

Should this overwrite the submodule's .git directory?

This becomes even more complicated if you consider subdirectories managed by other version control systems (Mercurial, Darcs, Fossil, etc.).

So, I think that chezmoi add returning an error when the add target is any entry in an external_ directory is the only sane thing to do.