Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Destructive operation on submodules with submodule.recurse = true git setting #2063

Closed
a666 opened this issue Sep 27, 2021 · 10 comments · Fixed by #2071
Closed

Destructive operation on submodules with submodule.recurse = true git setting #2063

a666 opened this issue Sep 27, 2021 · 10 comments · Fixed by #2071
Labels

Comments

@a666
Copy link

@a666 a666 commented Sep 27, 2021

describe your issue

I added pre-commit to an ansible playbook repository that does extensive usage of git submodules for roles.

I had two submodules with extensive changes:

$ gst
On branch master
Your branch is ahead of 'origin/master' by 3 commits.
  (use "git push" to publish your local commits)
...
        modified:   roles/dnscrypt-proxy (modified content) <-- submodule with extensive (non-commited) changes
        modified:   roles/syncthing (modified content) <-- another submodule with extensive (non-commited) changes
...

Upon doing a commit I received an error (check ~/.cache/pre-commit/pre-commit.log section) due to having some problems in my repo and its submodules.
Notice that I cold not even run git status afterwards:

$ gst
fatal: not a git repository: roles/matomo/../../.git/modules/roles/matomo

After fixing the problems:

$ rm -rf roles/matomo/ .git/modules/roles/matomo/
$ git submodule update
Cloning into '/home/apoc/w/posg-ops/roles/matomo'...
Submodule path 'roles/matomo': checked out '######'
fatal: could not get a repository handle for submodule 'roles/php'
$ rm -rf roles/php/ .git/modules/roles/php
$ git submodule update
Cloning into '/home/apoc/w/posg-ops/roles/php'...
Submodule path 'roles/php': checked out '######'

and running git apply to have my changes back:

$ git apply /home/a666/.cache/pre-commit/patch1632709290-1076629

I noticed that my two submodules now had no changes in them...
And unfortunately I see nothing in pre-commit cache area to help me recover them.

pre-commit --version

pre-commit 2.15.0

.pre-commit-config.yaml

---
repos:
  - repo: https://github.com/pre-commit/pre-commit-hooks
    rev: v4.0.1
    hooks:
      - id: trailing-whitespace
      - id: end-of-file-fixer
      - id: check-yaml
      - id: check-toml
      - id: check-added-large-files

  - repo: https://github.com/editorconfig-checker/editorconfig-checker.python
    rev: 2.3.54
    hooks:
      - id: editorconfig-checker

  - repo: https://github.com/antonbabenko/pre-commit-terraform
    rev: v1.51.0
    hooks:
      - id: terraform_validate
      - id: terraform_fmt
      - id: checkov
      - id: terrascan
      - id: terraform_tfsec

~/.cache/pre-commit/pre-commit.log (if present)

version information

pre-commit version: 2.15.0
sys.version:
    3.9.7 (default, Aug 31 2021, 13:28:12)
    [GCC 11.1.0]
sys.executable: /home/a666/.local/pipx/venvs/pre-commit/bin/python
os.name: posix
sys.platform: linux

error information

An unexpected error has occurred: CalledProcessError: command: ('/usr/lib/git-core/git', 'checkout', '--', '.')
return code: 255
expected return code: 0
stdout: (none)
stderr:
    Migrating git directory of 'roles/fail2ban' from
    '/home/a666/ops/roles/fail2ban/.git' to
    '/home/a666/ops/.git/modules/roles/fail2ban'
    fatal: not a git repository: ../../.git/modules/roles/matomo
    error: Submodule 'roles/matomo' could not be updated.
    Migrating git directory of 'roles/openproject' from
    '/home/a666/ops/roles/openproject/.git' to
    '/home/a666/ops/.git/modules/roles/openproject'
    Migrating git directory of 'roles/oryhydra' from
    '/home/a666/ops/roles/oryhydra/.git' to
    '/home/a666/ops/.git/modules/roles/oryhydra'
    fatal: not a git repository: ../../.git/modules/roles/php
    error: Submodule 'roles/php' could not be updated.

Traceback (most recent call last):
  File "/home/a666/.local/pipx/venvs/pre-commit/lib/python3.9/site-packages/pre_commit/error_handler.py", line 65, in error_handler
    yield
  File "/home/a666/.local/pipx/venvs/pre-commit/lib/python3.9/site-packages/pre_commit/main.py", line 368, in main
    return hook_impl(
  File "/home/a666/.local/pipx/venvs/pre-commit/lib/python3.9/site-packages/pre_commit/commands/hook_impl.py", line 237, in hook_impl
    return retv | run(config, store, ns)
  File "/home/a666/.local/pipx/venvs/pre-commit/lib/python3.9/site-packages/pre_commit/commands/run.py", line 399, in run
    exit_stack.enter_context(staged_files_only(store.directory))
  File "/usr/lib/python3.9/contextlib.py", line 448, in enter_context
    result = _cm_type.__enter__(cm)
  File "/usr/lib/python3.9/contextlib.py", line 119, in __enter__
    return next(self.gen)
  File "/home/a666/.local/pipx/venvs/pre-commit/lib/python3.9/site-packages/pre_commit/staged_files_only.py", line 92, in staged_files_only
    with _intent_to_add_cleared(), _unstaged_changes_cleared(patch_dir):
  File "/usr/lib/python3.9/contextlib.py", line 119, in __enter__
    return next(self.gen)
  File "/home/a666/.local/pipx/venvs/pre-commit/lib/python3.9/site-packages/pre_commit/staged_files_only.py", line 61, in _unstaged_changes_cleared
    cmd_output_b('git', 'checkout', '--', '.', env=no_checkout_env)
  File "/home/a666/.local/pipx/venvs/pre-commit/lib/python3.9/site-packages/pre_commit/util.py", line 154, in cmd_output_b
    raise CalledProcessError(returncode, cmd, retcode, stdout_b, stderr_b)
pre_commit.util.CalledProcessError: command: ('/usr/lib/git-core/git', 'checkout', '--', '.')
return code: 255
expected return code: 0
stdout: (none)
stderr:
    Migrating git directory of 'roles/fail2ban' from
    '/home/a666/ops/roles/fail2ban/.git' to
    '/home/a666/ops/.git/modules/roles/fail2ban'
    fatal: not a git repository: ../../.git/modules/roles/matomo
    error: Submodule 'roles/matomo' could not be updated.
    Migrating git directory of 'roles/openproject' from
    '/home/a666/ops/roles/openproject/.git' to
    '/home/a666/ops/.git/modules/roles/openproject'
    Migrating git directory of 'roles/oryhydra' from
    '/home/a666/ops/roles/oryhydra/.git' to
    '/home/a666/ops/.git/modules/roles/oryhydra'
    fatal: not a git repository: ../../.git/modules/roles/php
    error: Submodule 'roles/php' could not be updated.
@asottile
Copy link
Member

@asottile asottile commented Sep 27, 2021

there's a test in our testsuite which shows this works correctly -- do you have more information on how to reproduce this?

looking at your output it looks like your submodules may have already been broken since git was unable to clean them manage them properly?

your report is also incomplete -- you have not shared the command you ran and its output so I cannot help you further without that

@asottile
Copy link
Member

@asottile asottile commented Sep 27, 2021

also, do you have any post-checkout hooks installed?

@a666
Copy link
Author

@a666 a666 commented Sep 27, 2021

looking at your output it looks like your submodules may have already been broken since git was unable to clean them manage them properly?

There was definitively a problem with them, but the changes to these submodules date from months back, there's around 20 commits following those changes.

your report is also incomplete -- you have not shared the command you ran and its output so I cannot help you further without that

From the history, it was a simple git commit -v.
Unfortunately I no longer have the pane history due to machine reboot, the error displayed was very similar to the error information section, sorry about that.

also, do you have any post-checkout hooks installed?

None.

@asottile
Copy link
Member

@asottile asottile commented Sep 27, 2021

without a reproduction there isn't really much I can fix here :( -- the operations run are supposed to be safe (especially because --recurse-submodules is not passed) so I can only guess there's some bug with git itself that caused your issue

@asottile
Copy link
Member

@asottile asottile commented Sep 28, 2021

ahah -- is it possible you have this setting enabled? https://git-scm.com/docs/git-config#Documentation/git-config.txt-submodulerecurse

let me see if that's what can cause this ?

@a666
Copy link
Author

@a666 a666 commented Sep 28, 2021

Well, apparently the last sysadmin decided to put it in the config, and I ended getting it too after rsyncking the repo from his account...

$ git config --list --show-origin | rg recurse
file:.git/config        submodule.recurse=true

@asottile
Copy link
Member

@asottile asottile commented Sep 28, 2021

rough! well I'll at least be able to fix that by ignoring that setting for pre-commit's programmatic uses of git

@asottile asottile added bug and removed question labels Sep 28, 2021
@asottile asottile changed the title Destructive operation on submodules Destructive operation on submodules with submodule.recurse = true git setting Sep 28, 2021
@a666
Copy link
Author

@a666 a666 commented Sep 28, 2021

So if I understand well, having submodule.recurse=true made pre-commit run a git checkout . inside the submodules?

@asottile
Copy link
Member

@asottile asottile commented Sep 28, 2021

submodule.recurse = 1 makes git checkout -- . also run git checkout -- . in each submodule -- seems like a really bad git feature

@a666
Copy link
Author

@a666 a666 commented Sep 28, 2021

Well, submodules were always like a weird git workaround to a difficult problem (I think) and not very though out...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Linked pull requests

Successfully merging a pull request may close this issue.

2 participants