Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

unexpected error - b'worktreeConfig' #10395

Closed
ereli-sevenai opened this issue Apr 22, 2024 · 6 comments · Fixed by #10406
Closed

unexpected error - b'worktreeConfig' #10395

ereli-sevenai opened this issue Apr 22, 2024 · 6 comments · Fixed by #10406
Assignees
@dberenbaum @skshetry
Labels
bug Did we break something? p1-important Important, aka current backlog of things to do

Comments

@ereli-sevenai
Copy link

Bug Report

Issue name

pull: unexpected error - b'worktreeConfig'

Description

When attempting to use the dvc pull command in a repository with the worktreeConfig extension enabled in Git, DVC throws an unexpected error. The error message received is "ERROR: unexpected error - b'worktreeConfig'". This issue persists even after attempting to set worktreeConfig to false via the Git config.

Reproduce

  1. Initialize a Git repository with git init.
  2. Enable worktreeConfig in the Git configuration with git config extensions.worktreeConfig false.
  3. Initialize DVC within the repository with dvc init.
  4. Add a dataset or file to manage with DVC using dvc add file.
  5. Try to pull data or synchronize with a remote storage using dvc pull.

Expected

The dvc pull command should execute without any errors, syncing the DVC-tracked files from the remote storage. The presence of the worktreeConfig setting in Git should not impact DVC operations.
@shcheklein
Copy link
Member

I can reproduce this.

Full stack trace:

Traceback (most recent call last):
  File "/Users/ivan/Projects/dvc/dvc/cli/__init__.py", line 210, in main
    cmd = args.func(args)
  File "/Users/ivan/Projects/dvc/dvc/cli/command.py", line 21, in __init__
    self.repo: "Repo" = Repo(uninitialized=self.UNINITIALIZED)
  File "/Users/ivan/Projects/dvc/dvc/repo/__init__.py", line 218, in __init__
    self._ignore()
  File "/Users/ivan/Projects/dvc/dvc/repo/__init__.py", line 457, in _ignore
    self.scm_context.ignore(file)
  File "/Users/ivan/Projects/dvc/.venv/lib/python3.9/site-packages/funcy/objects.py", line 25, in __get__
    res = instance.__dict__[self.fget.__name__] = self.fget(instance)
  File "/Users/ivan/Projects/dvc/dvc/repo/__init__.py", line 320, in scm_context
    return SCMContext(self.scm, self.config)
  File "/Users/ivan/Projects/dvc/.venv/lib/python3.9/site-packages/funcy/objects.py", line 25, in __get__
    res = instance.__dict__[self.fget.__name__] = self.fget(instance)
  File "/Users/ivan/Projects/dvc/dvc/repo/__init__.py", line 308, in scm
    return SCM(self.root_dir, no_scm=no_scm)
  File "/Users/ivan/Projects/dvc/dvc/scm.py", line 111, in SCM
    return Git(root_dir, search_parent_directories=search_parent_directories)
  File "/Users/ivan/Projects/dvc/.venv/lib/python3.9/site-packages/scmrepo/git/__init__.py", line 106, in __init__
    first_ = first(self.backends.values())
  File "/Users/ivan/Projects/dvc/.venv/lib/python3.9/site-packages/funcy/seqs.py", line 63, in first
    return next(iter(seq), None)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/_collections_abc.py", line 871, in __iter__
    yield self._mapping[key]
  File "/Users/ivan/Projects/dvc/.venv/lib/python3.9/site-packages/scmrepo/git/__init__.py", line 63, in __getitem__
    initialized = backend(*self.args, **self.kwargs)
  File "/Users/ivan/Projects/dvc/.venv/lib/python3.9/site-packages/scmrepo/git/backend/dulwich/__init__.py", line 191, in __init__
    self.repo = Repo.discover(start=root_dir)
  File "/Users/ivan/Projects/dvc/.venv/lib/python3.9/site-packages/dulwich/repo.py", line 1263, in discover
    return cls(path)
  File "/Users/ivan/Projects/dvc/.venv/lib/python3.9/site-packages/dulwich/repo.py", line 1195, in __init__
    raise UnsupportedExtension(extension)
dulwich.repo.UnsupportedExtension: b'worktreeConfig'

Traceback (most recent call last):
  File "/Users/ivan/Projects/dvc/dvc/cli/__init__.py", line 210, in main
    cmd = args.func(args)
  File "/Users/ivan/Projects/dvc/dvc/cli/command.py", line 21, in __init__
    self.repo: "Repo" = Repo(uninitialized=self.UNINITIALIZED)
  File "/Users/ivan/Projects/dvc/dvc/repo/__init__.py", line 218, in __init__
    self._ignore()
  File "/Users/ivan/Projects/dvc/dvc/repo/__init__.py", line 457, in _ignore
    self.scm_context.ignore(file)
  File "/Users/ivan/Projects/dvc/.venv/lib/python3.9/site-packages/funcy/objects.py", line 25, in __get__
    res = instance.__dict__[self.fget.__name__] = self.fget(instance)
  File "/Users/ivan/Projects/dvc/dvc/repo/__init__.py", line 320, in scm_context
    return SCMContext(self.scm, self.config)
  File "/Users/ivan/Projects/dvc/.venv/lib/python3.9/site-packages/funcy/objects.py", line 25, in __get__
    res = instance.__dict__[self.fget.__name__] = self.fget(instance)
  File "/Users/ivan/Projects/dvc/dvc/repo/__init__.py", line 308, in scm
    return SCM(self.root_dir, no_scm=no_scm)
  File "/Users/ivan/Projects/dvc/dvc/scm.py", line 111, in SCM
    return Git(root_dir, search_parent_directories=search_parent_directories)
  File "/Users/ivan/Projects/dvc/.venv/lib/python3.9/site-packages/scmrepo/git/__init__.py", line 106, in __init__
    first_ = first(self.backends.values())
  File "/Users/ivan/Projects/dvc/.venv/lib/python3.9/site-packages/funcy/seqs.py", line 63, in first
    return next(iter(seq), None)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/_collections_abc.py", line 871, in __iter__
    yield self._mapping[key]
  File "/Users/ivan/Projects/dvc/.venv/lib/python3.9/site-packages/scmrepo/git/__init__.py", line 63, in __getitem__
    initialized = backend(*self.args, **self.kwargs)
  File "/Users/ivan/Projects/dvc/.venv/lib/python3.9/site-packages/scmrepo/git/backend/dulwich/__init__.py", line 191, in __init__
    self.repo = Repo.discover(start=root_dir)
  File "/Users/ivan/Projects/dvc/.venv/lib/python3.9/site-packages/dulwich/repo.py", line 1263, in discover
    return cls(path)
  File "/Users/ivan/Projects/dvc/.venv/lib/python3.9/site-packages/dulwich/repo.py", line 1195, in __init__
    raise UnsupportedExtension(extension)
dulwich.repo.UnsupportedExtension: b'worktreeConfig'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/ivan/Projects/dvc/.venv/bin/dvc", line 8, in <module>
    sys.exit(main())
  File "/Users/ivan/Projects/dvc/dvc/cli/__init__.py", line 236, in main
    ret = _log_exceptions(exc) or 255
  File "/Users/ivan/Projects/dvc/dvc/cli/__init__.py", line 147, in _log_exceptions
    _log_unknown_exceptions()
  File "/Users/ivan/Projects/dvc/dvc/cli/__init__.py", line 49, in _log_unknown_exceptions
    logger.debug("Version info for developers:\n%s", get_dvc_info())
  File "/Users/ivan/Projects/dvc/dvc/info.py", line 38, in get_dvc_info
    with Repo() as repo:
  File "/Users/ivan/Projects/dvc/dvc/repo/__init__.py", line 218, in __init__
    self._ignore()
  File "/Users/ivan/Projects/dvc/dvc/repo/__init__.py", line 457, in _ignore
    self.scm_context.ignore(file)
  File "/Users/ivan/Projects/dvc/.venv/lib/python3.9/site-packages/funcy/objects.py", line 25, in __get__
    res = instance.__dict__[self.fget.__name__] = self.fget(instance)
  File "/Users/ivan/Projects/dvc/dvc/repo/__init__.py", line 320, in scm_context
    return SCMContext(self.scm, self.config)
  File "/Users/ivan/Projects/dvc/.venv/lib/python3.9/site-packages/funcy/objects.py", line 25, in __get__
    res = instance.__dict__[self.fget.__name__] = self.fget(instance)
  File "/Users/ivan/Projects/dvc/dvc/repo/__init__.py", line 308, in scm
    return SCM(self.root_dir, no_scm=no_scm)
  File "/Users/ivan/Projects/dvc/dvc/scm.py", line 111, in SCM
    return Git(root_dir, search_parent_directories=search_parent_directories)
  File "/Users/ivan/Projects/dvc/.venv/lib/python3.9/site-packages/scmrepo/git/__init__.py", line 106, in __init__
    first_ = first(self.backends.values())
  File "/Users/ivan/Projects/dvc/.venv/lib/python3.9/site-packages/funcy/seqs.py", line 63, in first
    return next(iter(seq), None)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/_collections_abc.py", line 871, in __iter__
    yield self._mapping[key]
  File "/Users/ivan/Projects/dvc/.venv/lib/python3.9/site-packages/scmrepo/git/__init__.py", line 63, in __getitem__
    initialized = backend(*self.args, **self.kwargs)
  File "/Users/ivan/Projects/dvc/.venv/lib/python3.9/site-packages/scmrepo/git/backend/dulwich/__init__.py", line 191, in __init__
    self.repo = Repo.discover(start=root_dir)
  File "/Users/ivan/Projects/dvc/.venv/lib/python3.9/site-packages/dulwich/repo.py", line 1263, in discover
    return cls(path)
  File "/Users/ivan/Projects/dvc/.venv/lib/python3.9/site-packages/dulwich/repo.py", line 1195, in __init__
    raise UnsupportedExtension(extension)
dulwich.repo.UnsupportedExtension: b'worktreeConfig'

Also, CI is failing with this. Probably related to some libraries update. Not sure, but seems a p0 to research at least.

@shcheklein shcheklein added p0-critical Critical issue. Needs to be fixed ASAP. bug Did we break something? labels Apr 22, 2024
@DimaAmega
Copy link

DimaAmega commented Apr 22, 2024 •
edited
Loading

Yes, we also get an error in CI
https://github.com/evidentlyai/evidently/actions/runs/8790124101/job/24121478623
Corresponding workflow job

@shcheklein
Copy link
Member

@DimaAmega yep, it seems tbh that Github has changed something with the way they setup git on CI. Again, sorry about that. We are looking into this.

@0x2b3bfa0
Copy link
Member

@shcheklein
Copy link
Member

Temporary workaround on CI:

      - name: Temporary workaround for GitVersion
        shell: bash
        run: |
          git config --unset-all extensions.worktreeconfig
          # See https://github.com/GitTools/actions/issues/1115

(borrowed from one the linked PRs in the Dulwich issue)

shcheklein added a commit to iterative/scmrepo that referenced this issue Apr 23, 2024
@shcheklein
fix(deps): use the latest dulwich
c532b12 
Related to this iterative/dvc#10395
@shcheklein shcheklein mentioned this issue Apr 23, 2024
Merged
kristjansuligoj added a commit to kristjansuligoj/iis-vaja that referenced this issue Apr 24, 2024
@kristjansuligoj
Update fetch_process_merge_mbajk_weather_data.yml
28f812e 
Added temporary fix to this issue iterative/dvc#10395
dberenbaum pushed a commit to iterative/scmrepo that referenced this issue Apr 24, 2024
@shcheklein
fix(deps): use the latest dulwich (#353)
a5d93c1 
* fix(deps): use the latest dulwich

Related to this iterative/dvc#10395

* vendor in some test helpers from dulwich
@shcheklein
Copy link
Member

This is fixed upstream. A new fixed version of scmrepo and dulwich were published. @skshetry let me know if we should in this case bump the scmrepo on the DVC end.

@shcheklein shcheklein added p1-important Important, aka current backlog of things to do and removed p0-critical Critical issue. Needs to be fixed ASAP. labels Apr 25, 2024
@skshetry skshetry mentioned this issue Apr 25, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@dberenbaum dberenbaum

@skshetry skshetry

Labels
bug Did we break something? p1-important Important, aka current backlog of things to do
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

deps: bump scmrepo minver to >=3.3.2,<4 iterative/dvc
6 participants
@dberenbaum @shcheklein @0x2b3bfa0 @skshetry @DimaAmega @ereli-sevenai