exp push: breaking in GHA

[dberenbaum]

Bug Report

Description

dvc exp push --rev HEAD is breaking in GitHub Actions. The log is in https://github.com/iterative/lstm_seq2seq/actions/runs/4559164229/jobs/8042811435#step:7:4777.

Reproduce

Try to run the action in https://github.com/iterative/lstm_seq2seq/actions/workflows/cml.yaml from the debug branch (to get verbose output).

Note that I could not reproduce it locally, even from a shallow clone or detached head.

[daavoo]

I am currently debugging as I am not sure if it is something different, but exp push is also failing for me inside GitHub Codespaces

[daavoo]

This is what I get inside the codespace (git push from CLI works):

`dvc exp remove -A -g origin`

$ dvc exp remove -A -g origin -vv
2023-04-04 12:26:08,432 DEBUG: v2.52.1.dev12+gcf52efb8, CPython 3.10.10 on Linux-5.4.0-1105-azure-x86_64-with-glibc2.31
2023-04-04 12:26:08,432 DEBUG: command: /workspaces/example-repos-dev/example-get-started-experiments/build/example-get-started-experiments/.venv/bin/dvc exp remove -A -g origin -vv
2023-04-04 12:26:08,433 TRACE: Namespace(cprofile=False, yappi=False, yappi_separate_threads=False, viztracer=False, viztracer_depth=None, viztracer_async=False, cprofile_dump=None, pdb=False, instrument=False, instrument_open=False, show_stack=False, quiet=0, verbose=2, cd='.', cmd='remove', all_commits=True, rev=None, num=1, queue=False, git_remote='origin', experiment=[], func=<class 'dvc.commands.experiments.remove.CmdExperimentsRemove'>, parser=DvcParser(prog='dvc', usage=None, description='Data Version Control', formatter_class=<class 'argparse.RawTextHelpFormatter'>, conflict_handler='error', add_help=False))
2023-04-04 12:26:10,148 ERROR: unexpected error - The remote server unexpectedly closed the connection.                                                                                                 
Traceback (most recent call last):
  File "/workspaces/example-repos-dev/example-get-started-experiments/build/example-get-started-experiments/.venv/lib/python3.10/site-packages/dvc/cli/__init__.py", line 210, in main
    ret = cmd.do_run()
  File "/workspaces/example-repos-dev/example-get-started-experiments/build/example-get-started-experiments/.venv/lib/python3.10/site-packages/dvc/cli/command.py", line 26, in do_run
    return self.run()
  File "/workspaces/example-repos-dev/example-get-started-experiments/build/example-get-started-experiments/.venv/lib/python3.10/site-packages/dvc/commands/experiments/remove.py", line 31, in run
    removed = self.repo.experiments.remove(
  File "/workspaces/example-repos-dev/example-get-started-experiments/build/example-get-started-experiments/.venv/lib/python3.10/site-packages/dvc/repo/experiments/__init__.py", line 506, in remove
    return remove(self.repo, *args, **kwargs)
  File "/workspaces/example-repos-dev/example-get-started-experiments/build/example-get-started-experiments/.venv/lib/python3.10/site-packages/dvc/repo/__init__.py", line 65, in wrapper
    return f(repo, *args, **kwargs)
  File "/workspaces/example-repos-dev/example-get-started-experiments/build/example-get-started-experiments/.venv/lib/python3.10/site-packages/dvc/repo/scm_context.py", line 151, in run
    return method(repo, *args, **kw)
  File "/workspaces/example-repos-dev/example-get-started-experiments/build/example-get-started-experiments/.venv/lib/python3.10/site-packages/dvc/repo/experiments/remove.py", line 71, in remove
    _remove_commited_exps(repo.scm, exp_ref_list, git_remote)
  File "/workspaces/example-repos-dev/example-get-started-experiments/build/example-get-started-experiments/.venv/lib/python3.10/site-packages/dvc/repo/experiments/remove.py", line 112, in _remove_commited_exps
    push_refspec(
  File "/workspaces/example-repos-dev/example-get-started-experiments/build/example-get-started-experiments/.venv/lib/python3.10/site-packages/dvc/repo/experiments/utils.py", line 167, in push_refspec
    results = scm.push_refspecs(
  File "/workspaces/example-repos-dev/example-get-started-experiments/build/example-get-started-experiments/.venv/lib/python3.10/site-packages/scmrepo/git/__init__.py", line 286, in _backend_func
    result = func(*args, **kwargs)
  File "/workspaces/example-repos-dev/example-get-started-experiments/build/example-get-started-experiments/.venv/lib/python3.10/site-packages/scmrepo/git/backend/dulwich/__init__.py", line 571, in push_refspecs
    client.send_pack(
  File "/workspaces/example-repos-dev/example-get-started-experiments/build/example-get-started-experiments/.venv/lib/python3.10/site-packages/dulwich/client.py", line 2054, in send_pack
    ref_status = self._handle_receive_pack_tail(
  File "/workspaces/example-repos-dev/example-get-started-experiments/build/example-get-started-experiments/.venv/lib/python3.10/site-packages/dulwich/client.py", line 881, in _handle_receive_pack_tail
    for chan, data in _read_side_band64k_data(proto.read_pkt_seq()):
  File "/workspaces/example-repos-dev/example-get-started-experiments/build/example-get-started-experiments/.venv/lib/python3.10/site-packages/dulwich/client.py", line 458, in _read_side_band64k_data
    for pkt in pkt_seq:
  File "/workspaces/example-repos-dev/example-get-started-experiments/build/example-get-started-experiments/.venv/lib/python3.10/site-packages/dulwich/protocol.py", line 271, in read_pkt_seq
    pkt = self.read_pkt_line()
  File "/workspaces/example-repos-dev/example-get-started-experiments/build/example-get-started-experiments/.venv/lib/python3.10/site-packages/dulwich/protocol.py", line 214, in read_pkt_line
    raise HangupException()
dulwich.errors.HangupException: The remote server unexpectedly closed the connection.

2023-04-04 12:26:10,249 DEBUG: link type reflink is not available ([Errno 95] no more link types left to try out)
2023-04-04 12:26:10,249 DEBUG: Removing '/workspaces/example-repos-dev/example-get-started-experiments/build/.D69g23EAEMabcCHqmaow5u.tmp'
2023-04-04 12:26:10,249 DEBUG: Removing '/workspaces/example-repos-dev/example-get-started-experiments/build/.D69g23EAEMabcCHqmaow5u.tmp'
2023-04-04 12:26:10,249 DEBUG: Removing '/workspaces/example-repos-dev/example-get-started-experiments/build/.D69g23EAEMabcCHqmaow5u.tmp'
2023-04-04 12:26:10,250 DEBUG: Removing '/workspaces/example-repos-dev/example-get-started-experiments/build/example-get-started-experiments/.dvc/cache/.dRHZfWAaMPuhj4t4EBZiYD.tmp'
2023-04-04 12:26:10,257 DEBUG: Version info for developers:
DVC version: 2.52.1.dev12+gcf52efb8
-----------------------------------
Platform: Python 3.10.10 on Linux-5.4.0-1105-azure-x86_64-with-glibc2.31
Subprojects:
        dvc_data = 0.47.1
        dvc_objects = 0.21.1
        dvc_render = 0.3.1
        dvc_task = 0.2.0
        scmrepo = 0.2.0
Supports:
        http (aiohttp = 3.8.4, aiohttp-retry = 2.8.3),
        https (aiohttp = 3.8.4, aiohttp-retry = 2.8.3),
        s3 (s3fs = 2023.3.0, boto3 = 1.24.59)
Cache types: hardlink, symlink
Cache directory: ext4 on /dev/loop0
Caches: local
Remotes: s3
Workspace directory: ext4 on /dev/loop0
Repo: dvc, git
Repo.site_cache_dir: /var/tmp/dvc/repo/8320f4256dcd8a001d3e3ee03150ad17

Having any troubles? Hit us up at https://dvc.org/support, we are always happy to help!
2023-04-04 12:26:10,258 DEBUG: Analytics is enabled.
2023-04-04 12:26:10,318 DEBUG: Trying to spawn '['daemon', '-q', 'analytics', '/tmp/tmpaei65x3b']'
2023-04-04 12:26:10,320 DEBUG: Spawned '['daemon', '-q', 'analytics', '/tmp/tmpaei65x3b']'

`dvc exp push origin -A`

 $ dvc exp push origin -A -vv
2023-04-04 12:29:01,491 DEBUG: v2.52.1.dev12+gcf52efb8, CPython 3.10.10 on Linux-5.4.0-1105-azure-x86_64-with-glibc2.31
2023-04-04 12:29:01,491 DEBUG: command: /workspaces/example-repos-dev/example-get-started-experiments/build/example-get-started-experiments/.venv/bin/dvc exp push origin -A -vv
2023-04-04 12:29:01,491 TRACE: Namespace(cprofile=False, yappi=False, yappi_separate_threads=False, viztracer=False, viztracer_depth=None, viztracer_async=False, cprofile_dump=None, pdb=False, instrument=False, instrument_open=False, show_stack=False, quiet=0, verbose=2, cd='.', cmd='push', all_commits=True, rev=None, num=1, force=False, push_cache=True, dvc_remote=None, jobs=None, run_cache=False, git_remote='origin', experiment=[], func=<class 'dvc.commands.experiments.push.CmdExperimentsPush'>, parser=DvcParser(prog='dvc', usage=None, description='Data Version Control', formatter_class=<class 'argparse.RawTextHelpFormatter'>, conflict_handler='error', add_help=False))
2023-04-04 12:29:01,858 DEBUG: git push experiment ['refs/exps/13/5f0b84da997954e72fb87984680750c4207985/least-army:refs/exps/13/5f0b84da997954e72fb87984680750c4207985/least-army', 'refs/exps/13/67013071a1a68f716acce583990824ef3f112c/puffy-merk:refs/exps/13/67013071a1a68f716acce583990824ef3f112c/puffy-merk', 'refs/exps/13/67013071a1a68f716acce583990824ef3f112c/teary-flab:refs/exps/13/67013071a1a68f716acce583990824ef3f112c/teary-flab', 'refs/exps/13/5f0b84da997954e72fb87984680750c4207985/flawy-girl:refs/exps/13/5f0b84da997954e72fb87984680750c4207985/flawy-girl', 'refs/exps/13/5f0b84da997954e72fb87984680750c4207985/stiff-lieu:refs/exps/13/5f0b84da997954e72fb87984680750c4207985/stiff-lieu', 'refs/exps/13/67013071a1a68f716acce583990824ef3f112c/aglow-gigs:refs/exps/13/67013071a1a68f716acce583990824ef3f112c/aglow-gigs', 'refs/exps/f5/a6248e51e86b9b0220e4841248b3fe33da748b/grade-hows:refs/exps/f5/a6248e51e86b9b0220e4841248b3fe33da748b/grade-hows'] -> 'origin'
2023-04-04 12:29:03,225 ERROR: unexpected error - The remote server unexpectedly closed the connection.                                                                                                 
Traceback (most recent call last):
  File "/workspaces/example-repos-dev/example-get-started-experiments/build/example-get-started-experiments/.venv/lib/python3.10/site-packages/dvc/cli/__init__.py", line 210, in main
    ret = cmd.do_run()
  File "/workspaces/example-repos-dev/example-get-started-experiments/build/example-get-started-experiments/.venv/lib/python3.10/site-packages/dvc/cli/command.py", line 26, in do_run
    return self.run()
  File "/workspaces/example-repos-dev/example-get-started-experiments/build/example-get-started-experiments/.venv/lib/python3.10/site-packages/dvc/commands/experiments/push.py", line 26, in run
    result = self.repo.experiments.push(
  File "/workspaces/example-repos-dev/example-get-started-experiments/build/example-get-started-experiments/.venv/lib/python3.10/site-packages/dvc/repo/experiments/__init__.py", line 491, in push
    return push(self.repo, *args, **kwargs)
  File "/workspaces/example-repos-dev/example-get-started-experiments/build/example-get-started-experiments/.venv/lib/python3.10/site-packages/dvc/repo/__init__.py", line 65, in wrapper
    return f(repo, *args, **kwargs)
  File "/workspaces/example-repos-dev/example-get-started-experiments/build/example-get-started-experiments/.venv/lib/python3.10/site-packages/dvc/repo/scm_context.py", line 151, in run
    return method(repo, *args, **kw)
  File "/workspaces/example-repos-dev/example-get-started-experiments/build/example-get-started-experiments/.venv/lib/python3.10/site-packages/dvc/repo/experiments/push.py", line 106, in push
    push_result = _push(repo, git_remote, exp_ref_set, force)
  File "/workspaces/example-repos-dev/example-get-started-experiments/build/example-get-started-experiments/.venv/lib/python3.10/site-packages/dvc/repo/experiments/push.py", line 141, in _push
    results: Mapping[str, SyncStatus] = repo.scm.push_refspecs(
  File "/workspaces/example-repos-dev/example-get-started-experiments/build/example-get-started-experiments/.venv/lib/python3.10/site-packages/scmrepo/git/__init__.py", line 286, in _backend_func
    result = func(*args, **kwargs)
  File "/workspaces/example-repos-dev/example-get-started-experiments/build/example-get-started-experiments/.venv/lib/python3.10/site-packages/scmrepo/git/backend/dulwich/__init__.py", line 571, in push_refspecs
    client.send_pack(
  File "/workspaces/example-repos-dev/example-get-started-experiments/build/example-get-started-experiments/.venv/lib/python3.10/site-packages/dulwich/client.py", line 2054, in send_pack
    ref_status = self._handle_receive_pack_tail(
  File "/workspaces/example-repos-dev/example-get-started-experiments/build/example-get-started-experiments/.venv/lib/python3.10/site-packages/dulwich/client.py", line 881, in _handle_receive_pack_tail
    for chan, data in _read_side_band64k_data(proto.read_pkt_seq()):
  File "/workspaces/example-repos-dev/example-get-started-experiments/build/example-get-started-experiments/.venv/lib/python3.10/site-packages/dulwich/client.py", line 458, in _read_side_band64k_data
    for pkt in pkt_seq:
  File "/workspaces/example-repos-dev/example-get-started-experiments/build/example-get-started-experiments/.venv/lib/python3.10/site-packages/dulwich/protocol.py", line 271, in read_pkt_seq
    pkt = self.read_pkt_line()
  File "/workspaces/example-repos-dev/example-get-started-experiments/build/example-get-started-experiments/.venv/lib/python3.10/site-packages/dulwich/protocol.py", line 214, in read_pkt_line
    raise HangupException()
dulwich.errors.HangupException: The remote server unexpectedly closed the connection.

2023-04-04 12:29:03,341 DEBUG: link type reflink is not available ([Errno 95] no more link types left to try out)
2023-04-04 12:29:03,341 DEBUG: Removing '/workspaces/example-repos-dev/example-get-started-experiments/build/.deojpya2GcDwFFPBirgpyk.tmp'
2023-04-04 12:29:03,341 DEBUG: Removing '/workspaces/example-repos-dev/example-get-started-experiments/build/.deojpya2GcDwFFPBirgpyk.tmp'
2023-04-04 12:29:03,341 DEBUG: Removing '/workspaces/example-repos-dev/example-get-started-experiments/build/.deojpya2GcDwFFPBirgpyk.tmp'
2023-04-04 12:29:03,342 DEBUG: Removing '/workspaces/example-repos-dev/example-get-started-experiments/build/example-get-started-experiments/.dvc/cache/.hNCuuicWo3kgDvbMgoboxd.tmp'
2023-04-04 12:29:03,349 DEBUG: Version info for developers:
DVC version: 2.52.1.dev12+gcf52efb8
-----------------------------------
Platform: Python 3.10.10 on Linux-5.4.0-1105-azure-x86_64-with-glibc2.31
Subprojects:
        dvc_data = 0.47.1
        dvc_objects = 0.21.1
        dvc_render = 0.3.1
        dvc_task = 0.2.0
        scmrepo = 0.2.0
Supports:
        http (aiohttp = 3.8.4, aiohttp-retry = 2.8.3),
        https (aiohttp = 3.8.4, aiohttp-retry = 2.8.3),
        s3 (s3fs = 2023.3.0, boto3 = 1.24.59)
Cache types: hardlink, symlink
Cache directory: ext4 on /dev/loop0
Caches: local
Remotes: s3
Workspace directory: ext4 on /dev/loop0
Repo: dvc, git
Repo.site_cache_dir: /var/tmp/dvc/repo/8320f4256dcd8a001d3e3ee03150ad17

Having any troubles? Hit us up at https://dvc.org/support, we are always happy to help!
2023-04-04 12:29:03,353 DEBUG: Analytics is enabled.
2023-04-04 12:29:03,399 DEBUG: Trying to spawn '['daemon', '-q', 'analytics', '/tmp/tmp8i_tmd6h']'
2023-04-04 12:29:03,400 DEBUG: Spawned '['daemon', '-q', 'analytics', '/tmp/tmp8i_tmd6h']'

From this repo: https://github.com/iterative/example-repos-dev

[pmrowla]

Just an FYI that dvc exp remove -g ... will probably not work out of the box in codespaces, the default github access token generated for a codespace does not have delete permissions. (This can be verified by trying git push --delete refs/exps/... in the codespace)

I'm assuming it's possible to configure codespaces to use a specific token, in which case exp remove -g origin would work as long as the token had the appropriate delete permissions.

[daavoo]

I'm assuming it's possible to configure codespaces to use a specific token, in which case exp remove -g origin would work as long as the token had the appropriate delete permissions.

Yes, the example-repos-dev is correctly configured so exp remove -g and exp push both work:

https://github.com/iterative/example-repos-dev/blob/dcfd76301af427f9cc33f7e3e6248fa68669d16b/.devcontainer.json#L20-L22

[kyrylogy]

Hi! Is this fixed? I'm still running into the same issue, seems like hashes are somehow getting mixed up when running dvc exp push with pre-configured remote origin. Problem persists for me even after updating scmrepo>=0.2.1 and DVC version: 2.53.0

Here's my log:

https://github.com/kyrylogy/dvc-test/actions/runs/4627560834/jobs/8185613736

[pmrowla]

@kyrylogy can you try setting fetch-depth: 0 in your actions/checkout step (see https://github.com/actions/checkout#fetch-all-history-for-all-tags-and-branches)