dvc[ssh]>=2.8.2 `dvc exp push origin` unexpected error - argument 0 must be str, not NoneType

[agrizzli]

Bug Report

When using dvc exp push origin with dvc[ssh]==2.8.2, dvc[ssh]==2.8.3 or from repo, it results in unexpected error - argument 0 must be str, not NoneType. The same happens with dvc exp list origin. There is no problem with 2.8.1 at all.

Description

I have a repo with just few first experiments using dvc exp run. Trying to push experiments to git over ssh results in:

$ dvc exp push origin exp-fac6d

ERROR: unexpected error - argument 0 must be str, not NoneType                                                                                                                                             

Having any troubles? Hit us up at https://dvc.org/support, we are always happy to help!

The same for the list origin. Verbose versions of output attached.

Reproduce

• Install dvc[ssh]==2.8.2 or higher
• Setup DVC project and a pipeline with dvc.yaml
• Run some experiments using dvc exp run
• dvc exp list origin or dvc exp list push EXP-HASH

Expected

• dvc exp list origin should list experiments from git.
• dvc exp push origin EXP-HASH is expected to push experiment to git.

Environment information

dvc_list_origin.txt
dvc_push_origin.txt

Output of dvc doctor:

DVC version: 2.8.2 (pip)
---------------------------------
Platform: Python 3.8.10 on Linux-5.8.0-63-generic-x86_64-with-glibc2.32
Supports:
	hdfs (fsspec = 2021.11.1, pyarrow = 6.0.1),
	webhdfs (fsspec = 2021.11.1),
	http (aiohttp = 3.8.1, aiohttp-retry = 2.4.6),
	https (aiohttp = 3.8.1, aiohttp-retry = 2.4.6),
	ssh (sshfs = 2021.11.2)
Cache types: hardlink, symlink
Cache directory: ext4 on /dev/nvme0n1p3
Caches: local
Remotes: ssh, ssh
Workspace directory: ext4 on /dev/nvme0n1p3
Repo: dvc, git

[daavoo]

Hi @agrizzli ! Could you check if the problem persists in the pre-release version: https://dvc.org/doc/install/pre-release ?

[agrizzli]

Yes, the same issue.

Installed using pip install git+https://github.com/iterative/dvc#egg=dvc[ssh]

Output of dvc exp list -v origin attached.
dvc_exp_list_origin_pre_release.txt

DVC version: 2.8.4.dev93+g024f06d6 
---------------------------------
Platform: Python 3.8.10 on Linux-5.8.0-63-generic-x86_64-with-glibc2.32
Supports:
	hdfs (fsspec = 2021.11.1, pyarrow = 6.0.1),
	webhdfs (fsspec = 2021.11.1),
	http (aiohttp = 3.8.1, aiohttp-retry = 2.4.6),
	https (aiohttp = 3.8.1, aiohttp-retry = 2.4.6),
	ssh (sshfs = 2021.11.2)
Cache types: hardlink, symlink
Cache directory: ext4 on /dev/nvme0n1p3
Caches: local
Remotes: local, ssh
Workspace directory: ext4 on /dev/nvme0n1p3
Repo: dvc, git

[pmrowla]

@agrizzli can you show the output for

git remote -v

[agrizzli]

(venv) agrizzli@agrizzli-razer:~/Code/PyCharm/sparta-analysis$ git remote -v
origin	ssh://gitlabfw/pipeline/analysis.git (fetch)
origin	ssh://gitlabfw/pipeline/analysis.git (push)

gitlabfw is set up via .ssh/config on localhost

Host gitlabfw
    HostName localhost
    Port 8022
    User git
    PreferredAuthentications publickey
    IdentityFile ~/.ssh/id_rsa_sparta

It authenticates against a GitLab instance.

[pmrowla]

@agrizzli can you try installing the following (in the same venv where you already have upstream DVC) and see if it resolves the issue:

pip install git+https://github.com/iterative/scmrepo.git@refs/pull/3/merge

[agrizzli]

Yes, it does. I did the following steps:

pip uninstall dvc
pip install git+https://github.com/iterative/dvc#egg=dvc[ssh]
pip uninstall scmrepo
pip install git+https://github.com/iterative/scmrepo.git@refs/pull/3/merge
dvc exp list --all -v origin

I had explicitly to uninstall scmrepo after using the latest DVC version from git, otherways scmrepo it was not working, although scmrepo was already present in the same version 0.0.2 after DVC installation. DVC doctor says:

(venv) agrizzli@agrizzli-razer:~/Code/PyCharm/sparta-analysis$ dvc doctor
DVC version: 2.8.4.dev102+g954ac6ab 
---------------------------------
Platform: Python 3.8.10 on Linux-5.8.0-63-generic-x86_64-with-glibc2.32
Supports:
	hdfs (fsspec = 2021.11.1, pyarrow = 6.0.1),
	webhdfs (fsspec = 2021.11.1),
	http (aiohttp = 3.8.1, aiohttp-retry = 2.4.6),
	https (aiohttp = 3.8.1, aiohttp-retry = 2.4.6),
	ssh (sshfs = 2021.11.2)
Cache types: hardlink, symlink
Cache directory: ext4 on /dev/nvme0n1p3
Caches: local
Remotes: local, ssh
Workspace directory: ext4 on /dev/nvme0n1p3
Repo: dvc, git

(venv) agrizzli@agrizzli-razer:~/Code/PyCharm/sparta-analysis$ pip  list -v | grep scmrepo
scmrepo                  0.0.2                  /home/agrizzli/Code/PyCharm/sparta-analysis/venv/lib/python3.8/site-packages pip

The output of dvc exp list origin is now as expected:

(venv) agrizzli@agrizzli-razer:~/Code/PyCharm/sparta-analysis$ dvc exp list --all -v origin
2021-12-06 11:12:27,260 DEBUG: Adding '/home/agrizzli/Code/PyCharm/sparta-analysis/.dvc/config.local' to gitignore file.
2021-12-06 11:12:27,273 DEBUG: Adding '/home/agrizzli/Code/PyCharm/sparta-analysis/.dvc/tmp' to gitignore file.
2021-12-06 11:12:27,274 DEBUG: Adding '/home/agrizzli/Code/PyCharm/sparta-analysis/.dvc/cache' to gitignore file.
1e318ba:                                                              
	exp-6a8f8
4316cff:
	exp-fac6d
7bbcf90:
	exp-535f6
cec9b56:
	exp-0f519
2021-12-06 11:12:29,230 DEBUG: Analytics is enabled.
2021-12-06 11:12:29,266 DEBUG: Trying to spawn '['daemon', '-q', 'analytics', '/tmp/tmp53naol8m']'
2021-12-06 11:12:29,266 DEBUG: Spawned '['daemon', '-q', 'analytics', '/tmp/tmp53naol8m']'

As well as the procedure of dvc exp push origin, which works fine now:

(venv) agrizzli@agrizzli-razer:~/Code/PyCharm/sparta-analysis$ dvc exp push -v origin exp-44136
2021-12-06 11:13:32,117 DEBUG: Adding '/home/agrizzli/Code/PyCharm/sparta-analysis/.dvc/config.local' to gitignore file.
2021-12-06 11:13:32,130 DEBUG: Adding '/home/agrizzli/Code/PyCharm/sparta-analysis/.dvc/tmp' to gitignore file.
2021-12-06 11:13:32,131 DEBUG: Adding '/home/agrizzli/Code/PyCharm/sparta-analysis/.dvc/cache' to gitignore file.
2021-12-06 11:13:32,133 DEBUG: git push experiment 'refs/exps/91/7a97d70cf6a5b90947c9c097cbcb81d1b5b716/exp-44136' -> 'origin'
2021-12-06 11:13:35,678 DEBUG: dvc push experiment 'refs/exps/91/7a97d70cf6a5b90947c9c097cbcb81d1b5b716/exp-44136'                                                                                                                                                                                                                                                                  
2021-12-06 11:13:35,848 DEBUG: Preparing to transfer data from '/home/agrizzli/Code/PyCharm/sparta-analysis/.dvc/cache' to '/mnt/exafs/dvc'
2021-12-06 11:13:35,849 DEBUG: Preparing to collect status from '/mnt/exafs/dvc'
2021-12-06 11:13:35,849 DEBUG: Collecting status from '/mnt/exafs/dvc'
2021-12-06 11:13:36,798 DEBUG: Estimated remote size: 256 files                                                                                                                                                                                                                                                                                                                     
2021-12-06 11:13:36,800 DEBUG: Querying '9' hashes via traverse                                                                                                                                                                                                                                                                                                                     
Pushed experiment 'exp-44136'to Git remote 'origin'.                                                                                                                                                                                                                                                                                                                                
2021-12-06 11:13:39,247 DEBUG: Analytics is enabled.
2021-12-06 11:13:39,300 DEBUG: Trying to spawn '['daemon', '-q', 'analytics', '/tmp/tmpmcm7whw4']'
2021-12-06 11:13:39,301 DEBUG: Spawned '['daemon', '-q', 'analytics', '/tmp/tmpmcm7whw4']'