credential-store: support `store` for new credentials

[mirkolenz]

Bug Report

Description

When using the git credential manager to manage login information and using https-based remotes in dvc (e.g., dvc import https://github.com/...), the credentials are not properly cached. If I just try to fetch some repo using https, GCM correctly uses the saved credentials. In the case of dvc, the login window appears for every step of the pipeline, meaning I have to sign in three times for a single import. GitHub already warned me that GCM triggered some rate limit for my account because of this. Here is an exemplary terminal output:

Importing '/ (https://github.com/xxx/xxx.git)' -> 'xxx'
Cloning xxx.git|                                                                                                                                                                                                                  |0.00/? [00:00,      ?obj/s]
info: please complete authentication in your browser...
Cloning xxx.git|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| Compressing |876/876 [03:38,    219s/obj]
info: please complete authentication in your browser...
info: please complete authentication in your browser...

The same happens for subsequent runs of dvc pull.

Reproduce

1. dvc import https://github.com/xxx/xxx.git
2. sign in with GitHub in the GCM window
3. sign in again
4. ...

Expected

Credentials are used without the dialogue popping up at all (just like it does for the regular git commands).

Environment information

Output of dvc doctor:

DVC version: 2.43.1 (brew)
---------------------------------
Platform: Python 3.11.1 on macOS-13.2-x86_64-i386-64bit
Subprojects:
	dvc_data = 0.35.1
	dvc_objects = 0.19.0
	dvc_render = 0.0.17
	dvc_task = 0.1.11
	dvclive = 1.3.3
	scmrepo = 0.1.6
Supports:
	azure (adlfs = 2023.1.0, knack = 0.10.1, azure-identity = 1.12.0),
	gdrive (pydrive2 = 1.15.0),
	gs (gcsfs = 2023.1.0),
	http (aiohttp = 3.8.3, aiohttp-retry = 2.8.3),
	https (aiohttp = 3.8.3, aiohttp-retry = 2.8.3),
	oss (ossfs = 2021.8.0),
	s3 (s3fs = 2023.1.0, boto3 = 1.24.59),
	ssh (sshfs = 2023.1.0),
	webdav (webdav4 = 0.9.8),
	webdavs (webdav4 = 0.9.8),
	webhdfs (fsspec = 2023.1.0)
Cache types: reflink, hardlink, symlink
Cache directory: apfs on /dev/disk1s1s1
Caches: local
Remotes: None
Workspace directory: apfs on /dev/disk1s1s1
Repo: dvc, git

[dtrifiro]

The reason for this is that git-credential-manager stores the token acquired from github using the store method, which is currently not implemented for scmrepo: only get is currently supported (see #138)

[mirkolenz]

Thank you for the quick investigation! Is there anything I can do in the meantime? I need to switch to https-based imports due to treeverse/dvc#7702. My current (and somewhat hacky) workaround is to commit all dvc-files with the ssh connection. Then, if I need to pull this data in a GitHub action, I replace all git@github.com: urls with https://github.com/. This means that I use SSH locally and HTTPS in actions. I had some smaller issues in the past due to the different behavior in CI and on my machine, thus I would love to unify this 😄

[mirkolenz]

In case you are interested, here is my current workflow.

[dtrifiro]

@mirkolenz are you unable to use ssh key authentication due similarly to treeverse/dvc#7702? That is only relevant to gha on windows, I see that you're using ubuntu-latest on your workflow.

[mirkolenz]

I tried setting up SSH keys using the webfactory/ssh-agent action. However, dvc always complained that no valid credentials were provided even though the SSH keys were added as deploy keys to the individual repositories. This is the same action that was mentioned in treeverse/dvc#7702 as well, so I see some similarities here.

The action I currently use, de-vri-es/setup-git-credentials, should also automatically rewrite all ssh-based git operations to the corresponding https addressed (essentially doing the same like I do now). However, this also did not work due to the same issues: dvc complained that no valid credentials were provided.

[dtrifiro]

If you set up ssh keys with webfactory/ssh-agent, does this snippet show any keys?

import asyncio

import asyncssh


async def main():
    async with asyncssh.agent.connect_agent() as agent:
        keys = await agent.get_keys()
        for key in keys:
            print(key.algorithm, key.get_comment())


if __name__ == "__main__":
    asyncio.run(main())

[mirkolenz]

Here is the output:

python dvc_test.py
  shell: /usr/bin/bash -e {0}
  env:
    POETRY_VERSION: 1.3.1
    PYTHON_VERSION: 3.9
    DVC_VERSION: 2.43.0
    SSH_AUTH_SOCK: /tmp/ssh-XXXXXXiyBBBE/agent.1658
    SSH_AGENT_PID: 1659
    pythonLocation: /opt/hostedtoolcache/Python/3.9.16/x64
    PKG_CONFIG_PATH: /opt/hostedtoolcache/Python/3.9.16/x64/lib/pkgconfig
    Python_ROOT_DIR: /opt/hostedtoolcache/Python/3.9.16/x64
    Python2_ROOT_DIR: /opt/hostedtoolcache/Python/3.9.16/x64
    Python3_ROOT_DIR: /opt/hostedtoolcache/Python/3.9.16/x64
    LD_LIBRARY_PATH: /opt/hostedtoolcache/Python/3.9.16/x64/lib
b'ssh-ed25519' git@github.com:recap-utr/arguebase-public.git
b'ssh-ed25519' git@github.com:recap-utr/arguebase-private.git

So it seems that the keys are picked up.

[dtrifiro]

@mirkolenz could you open a separate issue related to the "No valid credentials" issue with ssh auth?