Clone repo from googlesource.com

[giswqs]

It seems duwich could not clone repo from googlesource.com. Any advice? Thanks.

from dulwich import porcelain
porcelain.clone("git://earthengine.googlesource.com/users/google/datasets", "datasets")

[jelmer]

That looks like an invalid URL to me - unless you have some indication that this works with C Git? AFAIK googlesource.com doesn't support the git:// protocol.

[giswqs]

https://earthengine.googlesource.com/users/google/datasets is a valid URL, although it requires users with Google Earth Engine access. I can clone the repo using Git, but not dulwich. See below.

[jelmer]

You're using a different URL in Dulwich - the URL you had in the fragment you posted used the "git" URL scheme, not "http".

This repository is not available over the plain git protocol.

[giswqs]

So the URL https://earthengine.googlesource.com/users/google/datasets is not supported by dulwich?

The following two commands work fine on my computer.

git clone "https://earthengine.googlesource.com/users/google/datasets"
git clone "https://earthengine.googlesource.com/users/google/datasets.git"

[jelmer]

Yes, that URL is supported, but it's not the URL you are using in your code fragment.

You're specifying the git:// protocol to Dulwich, not https://. You're using https in your git command.

Dulwich supports both git:// and https://, but Googlesource only supports https.

[giswqs]

I did try https, but it didn't work. See below.

GitProtocolError: unexpected http resp 401 for https://earthengine.googlesource.com/users/google/datasets/info/refs?service=git-upload-pack

[jelmer]

That's indeed because of authentication - you can specify the credentials in the URL, i.e. https://username:password@earthengine.googlesource.com/users/google/dataset

[giswqs]

Unfortunately, providing username and password is not an option. I am using Dulwich for another Python package. I could not ask the user to enter the user name and password. Once users authenticate Google Earth Engine by logging to their Google account, credentials are saved under the user directory (e.g., ~/.config/earthengine/credentials). I can use git clone without having to enter user name and password. Can Dulwich do the same thing? Thanks.

[jelmer]

How does git retrieve the credentials from ~/.config/earthengine/credentials? I'm pretty sure you'd need some kind of helper to do that, and that it doesn't check that location by default.

Either way, I'm afraid the Dulwich porcelain doesn't do any credentials retrieval itself. You can pass credentials into it that you've retrieved somewhere else, but it doesn't go out looking for credentials itself. That's probably not going to be a two line solution though.

You could take care of retrieving the credentials by parsing ~/.config/earthengine/credentials, and then pass those into dulwich?

[giswqs]

@jelmer Thanks for your insights! I am not sure how git retrieves the credentials. I simply create a new Jupyter notebook, then execute !git clone "https://earthengine.googlesource.com/users/google/datasets" and it works. However, the downside is that users will need to install git on their computer. That's why I want to use Dulwich.
The credentials inside ~/.config/earthengine/credentials is just a token with a very long string. It is not in the format of username and password. I am not sure how I can parse the token into Dulwich. I tried https://token@earthengine.googlesource.com/users/google/dataset, but it did not work.

[jelmer]

it probably has a Git credentials helper that can provide the credentials - see https://www.git-scm.com/docs/gitcredentials

dulwich does not support reading credentials from gitcredentials helpers, but that is not properly hooked up yet in the porcelain unfortunately. If you try this again in a couple of months it may work.

[giswqs]

@jelmer Good to know. Thank you very much for your help. I am closing this issue for now. Will check back dulwich in a couple of months.

[jameslittle]

Just wanted to share my solution for this. Although GCP does provide a credential helper, in the form of:

gcloud auth git-helper ...

all this does is obtain an oauth2 access token for the authenticated user, which is then used (along with the username) for basic http auth. Fortunately it's quite trivial to obtain an access token in python code; see https://google-auth.readthedocs.io/en/master/user-guide.html
Something like below should do the job:

import urllib
import google.auth
from google.auth.transport.requests import Request
from dulwich import porcelain

creds, _= google.auth.default()
creds.refresh(Request())
username = urllib.parse.quote("<USER>@<GOOGLE_ACCOUNT_EMAIL>")
password = creds.token

porcelain.ls_remote(f"https://{username}:{password}@source.developers.google.com/<MY_REPO_PATH>")