Add GitBucket support (refs #142) #144

kounoike · 2017-03-20T14:04:52Z

GitBucket is Git platform with high-API compatibility to GitHub (for client, GitBucket looks like as GitHub Enterprise).
This PR add GitBucket support with some actions.

There are few changes to git-repo side.

add support to ssh:// style remote.
also support http, not only https.
Change refspec in github.py:request_fetch

gitbucket.py now supports

login with user created access token.
list repositories
clone repository
fetch PR

these action doesn't supported now.

automatic get access token by git repo config
delete repository
create PR
list PR

guyzmo · 2017-04-28T21:11:51Z

hi @kounoike I'm starting to review your code, and I'd appreciate if you could split it in a four commits, one per file. Because it's likely I won't merge your PR as is, but rather gradually (the fixes first, and the gitbucket after), so I'd like to be able to preserve attribution to you for your changes 😉

kounoike · 2017-04-29T01:53:04Z

@guyzmo Thanks to review! OK, I will split my commit. But, I'm not accustomed to such non-standard Git operations. I split to four commit locally. How to push it for GitHub? git push origin or git push -f origin?

guyzmo · 2017-04-29T11:38:27Z

Thank you!

You should git push -f github pr-gitbucket (or origin instead of github).
It's OK to rewrite history into a branch that servers as a PR.

kounoike · 2017-04-29T11:52:54Z

@guyzmo OK. I pushed it!

guyzmo · 2017-04-29T12:46:13Z

git_repo/services/service.py

@@ -244,7 +244,10 @@ def format_path(self, repository, namespace=None, rw=False):
        if not rw and '/' in repo:
            return '{}/{}'.format(self.url_ro, repo)
        elif rw and '/' in repo:
-            return '{}:{}'.format(self.url_rw, repo)
+            if self.url_rw.startswith('ssh://'):


in what context do you need ssh:// URIs? As far as I can tell, it's not supported by neither Git or SSH directly (only by xdg- tools or browsers.

GitBucket's ssh url is: ssh://git@server:29418/root/repo.git

GitBucket doesn't use system sshd. It uses embedded sshd. So, it can't listen on 22 port.

By Git manual, "An alternative scp-like syntax" doesn't take ssh port. So it is required for GitBucket with ssh repository access. and ssh:// protocol is supported officially.

Interesting, that I did not know.

could you update the commits related to that change the title of the commits to 'Adds missing ssh:// URL support' and in the body the details above? It's not only for GitBucket support, it's actually an important fix 😉

I fixed it now. push -f again?

guyzmo · 2017-04-29T14:17:59Z

git_repo/repo.py

                        *_, user, name = url.split('/')
                        self.set_repo_slug('/'.join([user, name]))
                        return
+                    elif url.startswith('ssh://'):


instead of repeating the above code body below, you can simply write:

if url.startswith('https') or url.startswith('http') or url.startswith('ssh'):

OK I will fix it, but I think it is more better:

if url.startswith('https://') or url.startswith('http://') or url.startswith('ssh://'):

because, "An alternative scp-like syntax" can use any username such as:

http@server:owner/repo.git

https@server:owner/repo.git

ssh@server:owner/repo.git

I know these format can be used by Gogs. Gogs can configure ssh username. (Of course it is rare case)

Gogs can configure ssh username. (Of course it is rare case)

hmm... it means startswith('git@') doesn't match for Gogs...
In my Gogs instance running on Raspberry Pi, ssh URL is: pi@raspberrypi.local:test/test.git

But it is not GitBucket related fix. Which is better action?
Should I create new Issue/PR? or fix it in this PR?

It should be rewrite as:

for url in remote.urls: if url.endswith('.git'): url = url[:-4] if url.find('://') > -1: *_, user, name = url.split('/') self.set_repo_slug('/'.join([user, name])) return elif url.find('@') > -1: _, repo_slug = url.split(':') self.set_repo_slug(repo_slug) return

but it is not tested.

well, I'm ok if you take all the SSH format fixes as another PR.

I was anyway planning to merge them first before merging your gitbucket PR.

if you do another PR, make a branch for each PR and don't forget to rebase the branch for this PR on top of the other one! ☺

Thanks, I do it.
But it contains 'Change local branch name to remote repo name.' commit.
I think it is not related change for ssh:// URL things.

it's ok, it's a minor bug that also needs to get fixed 😉

I create new PR #148 for ssh:// and user@server:pathto/repo.git URL fixes. and I will remove these changes.

nah, you can keep this PR on top of the other PR it's ok, I'm merging it ASAP

just add the third commit you forgot 😉

guyzmo · 2017-04-29T14:18:55Z

git_repo/repo.py

                        *_, user, name = url.split('/')
                        self.set_repo_slug('/'.join([user, name]))
                        return
+                    elif url.startswith('ssh://'):
+                        *_, user, name = url.split('/')
+                        self.set_repo_slug('/'.join([user, name]))


if we were keeping that body, the return statement is missing here, to break from the for loop. (technically, it's more a break than a return, but in this case they're the same).

guyzmo

the commit " Change slug detection code. (for GitBucket compatibility)" should also change, it's related to the other ssh:// change, i.e.: it's a bugfix not just a GitBucket thing.

guyzmo · 2017-04-29T14:38:02Z

git_repo/services/ext/gitbucket.py

+    @classmethod
+    def get_auth_token(cls, login, password, prompt=None):
+        print("Please edit _YOUR_ACCESS_TOKEN_ to actual token in ~/.gitconfig or ~/.git/config after.")
+        print("And add ssh-url = ssh://git@{}:_YOUR_SSH_PORT_".format(cls.fqdn))


instead of hacking your way like that, you can actually do:

print("Please open the following URL: https://yourgitbucket/where/are/in/settings/auth/tokens") print("Generate a new token, and paste it at the following prompt.") return prompt('token> ')

the prompt parameter is actually the loop_input callback given from git-repo's CLI interface.

guyzmo · 2017-04-29T14:39:26Z

git_repo/services/ext/gitbucket.py

+        super(GitbucketService, self).__init__(*args, **kwarg)
+
+    def connect(self):
+        self.gh = github3.GitHubEnterprise(self.build_url(self))


maybe this could be done in GithubService itself: if the URL is github.com it's github3.GitHub, otherwise it's GithubEnterprise?

I want to replace self.gh object from github3.GitHub to github3.GithHubEnterprise with GitBucket's API endpoint URL.

ok, but that's going to be an issue. The self.gh instance shall be created in the constructor as first line even before the call the the constructor.

The reason for that is the tests, where it pulls the session instance out of the self.gh instance. If you change the self.gh instance at self.connect() time, there's a high probability that the tests will be testing another session instance than the one you're actually using in your methods.

So, we could patch GithubService.__init__() so that:

GITHUB_FQDN='github.com' @register_target('hub', 'github') class GithubService(RepositoryService): fqdn = GITHUB_FQDN def __init__(self, *args, **kwarg): if self.fqdn == GITHUB_FQDN: self.gh = github3.GitHub() else: self.gh = github3.GitHubEnterprise(RepositoryService.build_url(self)) super(GithubService, self).__init__(*args, **kwarg) ``` and as a side effect, there's a chance that actually it enables `GithubService` to work with self hosted github instances… out of the box 🙌

I try it. but it doesn't load fqdn/port from configuration on this phase. Because, load_configuration is called by RepositoryService.__init__. And RepositoryService.__init__ also calls self.connect, so self.gh must be initialized before init called.
@guyzmo do you have any idea?

ok, I'll make a ticket to github3.py for this issue, so he splits the init in several steps. (by having a default URL to an empty invalid value, and a set_url() method)

Currently, here's the code it is doing at startup time.

So we keep my suggestion above, but the connect() method shall change:

def connect(self): self.gh._session.base_url = url.rstrip('/') + '/api/v3' self.gh._session.verify = self.session_certificate or not self.session_insecure super(GitbucketService, self).connect()

...?
I talk about timing of loading settings from .gitconfig.

on GitBucketService.__init__ calls GithubService.__init__

on GithubService.__init__, assign self.gh to GitHubEnterprise with build_url

on GithubService.__init__, calls RepositoryService.__init__

on RepositoryService.__init__ calls load_configuration

.gitconfig's fqdn/port settings load at 4. but build_url called at 2.

and even if rewrite __init__ in github.py as:

def __init__(self, *args, **kwarg): super(GithubService, self).__init__(*args, **kwarg)``` if self.fqdn == GITHUB_COM_FQDN: self.gh = github3.GitHub() else: self.gh = github3.GitHubEnterprise(RepositoryService.build_url(self))

it get error because RepositoryService.__init__ calls self.connect. it uses self.gh.

cf how I did it in gitlab.

The order is:

init phase:

instanciate session (and stuff that go with it)

load configuration

connect phase:

if need be, overload the github reference (but preserve the session)

do the connect stuff

basically here's the fixed gist:

GITHUB_FQDN='github.com' @register_target('hub', 'github') class GithubService(RepositoryService): fqdn = GITHUB_FQDN def __init__(self, *args, **kwarg): self.gh = github3.GitHub() super(GithubService, self).__init__(*args, **kwarg) def connect(self): if self.fqdn != GITHUB_FQDN: # upgrade self.gh from a GitHub object to a GitHubEnterprise object gh = github3.GitHubEnterprise(RepositoryService.build_url(self)) self.gh._session.base_url = gh._session.base_url self.gh = gh # propagate ssl certificate parameter self.gh._session.verify = self.session_certificate or not self.session_insecure try: self.gh.login(token=self._privatekey) self.username = self.gh.user().login except github3.models.GitHubError as err: if 401 == err.code: if not self._privatekey: raise ConnectionError('Could not connect to Github. ' 'Please configure .gitconfig ' 'with your github private key.') from err else: raise ConnectionError('Could not connect to Github. ' 'Check your configuration and try again.') from err

guyzmo · 2017-04-29T14:40:54Z

git_repo/services/ext/gitbucket.py

+    def format_path(self, repository, namespace=None, rw=False):
+        path = super(GitbucketService, self).format_path(repository, namespace, rw)
+        if not path.endswith(".git"):
+            path = "{}.git".format(path)


is the .git ending of a path really mandatory with GitBucket? I find that quite surprising.

For the note, Github support both with and without the extension (i.e. https://github.com/guyzmo/git-repo.git or https://github.com/guyzmo/git-repo)

I know it.

In GitBucket,
for ssh access, .git is required.
for http(s) access, GitBucket's formal http(s) URL format is http://server:port/git/owner/repo.git.
But it also can access with:

http://server:port/git/owner/repo

http://server:port/owner/repo.git

But it can't access by http://server:port/owner/repo. Then it always add .git.

Recently, I found problem with git-2.12. I wrote it in gitbucket/gitbucket#1552.

In GitBucket-4.12, http://server:port/owner/repo.git returns redirect to http://server:port/git/owner/repo.git. git-2.12 doesn't allow redirect url.
so I will rewrite format_path such as:

never call super(...).format_path

returns ssh://user@server:port/owner/repo.git for rw=True

returns http://server:port/git/owner/repo.git for rw=False

ok, makes sense

guyzmo · 2017-04-29T14:41:36Z

git_repo/services/ext/gitbucket.py

+@register_target('bucket', 'gitbucket')
+class GitbucketService(GithubService):
+    fqdn = "localhost"
+    port = 8080


is there a public instance we can default to, for testing and tweaking?

There is demo site at https://gitbucket.herokuapp.com/
But I don't know who maintains it, who pays money for heroku's charge. So I can't decide it to default.

And it's not opened for accounts. It'd be nice to find an almost open instance, like gogs do, to default to it for tests and all.

try.gogs.io is down now...

it's nice that we recorded the tests before it went down 😁

guyzmo

can you implement also the tests?

you can basically copy/paste the github ones, and guard all the tests that are not implement with:

with pytest.raises(NotImplemented):
    ...

so they're already there for when they will be done.

Could you also add script(s) that can automatically instanciate a gitbucket instance, like for gogs?

guyzmo · 2017-04-30T00:34:10Z

can you rebase this branch on top of the devel branch of guyzmo/git-repo?

guyzmo · 2017-04-30T00:54:20Z

ok, so I've rebased this branch on devel in the branch requests/github/144 on my repository. I'll need you to change the current branch with the one I've added, so I'll need you to do the following.

Let's consider the following two remotes:

github points to your repository kounoike/git-repo
upstream points to my repository guyzmo/git-repo

if it's not like that, adapt accordingly to what you chose

# first make sure you're in this PR's branch
git checkout pr-gitbucket
# then make a backup of it
git checkout -b pr-gitbucket-backup
# remove the branch
git branch -D pr-gitbucket
# fetch the branch I've pushed
git fetch upstream requests/github/144
# check out that branch
git checkout upstream/requests/github/144
# store that branch locally as the new pr-gitbucket
git checkout -b pr-gitbucket
# and force push it to your github project
git push -f github pr-gitbucket
# and now this PR is rebased on the new main devel branch

PS: you can use git-repo to add the github and upstream branches easily:

git hub add github kounoike/git-repo
git hub add -a upstream guyzmo/git-repo

😉

guyzmo · 2017-04-30T01:02:25Z

Once the above is done, I'll need you to:

fix __init__()/connect() scheme
update get_auth_token() as I suggest
add tests to this branch
- and a deployment script for gitbucket would be nice to have, to replicate the tests when needed

and BTW, thank you for your care so far 🙌
and don't hesitate to join our IRC channel on freenode #git-repo

…model.

…at_path.

guyzmo · 2017-04-30T17:00:42Z

git_repo/services/ext/github.py

+        if self.fqdn != GITHUB_COM_FQDN:
+            # upgrade self.gh from a GitHub object to a GitHubEnterprise object
+            gh = github3.GitHubEnterprise(RepositoryService.build_url(self))
+            self.gh._session.base_url = gh._session.base_url


this is wrong, it should be the other way around:

gh._session.base_url = self.gh._session.base_url

OK I fixed it.

self.gh must change to new one(GitHubEnterprise)

self.gh._session must keeps old one(created by init)

self.gh._session.base_url must change to new URL

gh = github3.GitHubEnterprise(RepositoryService.build_url(self)) self.gh._session.base_url = gh._session.base_url gh._session = self.gh._session self.gh = gh

fixes #144 fixes #142

@kounoike

First anniversary release 🎂 This release marks the first anniversary and 10th release of git-repo 🎉 Now git-repo supports 6 git services: github, gitlab, bitbucket, gogs, gitea and gitbucket It supports the following commands: clone, fork, create, delete, add, open, ls, request, gist And it's becoming relatively stable (keep sending bug reports 🙏) ♥ Contributors Thanks to @kounoike for the gitbucket support 🙌 Now there are three contributors promoted to collaborators: * @kounoike, @pyhedgehog and @Crazybus 🚧 Features * When add has no parameters, add default remote #100 #141 * When add has 'upstream' parameter, add the upstream remote #99 #141 * Use default branch instead of hardcoded 'master' #91 * Refactor and complete bitbucket support #43 #11 #80 #14 #89 #90 #79 * Loosen versions of dependencies #133 * Report if the service is already setup #136 * Adds GitBucket support #142 #144 * Adds support for ssh:// URL #148 🚒 Bugfixes * fix various crashes #152 * fix gitlab clone crash #129 * fix request create command #147 #127 * fix request fetch command #138 #131 * fix wizard crash #149 * fix mishandling of renamed gitlab project urls #137 📝 Documentation * documentation #145 Signed-off-by: Guyzmo <guyzmo+github+pub@m0g.net>

guyzmo assigned guyzmo and kounoike Mar 21, 2017

guyzmo self-requested a review March 21, 2017 13:44

guyzmo added backlog enhancement labels Mar 21, 2017

guyzmo added this to the 1.10 milestone Mar 21, 2017

guyzmo force-pushed the devel branch 2 times, most recently from 1bf4c23 to cebf92d Compare April 28, 2017 21:31

kounoike force-pushed the pr-gitbucket branch from 8cd45c1 to 08d3ea2 Compare April 29, 2017 11:51

guyzmo reviewed Apr 29, 2017

View reviewed changes

kounoike force-pushed the pr-gitbucket branch from 08d3ea2 to da86444 Compare April 29, 2017 13:44

guyzmo reviewed Apr 29, 2017

View reviewed changes

guyzmo requested changes Apr 29, 2017

View reviewed changes

guyzmo force-pushed the devel branch from 6d72245 to b14e648 Compare April 29, 2017 23:44

kounoike mentioned this pull request Apr 30, 2017

Add support ssh:// URL and github local branch name fix. #148

Closed

kounoike added 3 commits April 30, 2017 02:43

💄 Minor improvements of the slug detection code

6412396

Change local branch name to remote repo name.

79cb879

Add GitBucket support.

e48a3b7

guyzmo added in progress and removed backlog labels Apr 30, 2017

kounoike force-pushed the pr-gitbucket branch from da86444 to e48a3b7 Compare April 30, 2017 01:22

kounoike added 8 commits April 30, 2017 23:27

Change __init__/connect scheme with create GitHub and upgrade to GHE …

e5af447

…model.

Change get_auth_token for prompt to user, delete connect, custom form…

ec3a83d

…at_path.

Create initialize local GitBucket instance's data script.

03bf93c

test_gitbucket.py first version.

57900fd

fix token environment variable name.

830dc5f

fix test

d9574c8

start ssh server.

10e515a

enable ssh-url, GITBUKCET_NAMESPACE

1947f36

guyzmo reviewed Apr 30, 2017

View reviewed changes

kounoike added 5 commits May 1, 2017 02:16

keep _session of GitHub object. reuse it by GitHubEnterprise.

4c22986

fix test.

9702d77

Add gitbucket cassettes.

215c92c

recreate gitbucket cassettes.

b63d440

recreate gitbucket cassettes again. change "user" to "git-repo-user"

b8ce99b

guyzmo merged commit b8ce99b into guyzmo:devel Apr 30, 2017

guyzmo added a commit that referenced this pull request Apr 30, 2017

Merge branch requests/github/144 into devel

2dcd901

fixes #144 fixes #142

guyzmo removed the in progress label Apr 30, 2017

Add GitBucket support (refs #142) #144

Add GitBucket support (refs #142) #144

Conversation

kounoike commented Mar 20, 2017

guyzmo commented Apr 28, 2017

kounoike commented Apr 29, 2017

guyzmo commented Apr 29, 2017

kounoike commented Apr 29, 2017

Choose a reason for hiding this comment

kounoike Apr 29, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kounoike Apr 29, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

guyzmo left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

guyzmo Apr 29, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

guyzmo left a comment • edited Loading

Choose a reason for hiding this comment

guyzmo commented Apr 30, 2017

guyzmo commented Apr 30, 2017 • edited Loading

guyzmo commented Apr 30, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kounoike Apr 29, 2017 •

edited

Loading

kounoike Apr 29, 2017 •

edited

Loading

guyzmo Apr 29, 2017 •

edited

Loading

guyzmo left a comment •

edited

Loading

guyzmo commented Apr 30, 2017 •

edited

Loading