Add GitLab repo sync and webhook support #1870

saily · 2015-12-20T17:51:39Z

still work in progress, but want to discuss with core team within this pull.

Implementation plan

Merge pull gitlab provider Gitlab provider pennersr/django-allauth#1231 into django-allauth.
Sync repositories after connecting social account gitlab.
Create a remote webhook in GitLab.
Add more tests
Update documentation

ericholscher · 2015-12-21T20:29:55Z

Sounds like a good plan! Thanks for working on this. Let us know if you have questions or need guidance. I know @agjohnson has done some work on these areas recently, so he might have some thoughts as well.

agjohnson · 2015-12-21T20:35:24Z

I was just talking about the ability to extend this to Gitlab, thanks @saily for digging into it.

I'm not sure on the best way to handle this. I'm currently in the same code, have refactored some already, and really wish to refactor the rest of the existing code, as it's a mess. It really should be written in a more constructed pattern, and there are some missing relationships that should have been added.

I'll try to get my work updated shortly, we might need to address a refactor in this PR before merging.

ghost · 2015-12-25T00:41:22Z

Can this be added to the GitLab milestone please?

saily · 2015-12-29T11:20:46Z

good news, pennersr/django-allauth#1231 was merged into master.

agjohnson · 2016-01-02T00:09:36Z

I've updated my work in #1850, and that PR is now in for review. We should get to it next week. That PR would be a good basis for this work, as the abstractions for OAuth services are structured cleanly there. I've already pinned django-allauth in #1850, as I had some changes required there as well. #1893 addresses pinning to the next release.

saily · 2016-01-09T11:42:40Z

readthedocs/oauth/constants.py

 OAUTH_SOURCE_BITBUCKET = 'bitbucket'

 OAUTH_SOURCE = (
    (OAUTH_SOURCE_GITHUB, _('GitHub')),
+    (OAUTH_SOURCE_GITLAB, _('GitLab')),


I'm not sure this is required, @agjohnson can you please check.

Yeah, this isn't required, it should have been refactored out. Dropped in #1912

ok, i'll drop it and update my PR

saily · 2016-01-09T11:50:03Z

@ericholscher, @agjohnson, please review.

I'll have to dig through your test-setup to implement some tests for this.
I saw you're using some s3 stuff in github/bitbucket testcases, don't you mock your request calls?
There some pretty good libs out there, example HTTMock.

agjohnson · 2016-01-09T22:58:27Z

@saily the test cases at https://github.com/rtfd/readthedocs.org/blob/master/readthedocs/rtd_tests/tests/test_oauth.py are only using fixture data responses, we're not mocking the requests. This might be a good addition though. We mock requests and responses without httpmock in other tests.

agjohnson · 2016-01-09T23:01:24Z

readthedocs/oauth/services/gitlab.py

+            repo.ssh_url = fields['ssh_url_to_repo']
+            repo.html_url = fields['web_url']
+            repo.private = not fields['public']
+            repo.clone_url = fields['http_url_to_repo']


clone_url should be set to ssh_url if this is a private repository.

scphantm · 2016-02-22T18:20:21Z

is there a good point where i can jump in on this and give a hand. This issue is holding up our RTD implementation.

agjohnson · 2016-02-26T21:52:43Z

@saily @scphantm The above feedback, along with test cases are still required. I don't have too much extra bandwidth to focus on this, but more than happy to guide anyone willing to wrap this work up. Gitlab integration would be a great addition!

BuddhaOhneHals · 2016-08-05T10:00:42Z

@saily I added some tests and documentation for the GitLab integration:
https://github.com/galileo-press/readthedocs.org/commits/sync-gitlab-repos

There's a PR: saily#1

Sync gitlab repos

…ries

BuddhaOhneHals · 2016-08-10T11:07:22Z

@saily @agjohnson I addressed some of the annotations here: saily#2

agjohnson · 2016-08-18T22:50:05Z

@BuddhaOhneHals thanks for picking up the last leg of this PR!

Because this PR is sizable, not very digestible, and has been active for such a long time, I think we should try testing this locally some as well. If things look good after a final review and some local testing, I think we can get this finally merged and deployed!

BuddhaOhneHals · 2016-08-19T06:46:28Z

@agjohnson sounds great! If you need any help please let me know.

connorshea

Some comments for this PR. If we can help in any way with getting this pushed through, we'd be happy to.

connorshea · 2016-09-26T23:13:14Z

docs/webhooks.rst

+GitLab
+------
+
+If your project is hosted on GitLab, you can manually set the webhook on Gitlab and


This should be formatted as GitLab

what do you mean here?

GitLab branding is to always call it GitLab.com (though I can't find their public documentation on that at this time).

However it's usually referred to as GitLab.

connorshea · 2016-09-26T23:14:16Z

readthedocs/oauth/services/gitlab.py

+        :param url: start url to get the data from.
+        :param kwargs: optional parameters passed to .get() method
+
+        See http://doc.gitlab.com/ce/api/README.html#pagination


This URL should be updated to https://docs.gitlab.com/ce/api/README.html#pagination

connorshea · 2016-09-26T23:14:59Z

readthedocs/oauth/services/gitlab.py

+        """
+        session = self.get_session()
+
+        # See: http://doc.gitlab.com/ce/api/projects.html#add-project-hook


Ditto for this one.

connorshea · 2016-09-26T23:18:10Z

readthedocs/oauth/services/gitlab.py

+
+        :param fields: dictionary of response data from API
+        :param privacy: privacy level to support
+        :param organization: remote organization to associate with


Organizations are called Groups in GitLab, not sure if that matters or if that'd be confusing for users.

I think Groups would be better here.

@connorshea, @destroyerofbuilds of course you're both right, but i had to adapt the given api and this api uses organization. My recommendation would be to get this into master and then discuss how to refactor the oauth-plugin system.

organization in this case is our internal representation of the various providers names for orgs/groups/what have you. I don't think this usage needs to reflect GitLab's naming.

connorshea · 2016-09-26T23:19:36Z

readthedocs/oauth/services/gitlab.py

+        :type organization: RemoteOrganization
+        :rtype: RemoteRepository
+        """
+        # See: http://doc.gitlab.com/ce/api/projects.html#projects


And this URL as well.

Support private repositories and cleanup

ghost · 2016-10-19T16:06:11Z

docs/webhooks.rst

+------
+
+If your project is hosted on GitLab, you can manually set the webhook on Gitlab and
+point it at ``https://readthedocs.org/gitlab``:


Should this webhook be updated to reflect the new webhook end points added in #2433 ?

So, without a webhook management page, these webhooks aren't completely deprecated yet. Once that feature exists, this page will need to be updated to instead reference that page and drop information on manual set up. Likewise, we should add a deprecation notice on GitHub Services for RTD as well. I'll open an issue outlining the work that will go into this one -- fine to point to this webhook endpoint for now.

agjohnson · 2016-10-20T01:02:51Z

readthedocs/oauth/services/gitlab.py

+            'merge_requests_events': False,
+            'note_events': False,
+            'tag_push_events': True,
+            'url': u'https://{0}/gitlab'.format(settings.PRODUCTION_DOMAIN),


In #2433 these webhook endpoints were deprecated. A new handler for GitLab will be required and this should be updated. We can either address this in the PR, or I'm also +1 on filing a new PR to address this change.

Oh, uh, I already implemented this.

Nothing to see here.

agjohnson · 2016-10-20T01:06:42Z

docs/webhooks.rst

+------
+
+If your project is hosted on GitLab, you can manually set the webhook on Gitlab and
+point it at ``https://readthedocs.org/gitlab``:


So, without a webhook management page, these webhooks aren't completely deprecated yet. Once that feature exists, this page will need to be updated to instead reference that page and drop information on manual set up. Likewise, we should add a deprecation notice on GitHub Services for RTD as well. I'll open an issue outlining the work that will go into this one -- fine to point to this webhook endpoint for now.

This is old and misleading. Folks should generally just be using the GitHub & BitBucket hooks.

segilbert · 2016-12-12T21:20:55Z

My team is looking to leverage GitLab integration. When do you expect this merge request will be processed and released into the wild? Thanks for all this work. Looking forward to this feature!

parmentelat · 2017-03-27T08:23:38Z

I was interested in using this feature, that's why I started with asking a related question on stackoverflow:
http://stackoverflow.com/questions/43023051/creating-a-readthedocs-io-repo-in-sync-with-a-public-gitlab-repo

Rephrasing the question here: in the meanwhile (until this PR makes it to deployment on gitlab.com), is there a way to achieve the same result (pushes on gitlab triggers a build on readthedocs) by manually adding a webhook on the gitlab side ?

ghost · 2017-03-27T15:12:05Z

@parmentelat you can setup a webhook on GitLab to trigger a build on readthedocs for a project hosted on GitLab.

The webhook already existed, but the documentation was never upstreamed.

I'm contributing @takotuesday's documentation work in #2747. Please take a look at that documentation on how to get started with configuring your project with a webhook.

parmentelat · 2017-03-27T19:56:30Z

@destroyerofbuilds : awesome, works like a charm ! Thank You :)

agjohnson added the PR: work in progress Pull request is not ready for full review label Dec 21, 2015

agjohnson added this to the Gitlab milestone Dec 27, 2015

agjohnson changed the title ~~WIP: Add basic GitLab repo sync support~~ Add GitLab repo sync and webhook support Dec 27, 2015

saily mentioned this pull request Jan 2, 2016

Changes to support Bitbucket OAuth2 #1850

Merged

5 tasks

saily force-pushed the sync-gitlab-repos branch from 7b37bc2 to e6de58d Compare January 9, 2016 11:41

saily reviewed Jan 9, 2016
View reviewed changes

agjohnson reviewed Jan 9, 2016
View reviewed changes

agjohnson added Needed: documentation Documentation is required Needed: tests Tests are required labels Feb 26, 2016

agjohnson mentioned this pull request Mar 18, 2016

Hook for GitLab #1068

Closed

Daniel Widerin and others added 5 commits June 16, 2016 07:11

Add GitLab repo sync and webhook support

ef9c792

oauth: gitlab: use unicode whenever format is used

cf5c47c

oauth: gitlab: prevent duplicate entries after sync

2356d84

oauth: gitlab: avoid KeyError when owner is missing

7113e94

oauth: gitlab: setup_webhook must return a tuple

5d7d6c1

Added some documentation for the Gitlab integration

aec6941

Merge pull request #1 from galileo-press/sync-gitlab-repos

9152bb6

Sync gitlab repos

BuddhaOhneHals mentioned this pull request Aug 5, 2016

Setup GitLab Web Hook on Project Import #1443

Closed

Timo Steidle added 6 commits August 5, 2016 14:11

Improved url_pattern of the GitLabService to support private reposito…

299a2b5

…ries

Fixed linting error

8a10848

Simplified exception handling

97a4fb1

Use PRODUCTION_DOMAIN as webhook id

dff4d7d

Make imports python3 compatible

4fbe6bb

Use the GitLab project ID, not the PRODUCTION_DOMAIN

50285fe

agjohnson mentioned this pull request Aug 18, 2016

Feature request: Add gitlab as connected service on readthedocs.org #2374

Closed

agjohnson removed Needed: documentation Documentation is required Needed: tests Tests are required labels Aug 18, 2016

connorshea reviewed Sep 26, 2016

View reviewed changes

Merge pull request #2 from galileo-press/sync-gitlab-repos

e288bf3

Support private repositories and cleanup

ghost reviewed Oct 19, 2016

View reviewed changes

Daniel Widerin and others added 2 commits October 19, 2016 18:12

Update urls as mentioned in comments

6c8d1c3

Update webhook docs as mentioned in comments

0cb1ed5

agjohnson reviewed Oct 20, 2016

View reviewed changes

jessetan referenced this pull request Dec 12, 2016

Remove the Post Commit Hook bit on project detail.

adb26ce

This is old and misleading. Folks should generally just be using the GitHub & BitBucket hooks.

humitos mentioned this pull request Apr 18, 2017

Support for Visual Studio Team Services #2798

Closed

humitos mentioned this pull request Nov 28, 2017

GitLab Integration #3327

Merged

3 tasks

agjohnson closed this in #3327 Dec 7, 2017

Add GitLab repo sync and webhook support #1870

Add GitLab repo sync and webhook support #1870

Conversation

saily commented Dec 20, 2015

Implementation plan

ericholscher commented Dec 21, 2015

agjohnson commented Dec 21, 2015

ghost commented Dec 25, 2015

saily commented Dec 29, 2015

agjohnson commented Jan 2, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

saily commented Jan 9, 2016

agjohnson commented Jan 9, 2016

Choose a reason for hiding this comment

scphantm commented Feb 22, 2016

agjohnson commented Feb 26, 2016

BuddhaOhneHals commented Aug 5, 2016

BuddhaOhneHals commented Aug 10, 2016

agjohnson commented Aug 18, 2016

BuddhaOhneHals commented Aug 19, 2016

connorshea left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ghost Oct 19, 2016 • edited by ghost

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

segilbert commented Dec 12, 2016

parmentelat commented Mar 27, 2017

ghost commented Mar 27, 2017

parmentelat commented Mar 27, 2017

ghost Oct 19, 2016 •

edited by ghost