Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Gitlab support #399

Open
greysteil opened this Issue Apr 27, 2018 · 29 comments

Comments

Projects
None yet
@greysteil
Copy link
Member

greysteil commented Apr 27, 2018

From @sobolevn on January 14, 2018 9:56

This is a big feature. Right now you only support GitHub, which is the biggest open-source community out there. But a lot of developers are using GitLab for private projects.

Motivation

We (wemake.services) host all our open-source project on GitHub. But, most of our software projects are private and hosted on GitLab (both self-hosted and SaaS versions).

While we really enjoy dependency upgrades in our open-source projects, our main projects are still upgraded by hands, which sometimes takes a lot of time.

Benefits for you

Why should you even consider adding GitLab? There are some key points I want you to consider:

  1. GitLab is popular. Here are the usage statics and market share:
    bitrise-self-hosted-chart
  2. GitLab is used primarily for private projects, so the would be more paying customers
  3. GitLab has CI, so it would be very useful to integrate tests that dependency can be updated

What do you think about this feature?
Do you consider adding new providers except GitHub?

Thanks!

Copied from original issue: dependabot/feedback#75

@greysteil

This comment has been minimized.

Copy link
Member Author

greysteil commented Apr 27, 2018

Thanks for the request. This is, indeed, a big one!

I'd love Dependabot to support GitLab (and Bitbucket). Doing so would require:

  • Changes to dependabot-core to make all classes host-provider agnostic. This should be relatively straightforward - I estimate ~ 1 day's work
  • Changes to dependabot-frontend (private) to handle auth for GitLab (and Bitbucket), which is ~ 1 day's work
  • Changes to dependabot-backend (private) to listen to use the GitLab API (when listing repos, asking for CI statuses, etc.) and listen for GitLab webhooks. There are a lot of unknowns here, so I think it's ~ 5 day's work.

Whilst I'd love to promise the above is coming, right now I'm not working on it. Dependabot doesn't make enough to support me working on it in more than my spare time (our revenue numbers are in this interview and right now they're less than $1,000 / month), so I can't justify spending more than a few hours on it a day.

I almost certainly will get to this eventually, but no in the next couple of months. Sorry!

@greysteil

This comment has been minimized.

Copy link
Member Author

greysteil commented Apr 27, 2018

From @cafferata on March 31, 2018 19:57

Hi @greysteil, I think the concept of Dependabot is very good. I have not come across it before and I wanted to get started right away. Until I came to the conclusion that there is no Gitlab integration. The consequence? I can not actually test it properly.

I've read your above reaction (and the Indie Hackers blog post), and asked myself: what would @greysteil need to spend more time on Dependabot? Would a promise by X number of Gitlab participants be a good stick? Would it help if, for example, we pay a year in advance so that healthy cash flow arises? We, as @JCID, are always willing to pay for tools and I think many of our competitors.

@greysteil

This comment has been minimized.

Copy link
Member Author

greysteil commented Apr 27, 2018

Thanks for these extremely motivating words @cafferata, and apologies for the delay getting back to you.

I've been spending a bit more time on Dependabot over the last few weeks, and adding GitLab support is rising up my priority list. There are a couple of things ahead of it (getting Java support out of beta and fleshing out our security advisory sources), but I should be able to prioritise this soon.

It would help a lot to have a list of companies that would be committed to pay for Dependabot if/when I do the work. Adding GitLab support will be a bunch of up-front work, but it's also likely to increase the maintenance burden for me on Dependabot, and I want to be sure that's worth it.

If your company would be up for paying for a small org / unlimited account on GitLab drop a comment below, and I'll try to get to this a little bit sooner.

@greysteil

This comment has been minimized.

Copy link
Member Author

greysteil commented Apr 27, 2018

From @cafferata on April 5, 2018 18:59

If your company would be up for paying for a small org / unlimited account on GitLab drop a comment below

@JCID will start with the first 5 projects and I expect that this is soon too small for our organization, after which we will go to the unlimited account. So, who is next? 👍

@greysteil

This comment has been minimized.

Copy link
Member Author

greysteil commented Apr 27, 2018

From @codisart on April 9, 2018 21:22

@greysteil I am also very interested to see this integrated with gitlab. But except the technical part (get the files from the repo, create the commit etc...) it seems to me that there others things to clarify on the integration with gitlab. Do you have an idea of the way you would get the credentials needed for the repo and of the way to pay for dependabot since it seems linked to github marketplace for now ?

@greysteil

This comment has been minimized.

Copy link
Member Author

greysteil commented Apr 27, 2018

There are certainly some open questions on how GitLab will work, but high level I'm thinking:

  • Ask for an API key, rather than using OAuth2
    • It doesn't cause problems if individuals leave a team
    • As far as I know it's how other applications do this
  • Handle billing from within Dependabot
    • I've asked GitLab previously whether they have any intention to build a marketplace and they've said "no".
    • Will require a Stripe integration and a frontend for switching plan within Dependabot

I don't use GitLab myself, so am slightly blind on the above right now. Obviously I'll rectify that when I kick off work on the GitLab integration!

@greysteil greysteil changed the title Gitlab support [Feature request] Gitlab support Apr 27, 2018

@greysteil greysteil changed the title [Feature request] Gitlab support Gitlab support Apr 27, 2018

@ylac

This comment has been minimized.

Copy link

ylac commented Aug 13, 2018

Hi @greysteil, I work for a small dev agency (~15 devs) that could really use Dependabot to manage our React/React Native dependencies. We use Gitlab atm so would be very interested in signing up if you add the integration.

@connorshea

This comment has been minimized.

Copy link

connorshea commented Dec 3, 2018

MR where I add Dependabot to my Jekyll blog: https://gitlab.com/connorshea/connorshea.gitlab.io/merge_requests/5

Credit to @deivid-rodriguez for most of this implementation (in https://gitlab.com/deivid-rodriguez/nipanipa)

I couldn't find anything like this elsewhere, so I figured I'd write it up myself for others to work from.


So here's a "hack" to get Dependabot working in GitLab right now (I apologize that this is probably gonna be a long comment):

  • Have GitLab CI set up (it should be easy for gitlab.com, if you're on a custom instance maybe less so)

  • Add dependabot-omnibus to your Gemfile (if you don't have one, read up on Bundler) like this:

# See the github repo for the current most recent tag, and probably use that.
gem "dependabot-omnibus", git: "https://github.com/dependabot/dependabot-core", tag: "v0.85.2"
  • Run bundle install to add/update your Gemfile.lock.

  • Add dependabot-script as a submodule with git submodule add https://github.com/dependabot/dependabot-script vendor/dependabot-script (this should create a .gitmodules file and a vendor/dependabot-script file. (Note you may need to get an older version of the dependabot-script submodule for this to work if there are any future breaking changes this comment doesn't take into account. Currently I'm using the version from December 13th, 2018.)

  • Create a file called bin/dependabot.rb (you can put this wherever you want, just make sure the .gitlab-ci.yml calls the right file and that you update the paths in this script to match their location relative to the script) with the following contents:

#!/usr/bin/env ruby
# frozen_string_literal: true

ENV["BUNDLE_GEMFILE"] ||= File.expand_path("../Gemfile", __dir__)

require "bundler/setup"

load File.expand_path("../vendor/dependabot-script/generic-update-script.rb", __dir__)
  • Add a dependabot job like the following to your .gitlab-ci.yml file (if you're using a package manager other than Bundler, you may need to change that variable, you may also need to customize GITLAB_HOST_NAME):
dependabot:
  # Uncomment this if you're using a different image for other jobs in the CI YML.
  # You might need a custom image if you're trying to upgrade dependencies in languages besides Ruby (not sure!)
  # image: ruby:2.5
  stage: build
  variables:
    GIT_SUBMODULE_STRATEGY: normal
    PROJECT_PATH: $CI_PROJECT_PATH
    PACKAGE_MANAGER: bundler
    GITLAB_HOST_NAME: gitlab.com
  before_script:
    - bundle install
  script:
    - ruby bin/dependabot.rb
  only:
    - schedules
  • You should also exclude unnecessary jobs from your scheduled runs by adding an exclude attribute to all your other jobs, like so:
job_name:
  script:
    - echo 'do stuff'
  except:
    - schedules
  • Commit your changes and open a merge request
  • In GitLab, create a Personal Access Token in the user settings, it'll need api access. Copy the token to your clipboard. This will allow Dependabot to create new merge requests using your user account (if you want to create a separate user for this purpose, you can do that too).
  • In the project, go to the sidebar and select CI/CD > Schedules. Read more about Pipeline Schedules here.
  • Create a new schedule with the following settings:
    • Right now, set the target branch to your dependabot branch, later you should change that to master.
    • Set the frequency to 0 4 * * *, which means "Every day at 4am" (or whatever other value you want)
    • Add a variable named GITLAB_ACCESS_TOKEN, set its value to the Access Token you copied earlier.
  • Save the schedule, and back on the pipeline schedules page you should see the schedule you just set up. Now, hit the "play" button to run it manually. If you've done everything correctly, it'll open a bunch of MRs that update your dependencies.

Make sure to change the branch to master once the merge request adding dependabot has been merged.

If you run into any problems and figure out how to fix them, please share with the class ;)

@connorshea

This comment has been minimized.

Copy link

connorshea commented Dec 3, 2018

Side note: It may be a lot easier/better to just copy-paste the generic-update-script.rb file into your repository, rather than using submodules, and just occasionally check to see if you need to update the script to match the upstream.

Unfortunately dependabot-script is unlicensed, so I don't know if there are any problems with that (or even with the submodule usage).

@deivid-rodriguez

This comment has been minimized.

Copy link

deivid-rodriguez commented Dec 3, 2018

Just note that this solution has the gotcha of creating a PR for every dependency, not only top level ones, so you end up with too many PRs, and thus you usually end up with a bunch of redundant PRs. I guess this can be fixed in the dependabot script, but I haven't looked into it.

@codisart

This comment has been minimized.

Copy link
Contributor

codisart commented Dec 3, 2018

@connorshea Very nice documentation, it should really help people.
I used the same process on Gitlab CI but since we work with PHP and node.js at my work, I put dependabot script and all ruby dependencies in a docker container. If anyone needs more details, I can take some time to outline them.

@deivid-rodriguez On my experience, neither in php nor node we had redundancy problems, maybe it is specific for ruby. Our bigger pain point is when some big package like symfony framework do a release, it usually for all their packages at the same time, so we get 20-40 PRs at the same time.

@greysteil

This comment has been minimized.

Copy link
Member Author

greysteil commented Dec 3, 2018

Thanks for this @connorshea!

Using a submodule is definitely a good shout - in future you'll be able to set Dependabot to keep that submodule up-to-date and it will automatically handle breaking changes for you (there shouldn't be many, but we've got a namespacing change coming down the pipe this week).

On license and top-level dependencies - I'll make those change to Dependabot Script now.

Update: License added, and the script will now filter for top-level dependencies only.

@connorshea

This comment has been minimized.

Copy link

connorshea commented Dec 3, 2018

Thanks @greysteil!

The only other major caveats right now are that this won’t update the MRs after creating them as Dependabot core doesn’t support this for GitLab yet, and that there’s no way to automatically have Dependabot merge the MR if CI passes.

The latter should be relatively easy to fix by enabling “Merge when pipeline succeeds” when the MR is created, which I’m fairly certain is possible to do via the API. The former is a lot harder :)

Also, I’d need to check but I don’t think Dependabot deletes the branches after merging. There’s a toggle for “Delete source branch upon merge”, which should work as a fix for that. We just need to update Dependabot-core and/or script to use those toggles :)

@deivid-rodriguez

This comment has been minimized.

Copy link

deivid-rodriguez commented Dec 3, 2018

Regarding this bit

  script:
    # I don't entirely understand why this is necessary, if anyone can find
    # a better workaround that lets the runner run the dependabot executable
    # that'd be cool.
    # REPLACE "connorshea" WITH YOUR USERNAME
    - find /builds/connorshea/ -type d -exec chmod 755 {} \;
    - bin/dependabot

I added this workaround because files were not being cloned with the right permissions. I'm not sure why but I think it's because of https://gitlab.com/gitlab-org/gitlab-runner/issues/1736.

@connorshea

This comment has been minimized.

Copy link

connorshea commented Dec 3, 2018

I wonder if you could name it dependabot.rb and then run ruby dependabot.rb to have GitLab let you run the script?

@deivid-rodriguez

This comment has been minimized.

Copy link

deivid-rodriguez commented Dec 3, 2018

Yeah, that would probably work too.

@greysteil

This comment has been minimized.

Copy link
Member Author

greysteil commented Dec 13, 2018

FYI, we've been making some changes to the structure of Dependabot Core that will affect anyone pulling it in using a script.

We're creating separate gems for each language, tied together by the dependabot-omnibus gem (which will probably be renamed once the transition is complete). For now you probably want to switch references to dependabot-core to dependabot-omnibus in your Gemfiles. You shouldn't notice any other changes.

@connorshea

This comment has been minimized.

Copy link

connorshea commented Dec 16, 2018

I've updated my comment to take that change into consideration and also to simplify/remove the unnecessary permissions stuff :)

@LeoColomb

This comment has been minimized.

Copy link
Contributor

LeoColomb commented Dec 23, 2018

The latter should be relatively easy to fix by enabling “Merge when pipeline succeeds” when the MR is created

@connorshea #857 😽 (dependabot/dependabot-script#169)

Also, I’d need to check but I don’t think Dependabot deletes the branches after merging

It does.

@jeremycook

This comment has been minimized.

Copy link

jeremycook commented Dec 31, 2018

We self host GitLab with more than 90 .NET projects. GitLab's dependency scanner does not support NuGet at this time. I like your pull request system, and the idea of being GitLab/GitHub/Team Services agnostic.

@gitfool

This comment has been minimized.

Copy link

gitfool commented Jan 22, 2019

If I understand correctly, the hack above is to add Dependabot support inline to the CI pipeline of a single GitLab project. Is there a recommended way to add Dependabot to a private / internally hosted GitLab site for all projects?

@connorshea

This comment has been minimized.

Copy link

connorshea commented Jan 22, 2019

@gitfool you might be able to abuse GitLab CI YML's import functionality, but generally I'd say no there isn't.

@LeoColomb

This comment has been minimized.

Copy link
Contributor

LeoColomb commented Jan 22, 2019

@gitfool @connorshea Did you take a look at the dependabot/dependabot-script?
You definitely can use this repo to check update for many repo, and there is even a .gitlab-ci.yml example.
Hack-free, import-free, independent runs. 🎉

@connorshea

This comment has been minimized.

Copy link

connorshea commented Feb 18, 2019

FWIW I'm not sure if my previous instructions still work since dependabot has been significantly refactored in the last few months. Has anyone managed to use my guide to get it working in the last couple weeks or do I need to make adjustments? :)

Sidenote: @LeoColomb is the purpose (or one of the extra benefits) of #986 to support this use case more easily?

@LeoColomb

This comment has been minimized.

Copy link
Contributor

LeoColomb commented Feb 18, 2019

or one of the extra benefits

Yes, it is. Reduce runner usage, more flexibility and isolation, speed up builds.

@phil-lgr

This comment has been minimized.

Copy link

phil-lgr commented Mar 14, 2019

@greysteil

Whilst I'd love to promise the above is coming, right now I'm not working on it. Dependabot doesn't make enough to support me working on it in more than my spare time (our revenue numbers are in this interview and right now they're less than $1,000 / month), so I can't justify spending more than a few hours on it a day.

what about changing to 1-5 priv repo max, I'd love to throw $$$$ at you! Dependabot saves me.. what maybe 1-2 hours per week, at my current rate you are making me and my clients save $320-640 per month or more like $3,840-7,680 per year! NO kidding there

@philipmeadows

This comment has been minimized.

Copy link

philipmeadows commented Mar 26, 2019

We would be interested in gitlab support.

@minhngocd

This comment has been minimized.

Copy link

minhngocd commented Mar 26, 2019

The client I am currently on is also interested in GitLab support

@bradleygolden

This comment has been minimized.

Copy link

bradleygolden commented Apr 8, 2019

I'm interested in GitLab support as well! I played around with the scripts at dependabot/dependabot-script and was able to get merge requests working in GitLab as a toy experiment.

My approach was to create a docker image called <somename>/dependabot-script that housed the dependabot scripts. That dockerfile looks like this:

FROM dependabot/dependabot-core:latest

WORKDIR /workspace
COPY DependabotGemfile.rb .
COPY generic-update-script.rb .
COPY update-script.rb .
COPY run.sh .

RUN BUNDLE_GEMFILE=DependabotGemfile.rb bundle install -j $(nproc) --path vendor

Run.sh looks like this:

cd /workspace
BUNDLE_GEMFILE=DependabotGemfile.rb bundle exec ruby /workspace/generic-update-script.rb

I then created a GitLab CI template called .dependabot-template.yaml that used the docker container with your scripts:

# GitLab CI configuration for Dependabot
#
# Usage:
# * Set the required global variables used in `./generic-update-script.rb`
#   https://docs.gitlab.com/ee/ci/variables/#variables
#
# * Create a pipeline schedule for each managed repository
#   https://docs.gitlab.com/ee/user/project/pipelines/schedules.html
#
# * Set in the schedule required variables
#
#     PROJECT_PATH = group/repository
#     PACKAGE_MANAGER_SET = bundler,composer,npm_and_yarn
#
# https://github.com/dependabot/dependabot-script
# https://docs.gitlab.com/ee/ci/yaml/

.dependabot:
  image: somename/dependabot
  variables:
    PACKAGE_MANAGER: $CI_JOB_NAME
  script:
    - sh /workspace/run.sh
  cache:
    paths:
      - vendor/
  only:
    - schedules

bundler:
  extends: .dependabot
  only:
    variables:
      - $PACKAGE_MANAGER_SET =~ /\bbundler\b/
...

In the end, I can add the following to any project's gitlab-ci.yml with the PACKAGE_MANAGER_SET and PROJECT_MANAGER variables set in the CI\CD variables.

include:
  - project: 'group/dependabot-script'
    ref: master
    file: '/.dependabot-template.yml'

Voila!

In the end, your problem will be that many GitLab instances you will not be able to access unless you are on premise or they open their firewall (which may not be likely). An alternative approach is that you can make a nice GitLab template that requires some license keys or something and allow your customers to run that template on a project by project basis on premise. I think this is a really great way to get something out quick without having to create some on premise server that customers can use that is similar to your fully featured Dependabot GitHib solution. You've already done a lot of the work and I hope my code above gives you some good ideas!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.