Dependabot doesn't work with projects in GitLab subgroups

[b1rdex]

I use dependabot-script to run Dependabot on self-hosted GitLab issue. The tool is working fine, but there is a problem with repositories, that are located in the subgroups – the names like group/subgroup/repository.

�[0KRunning with gitlab-runner 13.10.0 (54944146)
�[0;m�[0K  on infra-gitlab-runner-68856b776d-lx5qw 4SqrWnSc
�[0;m�[0K  feature flags: FF_GITLAB_REGISTRY_HELPER_IMAGE:true
�[0;msection_start:1627837305:prepare_executor
�[0K�[0K�[36;1mPreparing the "kubernetes" executor�[0;m
�[0;m�[0KUsing Kubernetes namespace: gitlab-runners
�[0;m�[0KUsing Kubernetes executor with image dependabot/dependabot-core ...
�[0;msection_end:1627837305:prepare_executor
�[0Ksection_start:1627837305:prepare_script
�[0K�[0K�[36;1mPreparing environment�[0;m
�[0;mWaiting for pod gitlab-runners/runner-4sqrwnsc-project-292-concurrent-0slnzq to be running, status is Pending
Waiting for pod gitlab-runners/runner-4sqrwnsc-project-292-concurrent-0slnzq to be running, status is Pending
	ContainersNotReady: "containers with unready status: [build helper svc-0]"
	ContainersNotReady: "containers with unready status: [build helper svc-0]"
Running on runner-4sqrwnsc-project-292-concurrent-0slnzq via infra-gitlab-runner-68856b776d-lx5qw...
section_end:1627837318:prepare_script
�[0Ksection_start:1627837318:get_sources
�[0K�[0K�[36;1mGetting source from Git repository�[0;m
�[0;m�[32;1mFetching changes with git depth set to 50...�[0;m
Initialized empty Git repository in /builds/infra/Dependabots/kira-dependencies/.git/
�[32;1mCreated fresh repository.�[0;m
�[32;1mChecking out ff5302ea as master...�[0;m

�[32;1mSkipping Git submodules setup�[0;m
section_end:1627837318:get_sources
�[0Ksection_start:1627837318:restore_cache
�[0K�[0K�[36;1mRestoring cache�[0;m
�[0;m�[32;1mChecking cache for default...�[0;m
No URL provided, cache will not be downloaded from shared cache server. Instead a local version of cache will be extracted.�[0;m 
�[32;1mSuccessfully extracted cache�[0;m
section_end:1627837319:restore_cache
�[0Ksection_start:1627837319:step_script
�[0K�[0K�[36;1mExecuting "step_script" stage of the job script�[0;m
�[0;m�[32;1m$ bundle install -j $(nproc) --path vendor�[0;m
[DEPRECATED] The `--path` flag is deprecated because it relies on being remembered across bundler invocations, which bundler will no longer do in future versions. Instead please use `bundle config set --local path 'vendor'`, and stop using this flag
Fetching gem metadata from https://rubygems.org/
Fetching gem metadata from https://rubygems.org/...........
Fetching aws-eventstream 1.1.1
Fetching minitest 5.14.4
Fetching jmespath 1.4.0
Fetching concurrent-ruby 1.1.9
Fetching public_suffix 4.0.6
Fetching zeitwerk 2.4.2
Fetching ast 2.4.2
Fetching aws-partitions 1.471.0
Installing ast 2.4.2
Installing aws-eventstream 1.1.1
Installing aws-partitions 1.471.0
Installing zeitwerk 2.4.2
Installing jmespath 1.4.0
Using bundler 2.2.20
Installing minitest 5.14.4
Fetching citrus 3.0.2
Fetching commonmarker 0.22.0
Fetching http-accept 1.7.0
Fetching unf_ext 0.0.7.7
Installing public_suffix 4.0.6
Fetching mime-types-data 3.2021.0225
Installing concurrent-ruby 1.1.9
Installing http-accept 1.7.0
Installing citrus 3.0.2
Fetching netrc 0.11.0
Installing commonmarker 0.22.0 with native extensions
Installing mime-types-data 3.2021.0225
Fetching excon 0.82.0
Installing netrc 0.11.0
Fetching multi_xml 0.6.0
Installing unf_ext 0.0.7.7 with native extensions
Fetching unicode-display_width 1.7.0
Installing multi_xml 0.6.0
Fetching racc 1.5.2
Installing unicode-display_width 1.7.0
Installing excon 0.82.0
Fetching faraday-em_http 1.0.0
Fetching faraday-em_synchrony 1.0.0
Fetching faraday-excon 1.1.0
Installing faraday-em_http 1.0.0
Installing faraday-em_synchrony 1.0.0
Installing faraday-excon 1.1.0
Fetching faraday-net_http 1.0.1
Fetching faraday-net_http_persistent 1.1.0
Fetching multipart-post 2.1.1
Fetching ruby2_keywords 0.0.4
Installing racc 1.5.2 with native extensions
Fetching pandoc-ruby 2.1.4
Installing faraday-net_http_persistent 1.1.0
Installing faraday-net_http 1.0.1
Fetching parseconfig 1.0.8
Fetching aws-sigv4 1.2.3
Installing multipart-post 2.1.1
Installing ruby2_keywords 0.0.4
Fetching parser 3.0.1.1
Installing pandoc-ruby 2.1.4
Fetching addressable 2.7.0
Installing parseconfig 1.0.8
Installing aws-sigv4 1.2.3
Fetching toml-rb 2.0.1
Fetching mime-types 3.3.1
Fetching terminal-table 1.8.0
Installing toml-rb 2.0.1
Installing mime-types 3.3.1
Installing terminal-table 1.8.0
Fetching i18n 1.8.10
Installing addressable 2.7.0
Fetching tzinfo 2.0.4
Fetching faraday 1.4.3
Fetching aws-sdk-core 3.115.0
Installing i18n 1.8.10
Installing parser 3.0.1.1
Fetching httparty 0.18.1
Installing tzinfo 2.0.4
Installing faraday 1.4.3
Fetching activesupport 6.1.4
Installing httparty 0.18.1
Fetching nokogiri 1.11.7 (x86_64-linux)
Installing aws-sdk-core 3.115.0
Fetching sawyer 0.8.2
Fetching gitlab 4.17.0
Installing activesupport 6.1.4
Installing sawyer 0.8.2
Installing gitlab 4.17.0
Fetching octokit 4.21.0
Installing octokit 4.21.0
Fetching unf 0.1.4
Fetching aws-sdk-ecr 1.42.0
Fetching aws-sdk-codecommit 1.42.0
Installing nokogiri 1.11.7 (x86_64-linux)
Installing unf 0.1.4
Installing aws-sdk-ecr 1.42.0
Installing aws-sdk-codecommit 1.42.0
Fetching domain_name 0.5.20190701
Installing domain_name 0.5.20190701
Fetching http-cookie 1.0.4
Installing http-cookie 1.0.4
Fetching rest-client 2.1.0
Installing rest-client 2.1.0
Fetching docker_registry2 1.10.0
Installing docker_registry2 1.10.0
Fetching dependabot-common 0.156.1
Installing dependabot-common 0.156.1
Fetching dependabot-bundler 0.156.1
Fetching dependabot-github_actions 0.156.1
Fetching dependabot-cargo 0.156.1
Fetching dependabot-dep 0.156.1
Fetching dependabot-elm 0.156.1
Fetching dependabot-composer 0.156.1
Fetching dependabot-docker 0.156.1
Fetching dependabot-git_submodules 0.156.1
Installing dependabot-git_submodules 0.156.1
Installing dependabot-elm 0.156.1
Installing dependabot-docker 0.156.1
Installing dependabot-cargo 0.156.1
Installing dependabot-dep 0.156.1
Installing dependabot-github_actions 0.156.1
Installing dependabot-composer 0.156.1
Installing dependabot-bundler 0.156.1
Fetching dependabot-go_modules 0.156.1
Fetching dependabot-maven 0.156.1
Fetching dependabot-hex 0.156.1
Fetching dependabot-npm_and_yarn 0.156.1
Fetching dependabot-nuget 0.156.1
Fetching dependabot-python 0.156.1
Fetching dependabot-terraform 0.156.1
Installing dependabot-go_modules 0.156.1
Installing dependabot-maven 0.156.1
Installing dependabot-hex 0.156.1
Installing dependabot-nuget 0.156.1
Installing dependabot-terraform 0.156.1
Installing dependabot-python 0.156.1
Fetching dependabot-gradle 0.156.1
Installing dependabot-npm_and_yarn 0.156.1
Installing dependabot-gradle 0.156.1
Fetching dependabot-omnibus 0.156.1
Installing dependabot-omnibus 0.156.1
Bundle complete! 1 Gemfile dependency, 69 gems now installed.
Bundled gems are installed into `./vendor`
Post-install message from httparty:
When you HTTParty, you must party hard!
�[32;1m$ bundle exec ruby ./update.rb�[0;m
warning: parser/current is loading parser/ruby27, which recognizes
warning: 2.7.3-compliant syntax, but you are running 2.7.1.
warning: please see https://github.com/whitequark/parser#compatibility-with-ruby-mri.
/builds/infra/Dependabots/kira-dependencies/vendor/ruby/2.7.0/gems/gitlab-4.17.0/lib/gitlab/paginated_response.rb:24:in `method_missing': undefined method `default_branch' for #<Gitlab::PaginatedResponse:0x000055a3fac7e040> (NoMethodError)
	from /builds/infra/Dependabots/kira-dependencies/vendor/ruby/2.7.0/gems/dependabot-common-0.156.1/lib/dependabot/clients/gitlab_with_retries.rb:51:in `fetch_default_branch'
	from /builds/infra/Dependabots/kira-dependencies/vendor/ruby/2.7.0/gems/dependabot-common-0.156.1/lib/dependabot/file_fetchers/base.rb:443:in `default_branch_for_repo'
	from /builds/infra/Dependabots/kira-dependencies/vendor/ruby/2.7.0/gems/dependabot-common-0.156.1/lib/dependabot/file_fetchers/base.rb:71:in `commit'
	from /builds/infra/Dependabots/kira-dependencies/vendor/ruby/2.7.0/gems/dependabot-common-0.156.1/lib/dependabot/file_fetchers/base.rb:360:in `_full_specification_for'
	from /builds/infra/Dependabots/kira-dependencies/vendor/ruby/2.7.0/gems/dependabot-common-0.156.1/lib/dependabot/file_fetchers/base.rb:175:in `_fetch_repo_contents'
	from /builds/infra/Dependabots/kira-dependencies/vendor/ruby/2.7.0/gems/dependabot-common-0.156.1/lib/dependabot/file_fetchers/base.rb:160:in `repo_contents'
	from /builds/infra/Dependabots/kira-dependencies/vendor/ruby/2.7.0/gems/dependabot-common-0.156.1/lib/dependabot/file_fetchers/base.rb:103:in `fetch_file_if_present'
	from /builds/infra/Dependabots/kira-dependencies/vendor/ruby/2.7.0/gems/dependabot-bundler-0.156.1/lib/dependabot/bundler/file_fetcher.rb:64:in `gemfile'
	from /builds/infra/Dependabots/kira-dependencies/vendor/ruby/2.7.0/gems/dependabot-bundler-0.156.1/lib/dependabot/bundler/file_fetcher.rb:30:in `fetch_files'
	from /builds/infra/Dependabots/kira-dependencies/vendor/ruby/2.7.0/gems/dependabot-common-0.156.1/lib/dependabot/file_fetchers/base.rb:65:in `files'
	from ./update.rb:94:in `<main>'
Fetching bundler dependency files for 
section_end:1627837334:step_script
�[0Ksection_start:1627837334:cleanup_file_variables
�[0K�[0K�[36;1mCleaning up file based variables�[0;m
�[0;msection_end:1627837334:cleanup_file_variables
�[0K�[31;1mERROR: Job failed: command terminated with exit code 1
�[0;m

[jurre]

Hi @b1rdex, currently our support for GitLab is community contributed as we don't have the resources to maintain this ourselves. I'd be happy to review a PR that addresses this, it looks like the error is raised here. I'm not sure why these specific repo's don't seem to have a default_branch specified, but I don't know much about how GitLab works and maybe we'll need to make a different API call for repo's in subgroups?

[argoyle]

I think the problem is in the regex for Gitlab in

dependabot-core/common/lib/dependabot/source.rb

Line 21 in ca9f236

GITLAB_SOURCE = %r{

I've tried to come up with something better but haven't managed yet.
I added this test (which currently fail) to source_spec.rb:

    context "with a GitLab subgroup URL" do
      let(:url) { "https://gitlab.com/org/group/abc/blob/master/dir/readme.md" }
      its(:provider) { is_expected.to eq("gitlab") }
      its(:repo) { is_expected.to eq("org/group/abc") }
      its(:directory) { is_expected.to eq("dir") }
    end

[stephenliberty]

@jurre - this popped up as a problem for an outfit I work for - their repos are all subgroups, and the repo is reporting back incorrectly. It looks like the fix above (#5297) may work. Is there someone that needs to be nudged to get some action on that?