recursive submodules are not cloned

[baszalmstra]

I just upgraded from Concourse v4.2.2 to v5.0.1. Since this update recursive submodules are no longer cloned. Submodules themselves are checked out just fine but their submodules are not cloned.

One example is breakpad which has a recursive submodules required to build the project. In version v4.2.2 the output would yield something like:

Submodule 'external/breakpad/source' (<redacted>) registered for path 'external/breakpad/source'
Cloning into '/tmp/build/get/external/breakpad/source'...
Submodule path 'external/breakpad/source': checked out 'a78ee23927cb3a5d4040e75c2b8a37c2f1c6be39'	
Submodule 'src/third_party/lss' (<redacted>) registered for path 'external/breakpad/source/src/third_party/lss'
Cloning into '/tmp/build/get/external/breakpad/source/src/third_party/lss'...
Submodule path 'external/breakpad/source/src/third_party/lss': checked out 'e6527b0cd469e3ff5764785dadcb39bf7d787154'

Version v5.0.1 outputs:

Submodule 'external/breakpad/source' (<redacted>) registered for path 'external/breakpad/source'
Cloning into '/tmp/build/get/external/breakpad/source'...
Submodule path 'external/breakpad/source': checked out 'a78ee23927cb3a5d4040e75c2b8a37c2f1c6be39'
Entering 'external/breakpad/source'
fetch: Fetching reference HEAD
WARNING: 'git lfs checkout' is deprecated and will be removed in v3.0.0.
'git checkout' has been updated in upstream Git to have comparable speeds
to 'git lfs checkout'.

Nothing related to the recursive lss submodule.

I've not specified any submodule parameters in the config:

- name: source
  type: git
  source:
    branch: ((branch))
    uri: <redacted>
    private_key: {{gitlab-private-key}}
    ignore_paths: 
    - ci/pipelines/**
    - images/**

And also none in the get step:

- get: source

Explicitly adding submodule_recursive doesn't change anything:

- get: source
  params: { submodule_recursive: true }

This currently breaks a lot of my pipelines. Reading through the source did not yield any solution. Any fix or workaround would be greatly appreciated!

[vito]

Is this the same as #246?

[baszalmstra]

Is this the same as #246?

No, I also ran into that issue, but fixed it by ensuring the paths and the names are the same in my .gitmodule file.

[baszalmstra]

I created a very minimal repro: https://github.com/baszalmstra/concourse-git-concourse-recursive-repro.

The pipeline simply clones a repository containing a single submodule which in turn contains recursive submodules:

resources:
- name: source
  type: git
  source:
    uri: https://github.com/baszalmstra/concourse-git-concourse-recursive-repro.git
    
jobs:
- name: test
  plan:
    - get: source

The build output of test shows that the recursive submodules are not cloned:

Cloning into '/tmp/build/get'...
fetch: Fetching reference HEAD
WARNING: 'git lfs checkout' is deprecated and will be removed in v3.0.0.	
'git checkout' has been updated in upstream Git to have comparable speeds
to 'git lfs checkout'.
71eff0a initial commit
Submodule 'header-parser' (https://github.com/baszalmstra/header-parser.git) registered for path 'header-parser'
Cloning into '/tmp/build/get/header-parser'...
Submodule path 'header-parser': checked out '56e748e62a0fadec3664d34770c99111aa8bd4e7'
Entering 'header-parser'
fetch: Fetching reference HEAD
WARNING: 'git lfs checkout' is deprecated and will be removed in v3.0.0.
'git checkout' has been updated in upstream Git to have comparable speeds
to 'git lfs checkout'.

Next I tried to use a different version of the git resource. Here is my summary of which version cloned the recursive submodules and which didn't:

Version	Works
1.0.0	☑
1.0.1	☑
1.1.0	❌
1.1.1	❌
1.2.0	❌
1.3.0	❌

So it appears this functionality broke with the 1.1.0 release of this resource.

[vito]

Thanks for investigating!

I opened #255 in an attempt to fix this and clean up a few things, but I haven't had time to verify it. If you have time to try it out, I've pushed it to vito/git-resource:pr-255:

resource_types:
- name: git
  type: registry-image
  source: {repository: vito/git-resource, tag: pr-255}

[baszalmstra]

I can confirm that this fixed the issue in all of our pipelines. Thank you!

[vito]

@baszalmstra Thanks for confirming! I've shipped this as v1.3.1. 👍

[margitabo]

Hi @vito, we came across the same issue after updating the bosh concourse-deployment from v4.2.1 to v5.0.1.
We see that a new version v5.1.0 already came out. Could you please tell us if the fix in the git-resource shipped in v1.3.1 is already available in the bosh concourse-deployment v5.1.0 or we should wait for the next version? I tried to find out where the git-resource version is defined but without any success. Where we can check the git-resource version the bosh concourse-deployment is using? Thanks a lot!

[baszalmstra]

I upgraded to 5.1.0 (not the bosh release) which doesn't yet contain 1.3.1 of the git resource.

[cirocosta]

@baszalmstra , that's true; these are the versions included in 5.1.0:

# extra the `.tgz`, then within the `concourse` dir:
find . -name "*.json"  | xargs -I[] cat [] | jq -c '. | {type: .type, version: .version}'
{"type":"s3","version":"1.0.2"}
{"type":"github-release","version":"1.1.1"}
{"type":"mock","version":"0.6.1"}
{"type":"cf","version":"1.1.0"}
{"type":"registry-image","version":"0.5.3"}
{"type":"tracker","version":"1.0.2"}
{"type":"docker-image","version":"1.3.0"}
{"type":"time","version":"1.2.0"}
{"type":"semver","version":"1.0.2"}
{"type":"bosh-io-release","version":"1.0.2"}
{"type":"hg","version":"1.0.2"}
{"type":"bosh-io-stemcell","version":"1.0.2"}
{"type":"pool","version":"1.0.2"}
{"type":"git","version":"1.3.0"}

If you'd like to consume 1.3.1 before a new release, as I see, you have two options:

1. put in every worker's resources directory the resource type (worker has a flag for specifying where resource types should be located), or
2. in your pipeline, configure a resource_types section (see https://concourse-ci.org/resource-types.html) - this is the most convenient.

Thanks!

[margitabo]

Thank you guys for the explanations and for the suggested workarounds! We already use a pre-defined resource_type in our pipelines pointing to the fix:
resource_types:

• name: git
  type: registry-image
  source:
  repository: vito/git-resource
  tag: pr-255

Just didn't understand how we can find out the git-resource version currently used of the bosh concourse-deployment submodule in our project. We checked out tag v5.1.0 and deployed Concourse with that version: https://github.com/concourse/concourse-bosh-deployment/tree/v5.1.0
@cirocosta which .tgz I should download and execute the command within?

Thanks!

[margitabo]

Hi, when is git-resource v1.3.1 having the fix expected to be available in concourse bosh release?
Thanks a lot!

[margitabo]

Hi, could you please update us when a new version of concourse-bosh-deployment having the fix is expected to come out? An approximate date would be greatly appreciated.
Thanks and regards!

[cirocosta]

Hi @margitabo , sorry for the delay!

Just didn't understand how we can find out the git-resource version currently used of the bosh concourse-deployment submodule in our project.

That information wouldn't be easily found by just looking at the versions.yml file in concourse-bosh-deployment or even concourse-bosh-release due to the changes in packaging that we performed.

Now, the BOSH release is just a thin layer on top of the compressed tarball that we generate for binary deployments. The default base resource types can be found in the tarball under concourse/resource-types (if you go to any worker that already unpacked such tarball, you can find those under /var/vcap/packages/concourse/resource-types).

Executing that command in #251 (comment) would reveal the metadata of each of the resource types that we had packaged for such release.

Alternativelly, with Concourse already running, you can hit /api/v1/worker to get a list of their resource types (it includes all of the loaded resource metadata).

could you please update us when a new version of concourse-bosh-deployment having the fix is expected to come out?

We're almost ready to ship 5.2 - all of the necessary code for 5.2 already got in (see https://github.com/concourse/concourse/milestone/40) -, now being in the process of getting it to our internal larger scale deployment of Concourse to ensure that it's really good to go.

In terms of date - we're aiming at the beginning of next week.

Please let me know if that covers your questions 😁
Thanks!