Add support for multi module/project built applications

[malston]

When a project is comprised of multiple Maven modules or Gradle subprojects, then I want a way to tell sc-pipelines which artifact of these is the deployable. I've already configured github-analytics as a multi subproject Gradle build here. I plan to do the same for github-webhook. Currently, when ./gradlew artifactId -q | tail -1 is called from the root of the project it returns the artifactId of the last subproject in the list. However, if I explicitly call ./gradlew :api:artifactId or ./gradlew -p api artifactId then I get the result I expect. So this may just be as simple as passing in another parameter to retrieveAppName function with an optional subproject/module name.

[marcingrzejszczak]

Let's take an example of 3 repos.

• multi-projects that has 2 separate fat jars (yes, the 12th factor app is broken with such an approach)
• multi-module that has 1 deployable but multiple modules
• a-very-stupid-name that has 1 deployable with a single project

In Jenkins we would have the following seed job

// a single seed job (generates other jobs)
REPOS = 'http://github.com/foo/multi-projects.git$foo,http://github.com/foo/multi-projects.git$bar,http://github.com/foo/multi-module.git$baz,http://github.com/foo/a-very-stupid-name.git$greeting'

The value after $ is the name of the project. We will also treat it as a potential folder to which we should cd to produce the binary. In this case, it will be foo and bar folders under multi-projects repo.

We will then assert if the folder called e.g. foo exists in the repo and if that's the case we will set the ROOT_PROJECT_DIR to that value. In Concourse we would just set the env var ROOT_PROJECT_DIR directly.

If there's no such folder then we assume that the root folder is ., the root folder.

SINGLE REPO

A single pipeline for a repo

E.g. a-very-stupid-name.git$greeting. The name of the pipeline will be greeting. If $... will contain a name of a folder that doesn't exist then the ROOT_PROJECT_DIR will be . . Actually this name MUST NOT be matched by any folder.

TODO: consider being less strict and check if sc-pipelines.yml is there in root. If it's there then it means that it's a multi-module project

The location of the pipeline descriptor: /sc-pipelines.yml
The location of the deployable: /target
The location of manifests (cf/k8s): /
The tag structure: /dev/version, /prod/version

Pseudo code

# build everything
PROJECT_ROOT="."
cd ${PROJECT_ROOT} 
./mvnw clean install
# upload the artifact from the root
 ./mvnw clean deploy
# deploy using sc pipelines from root project
useScPipelinesFromRootFolder

MULTI-MODULE

A single pipeline for a repo

E.g. multi-module.git$baz, the $... passed for Jenkins must not match any modules. If it does then it will be treated as a folder that contains an application, not the module itself. The name of the pipeline will be baz.

TODO: consider being less strict and check if sc-pipelines.yml is there in root. If it's there then it means that it's a multi-module project

In the sc-pipelines.yml you can provide the coordinates of the module that will produce the JAR.

build:
    coordinates: impl:core:foo:bar
    # force that the coordinates are the same as folders
    # impl/core/foo/bar

The location of the pipeline descriptor: /sc-pipelines.yml
The location of the deployable: impl/core/foo/bar/target
The location of manifests (cf/k8s): /
The tag structure: /dev/version, /prod/version

Pseudo code

# build everything
# retrieve group id from the root module `/pom.xml`
# retrieve artifact id from the child module `impl/core/foo/bar/pom.xml`
PROJECT_ROOT="."
cd ${PROJECT_ROOT} 
./mvnw clean install
# upload only the artifact from my specific module
moduleCoordinate="impl:core:foo:bar" #parsed from sc-pipelines
./mvnw clean deploy
# deploy using sc-pipelines from root project
useScPipelinesFromRootFolder

MULTI-PROJECT (LEGACY)

One pipeline for each sub project (in case we can reuse credentials we will have 1 seed job)
otherwise - separate seed jobs

IMPORTANT the presented approach in this section is an antipattern and shouldn't be used in your projects. It's here since it's treated as a legacy system to be refactored.

E.g. multi-projects.git$foo and multi-projects.git$bar. sc-pipelines will check if there's a folder foo for foo pipeline and bar for bar pipeline. Each subproject must have its own maven / gradle wrappers and pipeline descriptor. ROOT_PROJECT_DIR will be set to foo and bar respectively. That means that the project will cd to ROOT_PROJECT_DIR before executing any code.

Pseudo code

foo pipeline

# move to the folder with your application
PROJECT_ROOT="foo"
cd ${PROJECT_ROOT}
# build everything
./mvnw clean install
# upload only the artifact
./mvnw clean deploy
# deploy
useScPipelinesFromRootFolder

bar pipeline

# build everything
PROJECT_ROOT="bar"
cd ${PROJECT_ROOT}
./mvnw clean install
# upload only the artifact from my specific module
./mvnw clean deploy
# deploy
useScPipelinesFromRootFolder

The location of the pipeline descriptor: /${PROJECT_ROOT}/sc-pipelines.yml
The location of the deployable: ${PROJECT_ROOT}/target
The location of manifests (cf/k8s): /${PROJECT_ROOT}
The tag structure: /dev/${PROJECT_ROOT}/version, /prod/${PROJECT_ROOT}/version

SC-Pipelines structure

In order to provide information about which module builds the fat jar, we will verify the build.main-module entry to retrieve the name of the module that builds the fat jar.

Breaking changes

• The tags will change. The project name will be added - dev/${projectName}/${projectVersion}

[malston]

Looks like you've put a lot of thought into this... I don't see any issues with this approach. 👍