Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Git Stub Downloader #596

Merged
merged 12 commits into from Mar 31, 2018

Conversation

@marcingrzejszczak
Copy link
Contributor

marcingrzejszczak commented Mar 23, 2018

Description from the docs

Do I need a Binary Storage? Can't I use Git?

In the polyglot world, there are languages that don't use binary storages like
Artifactory or Nexus. Starting from Spring Cloud Contract version 2.0.0 we provide
mechanisms to store contracts and stubs in a SCM repository. Currently the
only supported SCM is Git.

The repository would have to the following setup
(which you can checkout here https://github.com/spring-cloud-samples/spring-cloud-contract-samples/tree/2.0.x/contracts_git/):

.
└── META-INF
    └── com.example
        └── beer-api-producer-git
            └── 0.0.1-SNAPSHOT
                ├── contracts
                │   └── beer-api-consumer
                │       ├── messaging
                │       │   ├── shouldSendAcceptedVerification.groovy
                │       │   └── shouldSendRejectedVerification.groovy
                │       └── rest
                │           ├── shouldGrantABeerIfOldEnough.groovy
                │           └── shouldRejectABeerIfTooYoung.groovy
                └── mappings
                    └── beer-api-consumer
                        └── rest
                            ├── shouldGrantABeerIfOldEnough.json
                            └── shouldRejectABeerIfTooYoung.json

Under META-INF folder:

  • we group applications via groupId (e.g. com.example)
  • then each application is represented via the artifactId (e.g. beer-api-producer-git)
  • next, the version of the application. The version is mandatory! (e.g. 0.0.1-SNAPSHOT)
  • Finally, there are two folders:
    ** contracts - the good practice is to store the contracts required by each
    consumer in the folder with the consumer name (e.g. beer-api-consumer). That way you
    can use the stubs-per-consumer feature. Further directory structure is arbitrary.
    ** mappings - in this folder the Maven / Gradle Spring Cloud Contract plugins will push
    the stub server mappings. On the consumer side, Stub Runner will scan this folder
    to start stub servers with stub definitions. The folder structure will be a copy
    of the one created in the contracts subfolder.

Protocol convention

In order to control the type and location of the source of contracts (whether it's
a binary storage or an SCM repository), you can use the protocol in the URL of
the repository. Spring Cloud Contract iterates over registered protocol resolvers
and tries to fetch the contracts (via a plugin) or stubs (via Stub Runner).

For the SCM functionality, currently, we support the Git repository. To use it,
in the property, where the repository URL needs to be placed you just have to prefix
the connection URL with git://. Here you can find a couple of examples:

git://file:///foo/bar
git://https://github.com/spring-cloud-samples/spring-cloud-contract-nodejs-contracts-git.git
git://git@github.com:spring-cloud-samples/spring-cloud-contract-nodejs-contracts-git.git

Producer

For the producer, to use the SCM approach, we can reuse the same mechanism we use for external contracts. We route Spring Cloud Contract to use the SCM implementation via the URL that contains the git:// protocol.

IMPORTANT: You have to manually add the pushStubsToScm goal in Maven or execute (bind) the pushStubsToScm task in Gradle. We don't push stubs to origin of your git repository out of the box.

.Maven

<plugin>
    <groupId>org.springframework.cloud</groupId>
    <artifactId>spring-cloud-contract-maven-plugin</artifactId>
    <version>${spring-cloud-contract.version}</version>
    <extensions>true</extensions>
    <configuration>
        <!-- Base class mappings etc. -->

        <!-- We want to pick contracts from a Git repository -->
        <contractsRepositoryUrl>git://https://github.com/spring-cloud-samples/spring-cloud-contract-nodejs-contracts-git.git</contractsRepositoryUrl>

        <!-- We reuse the contract dependency section to set up the path
        to the folder that contains the contract definitions. In our case the
        path will be /groupId/artifactId/version/contracts -->
        <contractDependency>
            <groupId>${project.groupId}</groupId>
            <artifactId>${project.artifactId}</artifactId>
            <version>${project.version}</version>
        </contractDependency>

        <!-- The contracts mode can't be classpath -->
        <contractsMode>REMOTE</contractsMode>
    </configuration>
    <executions>
        <execution>
            <phase>package</phase>
            <goals>
                <!-- By default we will not push the stubs back to SCM,
                you have to explicitly add it as a goal -->
                <goal>pushStubsToScm</goal>
            </goals>
        </execution>
    </executions>
</plugin>

.Gradle

contracts {
	// We want to pick contracts from a Git repository
	contractDependency {
		stringNotation = "${project.group}:${project.name}:${project.version}"
	}
	/*
	We reuse the contract dependency section to set up the path
	to the folder that contains the contract definitions. In our case the
	path will be /groupId/artifactId/version/contracts
	 */
	contractRepository {
		repositoryUrl = "git://https://github.com/spring-cloud-samples/spring-cloud-contract-nodejs-contracts-git.git"
	}
	// The mode can't be classpath
	contractsMode = "REMOTE"
	// Base class mappings etc.
}

/*
In this scenario we want to publish stubs to SCM whenever
the `publish` task is executed
*/
publish.dependsOn("publishStubsToScm")

With such a setup:

  • Git project will be cloned to a temporary directory
  • The SCM stub downloader will go to META-INF/groupId/artifactId/version/contracts folder
    to find contracts. E.g. for com.example:foo:1.0.0 the path would be
    META-INF/com.example/foo/1.0.0/contracts
  • Tests will be generated from the contracts
  • Stubs will be created from the contracts
  • Once the tests pass, the stubs will be committed in the cloned repository
  • Finally, a push will be done to that repo's origin

Consumer

On the consumer side when passing the repositoryRoot parameter,
either from the @AutoConfigureStubRunner annotation, the
JUnit rule or properties, it's enough to pass the URL of the
SCM repository, prefixed with the protocol. For example

@AutoConfigureStubRunner(
    stubsMode="REMOTE",
    repositoryRoot="git://https://github.com/spring-cloud-samples/spring-cloud-contract-nodejs-contracts-git.git",
    ids="com.example:bookstore:0.0.1.RELEASE"
)

With such a setup:

  • Git project will be cloned to a temporary directory
  • The SCM stub downloader will go to META-INF/groupId/artifactId/version/ folder
    to find stub definitions and contracts. E.g. for com.example:foo:1.0.0 the path would be
    META-INF/com.example/foo/1.0.0/
  • Stub servers will be started and fed with mappings
  • Messaging definitions will be read and used in the messaging tests

Using the SCM Stub Downloader

Whenever the repositoryRoot starts with a SCM protocol (currently we support only git://), the stub downloader will try to clone the repository and use it as a source of contracts to generate tests or stubs.

Either via environment variables, system properties, properties set inside the plugin or contracts repository configuration you can tweak the downloader's behaviour. Below you can find the list of properties

Technical issues

The approach

when stubsMode is set to LOCAL or REMOTE, and repositoryRoot starts with git:// we can clone the provided git repository, and search for the folder with stubs for the given artifact. So if the git repo has a folder structure of groupid/artifactid/version (where group id is either dot or slash separated), then we will provide a path to that repository for the stub runner to harvest the stubs

What's done?

  • externalized versions
  • added more debugging messages
  • added ContractProjectUpdater that updates the project containing contracts from SCM. ATM supports only git
  • added ResourceResolver that retrieves the ProtocolResolvers. It does it via spring.factories entries containing StubDownloaderBuilder. SDP extends ProtocolResovler.
  • added StubRunner.properties map, that will contain any properties that will be later used by any StubDownloader implementations
  • added PUBLISH_STUBS_TO_SCM env var for Docker, so that publishStubsToScm task gets called
  • updated docs

Breaking:

  • StubDownloaderBuilder extends ProtocolResovler. By default the ProtocolResolver methods return null.
  • stubRunnerOptions.stubRepositoryRoot is a Resource not a String
  • generateWireMockClientStubs Gradle task got removed
  • if folder with contracts has a subfolder called contracts, we will pick contracts from the subfolder

fixes #580

@marcingrzejszczak marcingrzejszczak added this to the 2.0.0.M9 milestone Mar 23, 2018
pom.xml Outdated
<dependency>
<groupId>org.eclipse.jgit</groupId>
<artifactId>org.eclipse.jgit</artifactId>
<version>4.6.0.201612231935-r</version>

This comment has been minimized.

Copy link
@marcingrzejszczak

marcingrzejszczak Mar 25, 2018

Author Contributor

parametrize

@codecov-io

This comment has been minimized.

Copy link

codecov-io commented Mar 25, 2018

Codecov Report

Merging #596 into master will increase coverage by 0.09%.
The diff coverage is 62.12%.

Impacted file tree graph

@@             Coverage Diff              @@
##             master     #596      +/-   ##
============================================
+ Coverage     60.88%   60.98%   +0.09%     
- Complexity     1760     1826      +66     
============================================
  Files           203      210       +7     
  Lines          7033     7507     +474     
  Branches       1063     1121      +58     
============================================
+ Hits           4282     4578     +296     
- Misses         2154     2291     +137     
- Partials        597      638      +41
Impacted Files Coverage Δ Complexity Δ
.../stubrunner/spring/cloud/StubMapperProperties.java 89.47% <ø> (ø) 8 <0> (ø) ⬇️
...ork/cloud/contract/stubrunner/AetherFactories.java 83.78% <ø> (ø) 10 <0> (ø) ⬇️
...rk/cloud/contract/stubrunner/util/StubsParser.java 77.41% <ø> (ø) 5 <0> (ø) ⬇️
...pring/cloud/ribbon/StubRunnerRibbonServerList.java 63.63% <ø> (ø) 4 <0> (ø) ⬇️
...runner/spring/cloud/StubRunnerDiscoveryClient.java 16.98% <ø> (ø) 3 <0> (ø) ⬇️
...oud/contract/stubrunner/ClasspathStubProvider.java 5.43% <0%> (ø) 3 <0> (ø) ⬇️
...work/cloud/contract/stubrunner/StubRepository.java 61.42% <0%> (-2.86%) 17 <0> (ø)
.../contract/verifier/file/ContractFileScanner.groovy 64.19% <0%> (ø) 35 <0> (ø) ⬇️
...rifier/stubrunner/AetherStubDownloaderFactory.java 50% <0%> (-5.56%) 2 <0> (ø)
...loud/contract/stubrunner/junit/StubRunnerRule.java 48.27% <0%> (-1.14%) 16 <0> (ø)
... and 32 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f6a8f49...488cbb0. Read the comment docs.

when stubsMode is set to LOCAL or REMOTE, and repository Root starts with git:// we can clone the provided git repository, and search for the folder with stubs for the given artifact. So if the git repo has a folder structure of groupid/artifactid/version (where group id is either dot or slash separated), then we will provide a path to that repository for the stub runner to harvest the stubs

part of #580
with this change we're making SDB extend PR. The user already has to register SDB in META-INF/spring.factories , so we won't have to ask the user to do it again for PR. We also introduce a map of properties passed via `stubrunner.properties` system propery or `STUBRUNNER_PROPERTIES` env var or just directly via Spring mechanism or Maven / Gradle plugins. That way one will be able to pass arbitrary properties to any SDB mechanism.

For Git Stub Downloader, this change also introduces passing of git credentials and git branch to check out.

Tested against https://github.com/spring-cloud-samples/spring-cloud-contract-samples/tree/git_downloader
@marcingrzejszczak marcingrzejszczak force-pushed the issues_#580_git_stub_downloader branch from 5b82209 to 53dad52 Mar 25, 2018
still to add gradle and docs and tests for maven, gradle and maybe one standalone (extension of the existing one)
- added more debugging messages
- if folder with contracts has a subfolder called `contracts`, we will pick contracts from the subfolder
- extracted a PROTOCOL field for GitStubDownloader
- GitStubDownloader is public but can't be extended
- preloads TemporaryFileStorage class wherever it's used cause, upon shutdown hook execution, exceptions are thrown that the class is not found
@marcingrzejszczak marcingrzejszczak merged commit 0fe31ce into master Mar 31, 2018
4 checks passed
4 checks passed
ci/circleci Your tests passed on CircleCI!
Details
ci/pivotal-cla Thank you for signing the Contributor License Agreement!
Details
codecov/patch 62.12% of diff hit (target 60.88%)
Details
codecov/project 60.98% (+0.09%) compared to f6a8f49
Details
@marcingrzejszczak marcingrzejszczak modified the milestones: 2.0.0.M9, 2.0.0.RC1 Apr 2, 2018
@jorahood

This comment has been minimized.

Copy link

jorahood commented Apr 19, 2018

I need to configure a CredentialsProvider to provide username and password for downloading from a private Git repo that requires https.

@marcingrzejszczak

This comment has been minimized.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.