Merge branch 'prepare-release' into release

SAP-archive · Jun 25, 2020 · 17488e4 · 17488e4
2 parents d50c779 + 9188814
commit 17488e4
Show file tree

Hide file tree

Showing 5 changed files with 189 additions and 62 deletions.
diff --git a/.gitignore b/.gitignore
@@ -0,0 +1 @@
+.vscode
diff --git a/configuration.md b/configuration.md
@@ -5,7 +5,6 @@
 * [Pipeline configuration](#pipeline-configuration)
   * [customDefaults](#customDefaults)
   * [General configuration](#general-configuration)
-    * [automaticVersioning](#automaticversioning)
     * [features](#features)
     * [jenkinsKubernetes](#jenkinskubernetes)
   * [Stage configuration](#stage-configuration)
@@ -22,15 +21,14 @@
     * [artifactDeployment](#artifactdeployment)
       * [nexus](#nexus)
     * [whitesourceScan](#whitesourcescan)
-    * [sourceClearScan](#sourceclearscan)
     * [fortifyScan](#fortifyscan)
     * [lint](#lint)
     * [sonarQubeScan](#sonarqubescan)
     * [postPipelineHook](#postpipelinehook)
   * [Step configuration](#step-configuration)
+    * [artifactPrepareVersion](#artifactprepareversion)
     * [mavenExecute](#mavenexecute)
     * [executeNpm](#executenpm)
-    * [executeSourceClearScan](#executesourceclearscan)
     * [cloudFoundryDeploy](#cloudfoundrydeploy)
     * [neoDeploy](#neodeploy)
     * [checkFindbugs](#checkfindbugs)
@@ -66,22 +64,8 @@ For more information on how to configure custom default configurations, please r
 | `projectName` | | `artifactId` from pom | Name of the project |
 | `collectTelemetryData` | | `true` | No personal data is collected. For details, consult the [analytics documentation](doc/operations/analytics.md). |
 | `unsafeMode` | | `false` | Enable unsafe mode to skip checking environment variables for insecure elements. Only use this for demo purposes, **never for productive usage**. |
+| `customDefaultsCredentialsId` | |  | Credentials (username / password) used to download [custom defaults](#customDefaults). |
 
-#### automaticVersioning
-The pipeline can be configured to store release candidates in a nexus repository after they passed all stages successfully. By turning on automatic versioning, one can avoid that multiple builds of a continuously delivered application lead to version collisions in nexus. When activated, the pipeline will assign unique maven versions for each release candidate. If you are not building a continuously delivered application, you will typically disable automatic versioning.
-Architectural details of this step can be found in [automatic-release.md](doc/architecture/decisions/automatic-release.md).
-
-| Property | Mandatory | Default Value | Description |
-| --- | --- | --- | --- |
-| `automaticVersioning` | | `true` | Apply automated versioning scheme. To disable this feature, set the value to `false` |
-
-Example:
-
-```yaml
-general:
-  productiveBranch: 'master'
-  automaticVersioning: true
-```
 
 #### features
 This section allows to enable or disable certain optional features.
@@ -551,30 +535,6 @@ Please ensure that all `package.json` files have a `name` and `version` configur
 
 Please note that you can not have a `whitesource.config.json` in your project, since the Pipeline generates one from this configuration.
 
-#### sourceClearScan
-Configure [SourceClear](https://www.sourceclear.com/) scans.
-
-**Note:** Please note that the SourceClear stage of this pipeline is not actively maintained anymore. In case of issues, feel free to contribute to this project by opening a pull request.
-
-| Property | Mandatory | Default Value | Description |
-| --- | --- | --- | --- |
-| `credentialsId` | X | | Jenkins credentials id for the SourceClear API token. See [SourceClear docs for details](https://www.sourceclear.com/docs/jenkins-script/). |
-| `config` | | | Additional configuration for the SourceClear agent. The key-value pairs will be added to `srcclr.yml`. |
-
-Please note that your project can't have a `srcclr.yml` file.
-The pipeline creates a config file with optimized settings.
-If you wish to configure SourceClear, add your config entries as in the example.
-
-Example:
-
-```yaml
-sourceClearScan:
-  credentialsId: 'SRCCLR_API_TOKEN'
-  config:
-    vuln_methods_extra_ignored_directories: docs, integration-tests
-    scope: compile
-```
-
 #### fortifyScan
 
 The Fortify scan is configured using the step fortifyExecuteScan. 
@@ -591,12 +551,16 @@ If no custom linter is configured, and the project has SAPUI5 components, it mak
 A component is identified by a `Component.js` file in the directory.
 
 If no custom linter is configured, and the project has no SAPUI5 components, the pipeline uses a general purpose configuration to lint Javascript and/or Typescript files in the project.
-By default, the pipeline does not fail based on lint findings.
+By default, the pipeline does not fail based on lint findings. 
+If it is required, the pipeline can be configured to fail based on lint findings, using the `failOnError` configuration option. 
 The goal of this lint is to warn of potential errors without insisting on any programming style.
 If you're not satisfied by the default configuration, you can opt-out using that by providing your [own configuration file](https://eslint.org/docs/user-guide/configuring) in your project.
 More details can be found in the [pipeline documentation](https://sap.github.io/jenkins-library/pipelines/cloud-sdk/build-tools/#lint).
 
-Note, the configuration specified for the Lint stage in `.pipeline/config.yml` is only applied in case SAPUI5 components are checked by the pipeline and is ignored otherwise.
+Note, the available configuration options when using a custom linting script or when relying on the default linting of the pipeline can be found in the [pipeline documentation](https://sap.github.io/jenkins-library/steps/npmExecuteLint/#parameters).
+These options should be provided as step configuration for the `npmExecuteLint` step. 
+
+The configuration for the SAPUI5 best practices linter needs to be placed under `ui5BestPractices` in the Lint stage configuration in `.pipeline/config.yml`.
 
 The following example shows how to enable thresholds for linting, in case the built-in SAPUI5 best practices linter is used:
 
@@ -622,7 +586,8 @@ To get the same, please configure `esLanguageLevel: es6`.
 
 Configure [SonarQube](https://www.sonarqube.org/) scans.
 
-This is an optional feature for teams who prefer to use SonarQube. Note that it does some scans that are already done by the pipeline by default.
+This is an optional feature for teams who prefer to use SonarQube.
+Note that it does some scans that are already done by the pipeline by default.
 
 | Property | Mandatory | Default Value | Description |
 | --- | --- | --- | --- |
@@ -632,6 +597,9 @@ This is an optional feature for teams who prefer to use SonarQube. Note that it
 | `dockerImage` | | ppiper/node-browsers:v3 | This property refers to a docker image which will be used for triggering the sonar scan. In case your sonar instance uses a self signed certificate, a docker image with that certificate installed can be used. |
 | `sonarProperties` | | | The properties are used to configure sonar. Please refer to the example below. |
 
+**Note:** The stage is skipped by default if you're not on a productive branch (`master` by default).
+You can change this by setting `runInAllBranches` to `true`, which requires the commercial version of SonarQube.
+
 Example:
 
 ```yaml
@@ -660,6 +628,25 @@ Also, the stage (and thus an extension) is only executed if a stage configuratio
 
 ### Step configuration
 
+#### artifactPrepareVersion
+
+The pipeline can be configured to store release candidates in a Nexus repository after they passed all stages successfully.
+By default, the pipeline will perform automatic versioning of artifacts via the step `artifactPrepareVersion`.
+This ensures that multiple builds of a continuously delivered application do not lead to version collisions in Nexus.
+If you are not building a continuously delivered application, you will typically disable automatic versioning.
+To do this, set the value of the parameter `versioningType` to the value `library`.
+Architectural details can be found in [automatic-release.md](doc/architecture/decisions/automatic-release.md).
+
+Example:
+
+```yaml
+steps:
+  artifactPrepareVersion:
+    versioningType: library
+```
+
+For more information on how to configure this step, please refer to the documentation in [project "Piper"](https://sap.github.io/jenkins-library/steps/artifactPrepareVersion/).
+
 #### mavenExecute
 The mavenExecute step is used for all invocations of the mvn build tool. It is either used directly for executing specific maven phases such as `test`, or indirectly for steps that execute maven plugins such as `checkPmd`.
 
@@ -680,12 +667,6 @@ The executeNpm step is used for all invocations of the npm build tool. It is, fo
 | `defaultNpmRegistry` | | | The default npm registry url to be used as the remote mirror. Bypasses the local download cache if specified.  |
 | `sapNpmRegistry` | | `https://npm.sap.com` | The default npm registry url to be used as the remote mirror for the SAP npm packages. Bypasses the local download cache if specified.  |
 
-#### executeSourceClearScan
-
-| Property | Mandatory | Default Value | Description |
-| --- | --- | --- | --- |
-| `dockerImage` | | `ppiper/mta-archive-builder` | The image to be used for running SourceClear scan. Must contain a version of Maven (and NPM if you have a frontend) which is capable of building your project. |
-
 #### cloudFoundryDeploy
 A step configuration regarding Cloud Foundry deployment. This is required by stages like end-to-end tests, performance tests, and production deployment.
 

diff --git a/doc/architecture/decisions/lint-stage-flexibility.md b/doc/architecture/decisions/lint-stage-flexibility.md
@@ -24,7 +24,7 @@ The new behavior of the lint stage of the SAP Cloud SDK Pipeline is as follows.
 In addition to the new behavior, we decided to not let the execution of the pipeline fail when linting errors or warnings are discovered.
 The thresholds defined in the pipeline configuration only apply for the built-in check of SAPUI5 components.
 
-ESLint is used as linting tool for the default linting, since it is the most-widely used and feature rich linting tool, at of the time of this writing.  
+ESLint is used as linting tool for the default linting, since it is the most-widely used and feature rich linting tool, at the time of this writing.  
 
 ## Consequences
 

diff --git a/release-notes-draft.md b/release-notes-draft.md
@@ -2,24 +2,169 @@
 
 This document describes the changes which will be part of the next release and are already available in the latest version (master branch) of the pipeline.
 
-## v36
+# v37
 
-### Improvements
+## :warning: Breaking changes
 
-#### Jenkinsfile
+- Support for SourceClear scans was removed from the pipeline.
+  Please use other 3rd party security scanning tools, such as Fortify, Checkmarx, WhiteSource or SonarQube instead.
+- We reimplemented the mechanism how custom defaults are retrieved while implementing some improvements as explained below.
+  Please note that the old parameter `extensionRepository` cannot be used anymore.
+  Instead, please use the option `globalExtensionsRepository`.
+  The option `globalExtensionsRepository` does not allow to specify a version with the option `-b` anymore.
+  Instead, please use the parameter `globalExtensionsVersion` to configure a version.
+  Please note that you can also configure these values as part of your custom defaults / shared configuration.
+  The precedence has not changed.
 
-We updated our bootstrapping Jenkinsfile so that it loads the pipeline directly from the library and not from this repository anymore.
-With this change we decrease the number of repositories checked out and the number of executors during the pipeline run.
+Example:
+
+```diff
+general:
+-  extensionRepository: 'https://my.git.example/extensions.git -b v35'
++  globalExtensionsRepository: 'https://my.git.example/extensions.git'
++  globalExtensionsVersion: 'v35'
+```
+
+### Automatic versioning
+
+Automatic versioning is solely configured via the new step `artifactPrepareVersion` as documented [here](https://sap.github.io/jenkins-library/steps/artifactPrepareVersion/)
+
+For internal use-cases, the pipeline is configuring this step to also push a tag for each generated version.
+This makes it necessary to provide the two parameters `gitHttpsCredentialsId` and `gitUserName`.
+For external use-cases, the default is not to push tags.
+Three types of versioning are supported via the parameter `versioningType`: `cloud`, `cloud_noTag`, and `library`.
+To disable automatic versioning, set the value to `library`.
+The pipeline will then pick up the artifact version configured in the build descriptor file, but not generate a new version.
+
+If you previously turned off automatic versioning via the parameter `automaticVersioning`, this diffs shows the necessary migration of the config file:
+
+```diff
+general:
+-  automaticVersioning: false
+steps:
++  artifactPrepareVersion:
++    versioningType: 'library'
+```
+
+If you previously configured pushing tags for each new version, this is how the configuration can be migrated:
+
+```diff
+steps:
+-  artifactSetVersion:
+-    commitVersion: true
+-    gitCredentialsId: 'Jenkins secret'
+-    gitSshUrl: 'repo-URL'
+-    gitUserEMail: 'email'
+-    gitUserName: 'username'
++  artifactPrepareVersion:
++    versioningType: 'cloud'
++    gitHttpsCredentialsId: 'Jenkins secret'
+```
+
+The repository URL for the project in Jenkins needs to be configured with `https://` scheme.
+It will be used also for pushing the tag.
+
+For Maven projects, the step `mavenExecute` is not used anymore to set the version.
+Instead, it is directly done by `artifactPrepareVersion`, which avoids starting new Docker containers and will improve the performance.
+
+If you have defined a project settings file for `mavenExecute` before, you must move this configuration into the general sections as follows:
+
+```diff
+general:
++  maven:
++    projectSettingsFile: 'settings.xml'
+steps:
+-  mavenExecute:
+-    projectSettingsFile: 'settings.xml'
+```
+
+The same applies to other options defined for `mavenExecute`.
+
+## Fixes
+
+## Improvements
+
+### Authenticated Access for Custom Defaults and Global Pipeline Extensions
+
+We updated the download mechanism for custom defaults and global pipeline extensions so that both can also be stored on
+locations which are secured by basic authentication.
+
+If you specify the option for `customDefaultsCredentialsId` in the section general of the configuration, the username and password will be used for accessing all custom defaults urls defined in the section customDefaults.
+Please find further details here: https://sap.github.io/jenkins-library/steps/setupCommonPipelineEnvironment/
+If you specify the option for `globalExtensionsRepositoryCredentialsId` in the section general of the configuration, the username and password will be used for cloning the extension repository.
+
+### Specify Global Pipeline Extensions in Shared Config
+
+Now it is possible to configure all configuration options regarding the global extensions as part of your custom defaults / shared configuration.
+
+### Lint Stage
+
+The pipeline can be configured to fail based on linting findings using the `failOnError` configuration option. 
+By default, the pipeline does not fail based on lint findings.
+This option is available when providing a custom linting script or when relying on the default linting of the pipeline.
+It is not available when using the SAPUI5 best practices linter, which is using thresholds instead.
+
+```diff
+steps:
++  npmExecuteLint:
++    failOnError: true
+```
+
+### Jenkinsfile
+
+We updated our bootstrapping Jenkinsfile so that it loads the pipeline directly from the library and not from the `cloud-s4-sdk-pipeline` repository anymore.
+This change improves efficiency of the pipeline because it decreases the number of repositories checked out and the number of executors during the pipeline run.
 
 The new version can be found here: https://github.com/SAP/cloud-s4-sdk-pipeline/blob/master/archetype-resources/Jenkinsfile
 
-You have to adopt the Jenkinsfile accordingly to the following change:
+To benefit from the improved efficiency, please update your Jenkinsfile like this:
+
 ```diff
 -node {
 -    deleteDir()
--    sh "git clone --depth 1 https://github.com/SAP/cloud-s4-sdk-pipeline.git -b ${pipelineVersion} pipelines"	
--    load './pipelines/s4sdk-pipeline.groovy'	
--}	
+-    sh "git clone --depth 1 https://github.com/SAP/cloud-s4-sdk-pipeline.git -b ${pipelineVersion} pipelines"
+-    load './pipelines/s4sdk-pipeline.groovy'
+-}
 +library "s4sdk-pipeline-library@${pipelineVersion}"
 +cloudSdkPipeline(script: this)
 ```
+
+### Merged "build" stage with Project "Piper" general purpose pipeline
+
+In an effort to reduce differences between _project "Piper" general purpose pipeline_ and _SAP Cloud SDK pipeline_, both use the same stage for "Build and Unit Test" now.
+This is a change under the hood and it should not require any changes in your project in most cases.
+
+A notable exception is the JavaScript pipeline.
+
+First, the new pipeline does not only run ci scripts in the top-level `package.json` file, but also in sub-directories.
+This only applies if the `package.json` file implements the respective scripts.
+Older pipeline versions required you to orchestrate the build inside nested `package.json` files.
+If your project has a build setup where the top level `package.json` file takes care of building sub-modules, please take care that this should not be done anymore.
+
+This is the list of scripts that are automatically executed by the pipeline, if they are implemented:
+
+- `ci-build`
+- `ci-backend-unit-test`
+- `ci-package`
+
+This might in particular be an issue if any of the mentioned scripts uses [lerna](https://lerna.js.org/).
+If you still want lerna to orchestrate the execution, make sure to use the mentioned names in the `package.json` files in the root of your project, and to use different names in sub-modules.
+
+You can try this out locally by running `piper npmExecuteScripts --install --runScripts=ci-build,ci-backend-unit-test,ci-package` in the [project "Piper" cli](https://sap.github.io/jenkins-library/cli/).
+
+Second, the new pipeline does not run `ci-package` in an isolated file system anymore.
+Therefore, make sure that `ci-package` only changes the deployment directory.
+
+If your JavaScript/TypeScript project was generated from an older template project, you might have to adjust the `ci-package` command like so:
+
+```diff
+"scripts": {
+-     "ci-package": "mkdir -p deployment/ && npm prune --production && cp -r node_modules dist package.json package-lock.json frontend index.html deployment/",	
++     "ci-package": "sap-cloud-sdk package --include=dist/**/*,package.json,package-lock.json,frontend/**/*,index.html",
+},
+  "devDependencies": {
++    "@sap-cloud-sdk/cli": "^0.1.9",
+}
+```
+
+If you notice any regressions, please let us know by opening an [issue](https://github.com/sap/cloud-s4-sdk-pipeline/issues).
diff --git a/s4sdk-pipeline.groovy b/s4sdk-pipeline.groovy
@@ -1,6 +1,6 @@
 #!/usr/bin/env groovy
 
-final def pipelineSdkVersion = 'v36'
+final def pipelineSdkVersion = 'v37'
 
 library "s4sdk-pipeline-library@${pipelineSdkVersion}"
 cloudSdkPipeline(script: this)