Managing Off-line Scans using Synopsys Black Duck

This project supplies a script which wraps Detect to show how to:

1. Generate scans off-line and store them
2. Create custom field data for the project-version that the scans will (later) be mapped to
3. How to upload the scans (later) using the Black Duck REST API and the blackduck PYPI library

References

• https://blackducksoftware.github.io/synopsys-detect/6.1.0/
• https://blackducksoftware.github.io/synopsys-detect/6.1.0/30-running/

Setup

1. Download the (latest) detect jar and place it in the detect_files folder

2. Download the signature scanner cli for whichever Black Duck version you have and place it in the detect_files folder

3. Install whatever package manager tools are needed

   • This repository includes a set of test files in test_project that includes a sample maven and node/npm project
   • For Synopsys Detect to inspect the maven and node/npm project files you must install maven and npm

4. Install the Python3 requirements using the supplied requirements.txt file, e.g.

   pip3 install -r requirements.txt

5. Create a .restconfig.json file to provide the blackduck PyPi library (installed in previous step) with the information to connect with your Black Duck server. See https://github.com/blackducksoftware/hub-rest-api-python/blob/master/restconfig.json.api_token.example for a sample file.

Downloading Synopsys Detect and the Signature Scanner CLI

One of the easiest ways to download both the Detect jar and the signature scanner is to run Synopsys Detect on-line with --detect.cleanup=false. Then:

1. Copy the detect jar from the /tmp folder
2. Copy the signature scanning CLI from ~/blackduck/tools/Black_Duck_Scan_Installation/scan.cli-version

When you have downloaded Synopsys Detect and the signature scanner cli, your detect_files folder should look something like this,

$ ls ~/detect_files/
scan.cli-2019.10.3/        scan.cli-2019.12.1/        scan.cli-2019.4.3/         scan.cli-2020.2.1/         scan.cli-2020.4.0/         synopsys-detect-6.2.1.jar

Note that in this instance signature scanner cli's were downloaded for multiple versions of Black Duck:

• v2019.4.3
• v2019.10.3
• v2019.12.1
• v2020.4.0

Synopsys Detect v6.2.1 was downloaded and is shown above.

Running the Script to Generate Off-line Scans

You need to edit and set the following parameters before running the script:

1. BD_VERSION if different from v2020.4.0
2. SCAN_CLI_VERSION if different from 2020.4.0 (note the 'v' is removed)

Once that is done you can run the script by doing,

./run_detect_local.bash

See the sample output in detect.log from the above to see what you should get.

The scan files, a file containing custom field values, and a manifest will be stored into a version-specific folder that is created to store the output files into. It will look something like this,

$ ls v2020.4.0/
custom-field-values.json                              test_project_1_0_maven_bom.jsonld
gsnyder-mac-test_project-2020-05-15T193210.648Z.json  test_project_1_0_npm_bom.jsonld
manifest.json

Uploading Scan Files

A python script is supplied for creating the Project Version custom fields on the Black Duck server which should be run one-time (or just create the fields manually using the BD GUI), e.g.

python3 create_custom_fields.py

Then, to upload the scan files along with custom field values generated by the run_detect_local.bash script do,

python3 upload_scans.py v2020.4.0/manifest.json

If you want to upload the scans and map them to a different project and/or version do,

python3 upload_scans.py v2020.4.0/manifest.json -p new-project -v new-version

The python script will modify the scan files, and custom field file, on-the-fly to re-map all the data to the desired project and/or version.