Linux (OpenJDK 10)	Windows (Oracle JDK 9)	JitPack (OpenJDK 8)

REUSE status	Interact with us!

Introduction

The OSS Review Toolkit (ORT) assists with verifying Free and Open Source Software license compliance by checking a project's source code and its dependencies.

From a bird's eye, it works by analyzing the project's build system for dependencies, downloading the source code of the dependencies, scanning all source code for license information, and summarizing the results.

The different tools that make up ORT are designed as libraries (for programmatic use), with a minimal command line interface (for scripted use).

The toolkit consists of the following tools:

• Analyzer - determines dependencies of a project. Supports multiple package managers and sub-projects. No changes to the projects are required.
• Downloader - fetches the source code referred to by the Analyzer result.
• Scanner - wraps existing license / copyright scanners to detect findings in local source code directories.
• Evaluator - evaluates license findings against customizable policy rules.
• Reporter - presents results in various formats such as visual reports, open source notices or Bill-Of-Materials (BOMs) to easily identify dependencies, licenses, copyrights or policy rule violations.

The following tools are planned but not yet available:

• Advisor - retrieves security advisories based on the Analyzer result.
• Documenter - generates the final outcome of the review process incl. legal conclusions, e.g. annotated SPDX files that can be included into the distribution.

Installation

From binaries

Preliminary binary artifacts for ORT are currently available via JitPack. Please note that due to limitations with the JitPack build environment, the reporter is not able to create the Web App report.

From sources

Install the following basic prerequisites:

• Git (any recent version will do).

Then clone this repository. If you intend to run tests, you need to clone with submodules by running git clone --recurse-submodules. If you have already cloned non-recursively, you can initialize submodules afterwards by running git submodule update --init --recursive.

Build using Docker

Install the following basic prerequisites:

• Docker (and ensure its daemon is running).

Change into the directory with ORT's source code and run docker build -t ort ..

Build natively

Install these additional prerequisites:

• OpenJDK 8 or Oracle JDK 8u161 or later (not the JRE as you need the javac compiler); also remember to set the JAVA_HOME environment variable accordingly.

Change into the directory with ORT's source code and run ./gradlew installDist (on the first run this will bootstrap Gradle and download all required dependencies).

Basic usage

ORT can now be run using

./cli/build/install/ort/bin/ort --help

Note that if you make any changes to ORT's source code, you would have to regenerate the distribution using the steps above.

To avoid that, you can also build and run ORT in one go (if you have the prerequisites from the Build natively section installed):

./gradlew cli:run --args="--help"

Note that in this case the working directory used by ORT is that of the cli project, not the directory gradlew is located in (see gradle/gradle#6074).

Running the tools

Like for building ORT from sources you have the option to run ORT from a Docker image (which comes with all runtime dependencies) or to run ORT natively (in which case some additional requirements need to be fulfilled).

Run using Docker

After you have built the image as described above, simply run docker run <DOCKER_ARGS> ort <ORT_ARGS>. You typically use <DOCKER_ARGS> to mount the project directory to analyze into the container for ORT to access it, like:

docker run -v /workspace:/project ort --info analyze -f JSON -i /project -o /project/ort/analyzer

Run natively

First of all, make sure that the locale of your system is set to en_US.UTF-8 as using other locales might lead to issues with parsing the output of some external tools.

Then install any missing external command line tools as listed by

./cli/build/install/ort/bin/ort requirements

or

./gradlew cli:run --args="requirements"

Then run ORT like

./cli/build/install/ort/bin/ort --info analyze -f JSON -i /project -o /project/ort/analyzer

or

./gradlew cli:run --args="--info analyze -f JSON -i /project -o /project/ort/analyzer"

Running on CI

A basic ORT pipeline (using the analyzer, scanner and reporter) can easily be run on Jenkins CI by using the Jenkinsfile in a (declarative) pipeline job. The job accepts various parameters that are translated to ORT command line arguments. Additionally, one can trigger a downstream job which e.g. further processes scan results. Note that it is the downstream job's responsibility to copy any artifacts it needs from the upstream job.

Getting started

Please see Getting Started for an introduction to the individual tools.

Configuration

Please see the documentation below for details about the ORT configuration.

• The .ort.yml file - project-specific license finding curations, exclusions and resolutions to address issues found within a project's code repository.
• The curations.yml file - curations correct invalid or missing package metadata and set the concluded license for packages.
• The resolutions.yml file - resolutions allow resolving any issues or policy rule violations by providing a reason why they are acceptable and can be ignored.

Details on the tools

The analyzer is a Software Composition Analysis (SCA) tool that determines the dependencies of software projects inside the specified input directory (-i). It does so by querying the detected package managers; no modifications to your existing project source code, like applying build system plugins, are necessary for that to work. The tree of transitive dependencies per project is written out as part of an OrtResult in YAML (or JSON, see -f) format to a file named analyzer-result.yml in the specified output directory (-o). The output file exactly documents the status quo of all package-related meta-data. It can be further processed or manually edited before passing it to one of the other tools.

Currently, the following package managers are supported:

• Bower (JavaScript)
• Bundler (Ruby)
• Cargo (Rust)
• Conan (C / C++, experimental as the VCS locations often times do not contain the actual source code, see #2037)
• dep (Go)
• DotNet (.NET, with currently some limitations)
• Glide (Go)
• Godep (Go)
• GoMod (Go, experimental as only proxy-based source artifacts but no VCS locations are supported)
• Gradle (Java)
• Maven (Java)
• NPM (Node.js)
• NuGet (.NET, with currently some limitations)
• Composer (PHP)
• PIP (Python)
• Pipenv (Python)
• Pub (Dart / Flutter)
• SBT (Scala)
• Stack (Haskell)
• Yarn (Node.js)

 

Taking an ORT result file with an analyzer result as the input (-a), the downloader retrieves the source code of all contained packages to the specified output directory (-o). The downloader takes care of things like normalizing URLs and using the appropriate VCS tool to checkout source code from version control.

Currently, the following Version Control Systems are supported:

• CVS
• Git
• Git-Repo
• Mercurial
• Subversion

 

This tool wraps underlying license / copyright scanners with a common API so all supported scanners can be used in the same way to easily run them and compare their results. If passed an ORT result file with an analyzer result (-a), the scanner will automatically download the sources of the dependencies via the downloader and scan them afterwards.

Currently, the following license scanners are supported:

• Askalono
• lc
• Licensee
• ScanCode

For a comparison of some of these, see this Bachelor Thesis.

Storage Backends

In order to not download or scan any previously scanned sources again, the scanner can use a storage backend to store scan results for later reuse.

Local File Storage

By default the scanner stores scan results on the local file system in the current user's home directory (i.e. ~/.ort/scanner/scan-results) for later reuse. The storage directory can be customized by passing an ORT configuration file (-c) that contains a respective local file storage configuration:

ort {
  scanner {
    fileBasedStorage {
      backend {
        localFileStorage {
          directory = "/tmp/ort/scan-results"
          compression = false
        }
      }
    }
  }
}

HTTP Storage

Any HTTP file server can be used to store scan results. Custom headers can be configured to provide authentication credentials. For example, to use Artifactory to store scan results, use the following configuration:

ort {
  scanner {
    fileBasedStorage {
      backend {
        httpFileStorage {
          url = "https://artifactory.domain.com/artifactory/repository/scan-results"
          headers {
            X-JFrog-Art-Api = "api-token"
          }
        }
      }
    }
  }
}

PostgreSQL Storage

To use PostgreSQL to store scan results, use the following configuration:

ort {
  scanner {
    postgresStorage {
      url = "jdbc:postgresql://example.com:5444/database"
      schema = "schema"
      username = "username"
      password = "password"
      sslmode = "verify-full"
    }
  }
}

The scanner creates a table called scan_results and stores the data in a jsonb column.

If you do not want to use SSL set the sslmode to disable, other possible values are explained in the documentation. For other supported configuration options see PostgresStorageConfiguration.kt.

 

The evaluator is used to perform custom license policy checks on scan results. The rules to check against are implemented as scripts (currently Kontlin scripts, with a dedicated DSL, but support for other scripting can be added as well. See rules.kts for an example file.

 

The reporter generates human-readable reports from the scan result file generated by the scanner (-s). It is designed to support multiple output formats.

Currently, the following report formats are supported (reporter names are case-insensitive):

• CycloneDX BOM (-f CycloneDx)
• Excel sheet (-f Excel)
• NOTICE file in two variants
  • List license texts and copyrights by package (-f NoticeByPackage)
  • Summarize all license texts and copyrights (-f NoticeSummary)
• Static HTML (-f StaticHtml)
• Web App (-f WebApp)

Development

ORT is written in Kotlin and uses Gradle as the build system, with Kotlin script instead of Groovy as the DSL.

When developing on the command line, use the committed Gradle wrapper to bootstrap Gradle in the configured version and execute any given tasks. The most important tasks for this project are:

Task	Purpose
assemble	Build the JAR artifacts for all projects
detekt	Run static code analysis on all projects
test	Run unit tests for all projects
funTest	Run functional tests for all projects
installDist	Build all projects and install the start scripts for distribution

All contributions need to pass the detekt, test and funTest checks before they can be merged.

For IDE development we recommend the IntelliJ IDEA Community Edition which can directly import the Gradle build files. After cloning the project's source code recursively, simply run IDEA and use the following steps to import the project.

1. From the wizard dialog: Select Import Project.

   From a running IDEA instance: Select File -> New -> Project from Existing Sources...

2. Browse to ORT's source code directory and select either the build.gradle.kts or the settings.gradle.kts file.

3. In the Import Project from Gradle dialog select Use auto-import and leave all other settings at their defaults.

To set up a basic run configuration for debugging, navigate to Main.kt in the cli module and look for the fun main(args: Array<String>) function. In the gutter next to it, a green "Play" icon should be displayed. Click on it and select Run 'com.here.ort.Main' to run the entry point, which implicitly creates a run configuration. Double-check that running ORT without any arguments will simply show the command line help in IDEA's Run tool window. Finally, edit the created run configuration to your needs, e.g. by adding an argument and options to run a specific ORT sub-command.

License

Copyright (C) 2017-2019 HERE Europe B.V.

See the LICENSE file in the root of this project for license details.