Skip to content
Permalink
Browse files
refactored routines for creating LICENSE and NOTICE files
see README.md for info. note many other dependent commits in many projects.
  • Loading branch information
ahgittin committed Jun 26, 2018
1 parent 5e1d313 commit 527d48e9e731831d17e6d4cb8752c54afb101f05
Show file tree
Hide file tree
Showing 56 changed files with 2,047 additions and 2,486 deletions.

This file was deleted.

@@ -1,93 +1,171 @@

We use a special maven plugin and scripts to automatically create the LICENSE
files, including all notices, based on metadata in overrides.yaml and **/source-inclusions.yaml.
We use a special maven plugin and scripts to automatically create the LICENSE and NOTICE
files, including all notices, based on metadata in `license-*` files.

First install https://github.com/ahgittin/license-audit-maven-plugin


# Update license information
# Quick Usage - Update LICENSE and NOTICE Files

**TODO**: Need to re-work the script for the first Karaf-based release.
Ensure all projects are at the right version and built.
Then in the `brooklyn-dist/dist/licensing` folder execute:

* Check if new projects are added to `brooklyn-client/cli/glide.yaml`; compare to and update `/brooklyn-client/cli/release/license/source-inclusions.yaml` && `overrides.yaml`
* Check for new/updated projects in UI (in `src/main/webapp/assets/js/libs`, `src/main/webapp/assets/css`); compare to and update `brooklyn-jsgui/src/main/license/source-inclusions.yaml` && `overrides.yaml`
* Check for newly introduced non-jar dependencies, add create corresponding `source-inclusions.yaml`. For example `brooklyn-server/server-cli/src/main/license/source-inclusions.yaml`.
./generate-all.sh

In the `brooklyn-dist/dist/licensing` folder execute:
This will generate updated LICENSE and NOTICE files everywhere that is needed.
Compare the differences across all projects just to be sure and commit them.

./make-all-licenses.sh || ( echo 'FAILED!!!' && rm LICENSE.autogenerated )
* Check that the `license-inclusions-*` files in the various projects are up-to-date.
This may need manual verification for Go and JS contents (see glide.yaml for Go and package.json for JS).


Cross check with running `mvn project-info-reports:dependencies` and checking the output at `target/site/dependencies.html`
# Detailed Usage

# Quick Usage
If dependencies change, it may be necessary to supply new metadata.
First it is useful to know how these routines work.

In the usage/dist/ project of Brooklyn...
It builds for various modes depending where the item is being used:

To see a tree of license info:
* The projects and JARs strictly speaking only require any 3rd party items
included therein. There usually aren't any (one exception is JS has been checked out
locally). Where these are required, they should be in a `license-inclusions-source-*` file.
This mode is called `binary-omitted`. (For projects where there are no such inclusions,
the build adds a stock Apache LICENSE and Apache Brooklyn NOTICE.)

* Because people will usually build these and need the resulting information, and
because attribution is good, we include details of runtime dependencies in some places
(e.g. the root of projects) in a separate section in the NOTICE. These dependencies can
be specified in files matching `license-inclusions-binary-*` (this is not needed for `mvn`
deps which are inferred automatically). This mode is called `binary-additional`.

mvn org.heneveld.maven:license-audit-maven-plugin:report \
-Dformat=summary \
-DlicensesPreferred=ASL2,ASL,EPL1,BSD-2-Clause,BSD-3-Clause,CDDL1.1,CDDL1,CDDL \
-DoverridesFile=licensing/overrides.yaml \
-DextrasFiles=`cat licensing/extras-files`
* The TGZ includes third-party dependencies and so needs LICENSE and NOTICE updates
for everything that is bundled, including all `license-inclusions-{source,binary}-*`.
This mode is called `binary-primary`.

The generation then proceeds by collecting the relevant `license-inclusions-*` under the project
directory, giving a list of project `id` fields, and collecting `license-metadata-*` files containing
entries with the same `id` and other metadata. (The `license-metadata-*` allows metadata to be shared
across `id`s used in `license-inclusions` for many projects. The `license-metadata` files are usually
in this directory, but if there is reasons they may be project-specific or even supplied in the
`license-inclusions` files.)

To create the LICENSE files needed for Apache Brooklyn
(consisting of: one in the root for the source build, in the dist project for the binary build,
and in `projects-with-custom-licenses` for those JARs which include other source code):
The generation invokes the `license-audit-maven-plugin:notices` mojo to generate YAML
for the dependencies. These are combined with text files (in `parts/`) to create the NOTICE file.
The NOTICE file is scanned for licenses and text pulled from `license-text/` to create the LICENSE file.

pushd licensing
./make-all-licenses.sh || ( echo 'FAILED!!!' && rm LICENSE.autogenerated )
popd
## Adding New Dependencies

This combines the relevant `source-inclusions.yaml` files to create `extrasFile` files
for `license-audit-maven-plugin` then runs the `notices` target for the various dists we make.
If you need to add a new project to those which require custom LICENSE files, see below.
Add the dependencies to the relevant `license-inclusions` file, and
add metadata to the relevant `license-metadata` file. That's it.

Note that many of the dependencies are detected automatically, but their metadata might
not be perfectly inferred. You can add/edit the corresponding items in `license-metadata`
to override the data that is detected automatically.

# CSV License Report
## Giving Attribution

If you need to generate a CSV report of license usage, e.g. for use in a spreadsheet:
Note that _most_ licenses require attribution. This is done through a `notice` block in the metadata.
When adding new dependencies, check whether the specific license includes a copyright notice.

mvn org.heneveld.maven:license-audit-maven-plugin:report \
-Dformat=csv \
-DlistDependencyIdOnly=true \
-DsuppressExcludedDependencies=true \
-DlicensesPreferred=ASL2,ASL,EPL1,BSD-2-Clause,BSD-3-Clause,CDDL1.1,CDDL1,CDDL \
-DoverridesFile=licensing/overrides.yaml \
-DextrasFiles=`cat licensing/extras-files` \
-DoutputFile=dependencies-licenses.csv
## Adding New Licenses

If a new license is required, simply create a file named with the license name in `license-text/`
containing the text of the license. The license name can be set with a `license: { name: "..." }`
block in the corresponding `id` in `license-metadata`.

# When Projects Need Custom Licenses
Note that most projects use one of a small number of licenses, but with custom attribution,
either a "Copyright" line in the license or an accompanying NOTICE file. In such cases you
should use the shared license (so it is easier for people to see that the sub-license is
acceptable to them), and put the attribution/notice in a `notice` block in the `license-metadata`,
as described above.

If a JAR includes 3rd party source code (as opposed to binary dependencies), it will typically
require a custom LICENSE file.
## New Source Projects

To support this:
If creating a new project, check whether it requires special NOTICE / LICENSE files.
If so:

1. Add the relative path to that project to `projects-with-custom-licenses`,
1. Create `license-inclusions-*` files as needed, usually in that project at `src/main/license/`
2. Add instructions to that project's POM to include `src/main/license/files/*`
3. Add the reference to the project in `generate-all.sh`

2. Create the necessary file structure and maven instructions:

* in `src/main/license/` put a `README.md` pointing here
* in `src/main/license/` put a `source-inclusion.yaml` listing the included 3rd-party projects by an `id:`
(you may need to choose an ID; this should be a meaningful name, e.g. `typeahead.js` or `Swagger UI`)
* in `src/main/license/files` put the relevant non-LICENSE license files you want included (e.g. NOTICE)
* in `src/test/license/files` put the standard files *including* LICENSE to include in the test JAR
(NB: these scripts don't generate custom licenses for test JARs, as so far that has not been needed)
* in `pom.xml` add instructions to include `src/{main,test}/license/files` in the main and test JARs
# Reports

You can follow the pattern done for `cli` and `jsgui`.
In the usage/dist/ project of Brooklyn

3. In `licensing/overrides.yaml` in this directory, add the metadata for the included projects.
To see a tree of license info:

4. In `licensing/licenses/<your_project>/` include the relevant licenses for any included 3rd-party projects;
look in `licensing/licenses/binary` for samples.
mvn org.heneveld.maven:license-audit-maven-plugin:report \
-Dformat=summary \
-DlicensesPreferred=ASL2,ASL,EPL1,BSD-2-Clause,BSD-3-Clause,CDDL1.1,CDDL1,CDDL \
-DoverridesFile=licensing/overrides.yaml \
-DextrasFiles=`cat licensing/extras-files`

## CSV License Report

5. Run `make-all-licenses.sh` to auto-generate required config files and license copies,
and to generate the LICENSE for your project.
If you need to generate a CSV report of license usage, e.g. for use in a spreadsheet:

Confirm that a LICENSE file for your project was generated, and that it is present in the JAR,
and then open a pull-request, and confirm the changes there are appropriate.
mvn org.heneveld.maven:license-audit-maven-plugin:report \
-Dformat=csv \
-DlistDependencyIdOnly=true \
-DsuppressExcludedDependencies=true \
-DlicensesPreferred=ASL2,ASL,EPL1,BSD-2-Clause,BSD-3-Clause,CDDL1.1,CDDL1,CDDL \
-DoverridesFile=licensing/overrides.yaml \
-DextrasFiles=`cat licensing/extras-files` \
-DoutputFile=dependencies-licenses.csv

## Other Tools

The command `mvn project-info-reports:dependencies` is a different tool for checking mvn dependencies;
compare the output at `target/site/dependencies.html` with the previous.

Similarly the `org.apache:apache-jar-resource-bundle` resource bundle computes and reports mvn dependencies
in a `META-INF/DEPENDENCIES` file placed into JARs. This is not complete (but it is elegant how it works!).


# Appendix - POM Setup

The usual process where a project has custom `NOTICE` and `LICENSE` for its _source_
is to write it to the root of that project, and where it is desired in the JAR or a build,
to use a POM config such as the following (note you must declare _all_ resources):

<resources>
<resource>
<directory>${project.basedir}/src/main/resources</directory>
</resource>
<resource>
<targetPath>META-INF</targetPath>
<directory>${project.basedir}</directory>
<includes>
<include>NOTICE</include>
<include>LICENSE</include>
</includes>
</resource>
</resources>

This is necessary where there are embedded depenencies in the resulting JAR or built items.
It is not necessary, and typically not done, where there are no embedded dependencies, as the
`org.apache:apache-jar-resource-bundle` mojo will take the standard ASF files which are sufficient.

If a build needs a different set of files than the source for some artifact (e.g. because it is bundling
dependencies within it) they will usually be created to a separate folder, e.g. `src/main/license/files/MODE/`
and used by the build to add it with a variant of the POM fragment above.

A section such as the following will suppress the installation of default `NOTICE` and `LICENSE` files
(and `DEPENDENCIES`). However it is not needed as the build installs our resources _after_, and it is
useful to keep because it means `-test-` resources get these files.

<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-remote-resources-plugin</artifactId>
<executions>
<execution>
<goals>
<goal>process</goal>
</goals>
<configuration>
<skip>true</skip>
</configuration>
</execution>
</executions>
</plugin>
@@ -0,0 +1,144 @@
#
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
#

set -e

usage() {
cat >&2 << EOF
Usage: generate-all.sh
Execute generate-license-and-notice.sh to generate LICENSE and NOTICE files for all Brooklyn projects.
EOF
}

while [ ! -z "$*" ] ; do

if [ "$1" == "--help" ]; then usage ; exit 0; fi

usage
echo Unexpected argument: $1
exit 1

done


REF_DIR=$(pushd $(dirname $0) > /dev/null ; pwd -P ; popd > /dev/null)
PARTS_DIR=$REF_DIR/parts
ROOT_DIR=$REF_DIR/../../..
MVN_OUTFILE=$REF_DIR/notices.autogenerated

prefix_and_join_array() {
PREFIX=$2
JOIN_BEFORE_PREFIX=$1
JOIN_AFTER_PREFIX=$3
echo -n ${PREFIX}$4
shift 4
while (($# >= 1)) ; do
echo -n "${JOIN_BEFORE_PREFIX}${PREFIX}${JOIN_AFTER_PREFIX}$1"
shift
done
}

# takes root dir in first arg, then regex expression
make_for() {
PROJ=$(cd $1 ; pwd -P)
OUT=${PROJ}/$2
MODE=$3
SEARCH_ROOT=$4
if [ -z "$SEARCH_ROOT" ] ; then SEARCH_ROOT=$PROJ ; fi

echo Generating for $PROJ mode $MODE to $2...
echo ""

pushd $PROJ > /dev/null

if [ "$MODE" == "binary-additional" ] ; then

$REF_DIR/generate-license-and-notice.sh \
-o $OUT \
--license $PARTS_DIR/license-top \
--license $PARTS_DIR/license-deps \
--notice $PARTS_DIR/notice-top --notice-compute-with-flags "
-DextrasFiles=$(prefix_and_join_array "" ":" "" $(find $SEARCH_ROOT -name "license-inclusions-source-*"))
-DonlyExtras=true" \
--notice $PARTS_DIR/notice-additional --notice-compute-with-flags "
-DextrasFiles=$(prefix_and_join_array "" ":" "" $(find $SEARCH_ROOT -name "license-inclusions-binary-*"))" \
--libraries ${REF_DIR} ${SEARCH_ROOT}

elif [ "$MODE" == "binary-primary" ] ; then

$REF_DIR/generate-license-and-notice.sh \
-o $OUT \
--license $PARTS_DIR/license-top \
--license $PARTS_DIR/license-deps \
--notice $PARTS_DIR/notice-top --notice-compute-with-flags "
-DextrasFiles=$(prefix_and_join_array "" ":" "" $(find $SEARCH_ROOT -name "license-inclusions-source-*" -or -name "license-inclusions-binary-*"))" \
--libraries ${REF_DIR} ${SEARCH_ROOT}

elif [ "$MODE" == "binary-omitted" ] ; then

$REF_DIR/generate-license-and-notice.sh \
-o $OUT \
--license $PARTS_DIR/license-top \
--license $PARTS_DIR/license-deps \
--notice $PARTS_DIR/notice-top --notice-compute-with-flags "
-DextrasFiles=$(prefix_and_join_array "" ":" "" $(find $SEARCH_ROOT -name "license-inclusions-source-*"))
-DonlyExtras=true" \
--libraries ${REF_DIR} ${SEARCH_ROOT}

else
echo FAILED - unknown mode $MODE
exit 1
fi
echo ""

popd > /dev/null
}


# build all the projects

# include deps in files pulled in to Go CLI binary builds
make_for $ROOT_DIR/brooklyn-client/cli/ release/license/files binary-primary
make_for $ROOT_DIR/brooklyn-client/cli/ . binary-additional

# Server CLI has embedded JS; gets custom files in sub-project root, also included in JAR
make_for $ROOT_DIR/brooklyn-server/server-cli/ . binary-additional

# UI gets files at root, also included in WAR
make_for $ROOT_DIR/brooklyn-ui/ . binary-additional

# main projects have their binaries included at root
make_for $ROOT_DIR/brooklyn-server/ . binary-additional
make_for $ROOT_DIR/brooklyn-client/ . binary-additional
make_for $ROOT_DIR/brooklyn-library/ . binary-additional
# dist is trickier, just don't mention binaries in the generated items
make_for $ROOT_DIR/brooklyn-dist/ . binary-omitted

# brooklyn-docs skipped
# the docs don't make a build and don't include embedded code so no special license there

# and the binary dists; dist/ project which has biggest deps set, but search in all brooklyn projects
make_for $ROOT_DIR/brooklyn-dist/dist src/main/license/files/ binary-primary $ROOT_DIR
cp $OUT/{NOTICE,LICENSE} $PROJ/../karaf/apache-brooklyn/src/main/resources/

# finally in root project list everything
make_for $ROOT_DIR/brooklyn-dist/dist ../.. binary-additional $ROOT_DIR

0 comments on commit 527d48e

Please sign in to comment.