Packaging: Add LICENSE and NOTICE files for all core dependencies

[clintongormley]

Also adds dev-tools/check_license_and_sha.pl which will ensure that the
sha1 file in the licenses/ directory is the same as the sha1 of each
dependency shipped with Elasticsearch, and that each dependency has
a LICENSE file. Can also be used to update the sha1 file when upgrading
dependencies.

Closes #2794
Closes #10684

[clintongormley]

Not sure if we need to include these files from the groovy JAR:

• groovy-all-ANTLR-LICENSE.txt
• groovy-all-ASM-LICENSE.txt
• groovy-all-CLI-LICENSE.txt
• groovy-all-JSR223-LICENSE.txt

Also, would be good to update maven to execute this at build time:

dev-tools/check_license_and_sha.pl --check core

/cc @dadoonet

[rmuir]

About checking at build time: maybe we don't want to do it quite yet? How long does it take the perl script to run? At least we want to start with it working via jenkins on linux... but I agree we want a way for a developer to manually check it before pushing (if that is running a perl script manually in both cases, fine for a start).

We could later see if it could be made more portable, e.g. just doing some custom logic in maven-ant-run-plugin here (groovy script or something simple). Maybe it won't be so simple if its in groovy but it means it would work without 'shasum' (windows, etc).

Also i couldnt tell if the verification logic will fail on extra license files that shouldn't be there. I see it definitely will for extra sha's: that is really good to see because it keeps everything tidy. Its just an idea for extra paranoia if its not tough to add.

[s1monw]

mvn clean package yields this:

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.8:run (check-invalid-patterns) on project elasticsearch-parent: An Ant BuildException has occured: The following files contain tabs or
[ERROR] nocommits:
[ERROR] * core/licenses/lucene-LICENSE.txt
[ERROR] around Ant part ...<fail if="validate.patternsFound">The following files contain tabs or... @ 24:37 in /Users/simon/projects/elasticsearch/target/antrun/build-main.xml
[ERROR] -> [Help 1]

I think you need to fix you tabs vs. spaces in your editor :)

[clintongormley]

@s1monw tabs removed from Lucene license ;)

[clintongormley]

About checking at build time: maybe we don't want to do it quite yet? How long does it take the perl script to run?

about 150ms :)

We could later see if it could be made more portable, e.g. just doing some custom logic in maven-ant-run-plugin here (groovy script or something simple). Maybe it won't be so simple if its in groovy but it means it would work without 'shasum' (windows, etc).

agreed - although shasum is just a perl script installed by default on most *nixes. I could equally roll the module into the script directly so that it doesn't have the command line tool dependency.

That said, rewriting it in eg java would make it work wherever es works.

Also i couldnt tell if the verification logic will fail on extra license files that shouldn't be there. I see it definitely will for extra sha's: that is really good to see because it keeps everything tidy. Its just an idea for extra paranoia if its not tough to add.

The only reason I didn't was because of the extra licensing files for groovy.

[s1monw]

I think you should move the script into the source folder that way it gets packaged and can be used by plugins etc.

 dev-tools/check_license_and_sha.pl -> dev-tools/src/main/resources/license-check/check_license_and_sha.pl

if you do this then you can use this diff:

diff --git a/pom.xml b/pom.xml
index 304a060..43ce99c 100644
--- a/pom.xml
+++ b/pom.xml
@@ -1148,6 +1148,26 @@ org.eclipse.jdt.ui.text.custom_code_templates=<?xml version\="1.0" encoding\="UT
                     <version>1.8</version>
                     <executions>
                         <execution>
+                            <id>check-license</id>
+                            <phase>verify</phase>
+                            <goals>
+                                <goal>run</goal>
+                            </goals>
+                            <configuration>
+                                <target>
+                                    <condition property="license.exists">
+                                        <available file="${basedir}/license" type="dir"/>
+                                    </condition>
+                                    <echo taskName="license check">Running license check</echo>
+                                    <exec failonerror="${license.exists}" executable="perl" dir="${elasticsearch.tools.directory}/license-check" >
+                                        <arg value="check_license_and_sha.pl"/>
+                                        <arg value="--check"/>
+                                        <arg value="${basedir}"/>
+                                    </exec>
+                                </target>
+                            </configuration>
+                        </execution>
+                        <execution>
                             <id>print-jvm</id>
                             <phase>validate</phase>
                             <goals>

and mvn verify -DskipTests will run the entire build and execute the script on every plugin that has a license folder

[clintongormley]

@rmuir I've extended the file checks to complain about extra LICENSE files, and added a check to be sure that exactly one NOTICE file exists as well.

@s1monw I've added the POM changes so that this check runs automatically.

[s1monw]

LGTM