Merge branch 'master' of https://github.com/apache/incubator-metron

# By cstella (4) and others # Via cstella * 'master' of https://github.com/apache/incubator-metron: METRON-457 Correct GrokParser logging spelling error (mmiklavc via cestella) closes apache#274 METRON-449 JSONMapParser should unfold maps to arbitrary depths closes apache#271 METRON-453: Add a stellar shell function to open an external editor and return the editor's contents closes apache#272 METRON-452: Add rudimentary configuration management functions to Stellar closes apache#269 METRON-374: Add appropriate bundled 3rd party licenses to NOTICE and LICENSE where appropriate closes apache#229 flush METRON-427 Create Ambari Management Pack for Metron Installation closes apache#266
Cognami · Sep 27, 2016 · 8f83fb4 · 8f83fb4
2 parents 1e45d05 + 5378f0c
commit 8f83fb4
Show file tree

Hide file tree

Showing 143 changed files with 10,147 additions and 1,126 deletions.
diff --git a/.gitignore b/.gitignore
@@ -20,3 +20,4 @@ tmp/**/*
 temp/**
 temp/**/*
 
+repodata/
diff --git a/.travis.yml b/.travis.yml
@@ -3,7 +3,7 @@ jdk:
   - oraclejdk8
 script:
   - |
-    mvn -q integration-test install
+    mvn -q integration-test install && build_utils/verify_licenses.sh
 cache:
   directories:
   - $HOME/.m2
diff --git a/README.md b/README.md
@@ -1,7 +1,7 @@
 [![Build Status](https://travis-ci.org/apache/incubator-metron.svg?branch=master)](https://travis-ci.org/apache/incubator-metron)
 
 # Metron
-
+ 
 Metron integrates a variety of open source big data technologies in order
 to offer a centralized tool for security monitoring and analysis. Metron
 provides capabilities for log aggregation, full packet capture indexing,

diff --git a/build_utils/README.md b/build_utils/README.md
@@ -0,0 +1,40 @@
+# Build Utilities
+
+The aim of the build utilities project is to provide some scripting
+around the care and maintenance of the building infrastructure.  At the
+moment the primary mission is around utilities that assist us manage the
+licenses of our dependencies and generate the appropriate notices or
+licenses.
+
+## `dependencies_with_url.csv`
+This file is the reference file for all of our dependencies.
+If you add a dependency, you must add a line to the
+`dependencies_with_url.csv` file.
+
+## `list_dependencies.sh`
+
+List all of the transitive dependencies for the project rooted at cwd.
+
+## `verify_licenses.sh`
+
+This script, as run by our travis build infrastructure, will look at the
+dependencies and verify that we know about them. Travis will use this
+script which takes the transitive dependency list and check against the 
+`dependencies_with_url.csv` file to ensure that it's listed. This will
+make sure we track dependencies and do not have any unacceptable
+dependencies.
+
+If you want to dump all of the dependencies that it doesn't know about,
+from the top level directory:
+`build_utils/list_dependencies.sh | python build_utils/verify_license.py ./dependencies_with_url.csv dump`
+
+## `create_bundled_licenses.sh`
+
+This script is intended to regenerate the licenses for each project that
+bundles its dependencies.  Because we bundle our dependencies in a
+shaded jar, we
+[must](http://www.apache.org/dev/licensing-howto.html#deps-of-deps) specify a `LICENSE` file with the permissively
+licensed dependencies notated as per [here](http://www.apache.org/dev/licensing-howto.html#permissive-deps)
+
+Example command to regenerate licenses (run from top level directory):
+`for i in $(find . -name LICENSE | grep src | grep META-INF | awk -Fsrc '{print $1}');do build_utils/create_bundled_licenses.sh $i;done`
diff --git a/build_utils/create_bundled_licenses.sh b/build_utils/create_bundled_licenses.sh
@@ -0,0 +1,23 @@
+#!/bin/bash
+#
+#  Licensed to the Apache Software Foundation (ASF) under one or more
+#  contributor license agreements.  See the NOTICE file distributed with
+#  this work for additional information regarding copyright ownership.
+#  The ASF licenses this file to You under the Apache License, Version 2.0
+#  (the "License"); you may not use this file except in compliance with
+#  the License.  You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+#
+
+TLD=$(pwd)
+LICENSE_FILE="src/main/resources/META-INF/LICENSE"
+pushd $1
+$TLD/build_utils/list_dependencies.sh | python $TLD/build_utils/generate_license.py $TLD/dependencies_with_url.csv $TLD/LICENSE 1> $LICENSE_FILE
+popd
diff --git a/build_utils/generate_license.py b/build_utils/generate_license.py
@@ -0,0 +1,108 @@
+#
+#  Licensed to the Apache Software Foundation (ASF) under one or more
+#  contributor license agreements.  See the NOTICE file distributed with
+#  this work for additional information regarding copyright ownership.
+#  The ASF licenses this file to You under the Apache License, Version 2.0
+#  (the "License"); you may not use this file except in compliance with
+#  the License.  You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+#
+
+import sets
+import sys
+
+category_a_licenses = [ "BSD Software License", 
+                        "Creative Commons License",
+                        "Common Development and Distribution License", 
+                        "Common Development and Distribution License v1.1",
+                        "Common Development and Distribution License v1.0",
+                        "Common Public License v1.0",
+                        "Eclipse Public License v1.0",
+                        "MIT Software License",
+                        "Mozilla Public License v2.0"
+                     ]
+
+
+license_mapping = {
+    "ASLv2": "Apache Software License v2",
+    "Apache 2": "Apache Software License v2",
+    "Apache 2.0": "Apache Software License v2",
+    "Apache License": "Apache Software License v2",
+    "Apache License 2.0": "Apache Software License v2",
+    "Apache License V2.0": "Apache Software License v2",
+    "Apache License Version 2.0": "Apache Software License v2",
+    "Apache Software Licenses": "Apache Software License v2",
+    "Apache v2": "Apache Software License v2",
+    "The Apache Software License": "Apache Software License v2",
+    "BSD": "BSD Software License",
+    "BSD 2-clause": "BSD Software License",
+    "BSD 3-Clause \"New\" or \"Revised\" License (BSD-3-Clause)" : "BSD Software License",
+    "BSD 3-Clause License": "BSD Software License",
+    "BSD 3-clause": "BSD Software License",
+    "BSD-like": "BSD Software License",
+    "CC0 1.0 Universal": "Creative Commons License",
+    "CDDL": "Common Development and Distribution License",
+    "CDDL 1.1": "Common Development and Distribution License v1.1",
+    "COMMON DEVELOPMENT AND DISTRIBUTION LICENSE (CDDL) Version 1.0": "Common Development and Distribution License v1.0",
+    "Common Development and Distribution License (CDDL) v1.0": "Common Development and Distribution License v1.0",
+    "Common Public License Version 1.0": "Common Public License v1.0",
+    "Eclipse Public License 1.0": "Eclipse Public License v1.0",
+    "MIT" : "MIT Software License",
+    "MIT License" : "MIT Software License",
+    "Mozilla Public License Version 2.0" : "Mozilla Public License v2.0",
+    "New BSD License" : "BSD Software License",
+    "New BSD license" : "BSD Software License",
+    "Public" : "Public Domain",
+    "Public Domain" : "Public Domain",
+    "The BSD 3-Clause License" : "BSD Software License",
+    "The BSD License" : "BSD Software License",
+    "The MIT License" : "MIT Software License",
+    "ACCEPTABLE" : "Ignore"
+}
+
+def read_component(i):
+    with open(i, 'r') as fp:
+        component_lines = fp.readlines()
+        ret = {}
+        for line in component_lines:
+            if len(line) > 0:
+                tokens = line.split(',')
+                key = tokens[0]
+                url = tokens[-1].strip()
+                license = license_mapping[tokens[1].strip()]
+                if license is None:
+                    raise ValueError("unable to normalize license: " + tokens[1])
+                l = line.split(',')[0].strip()
+                ret[key] = { 'url' : url, 'license' : license }
+        return ret
+
+def read_license(f) :
+    with open(f) as fp:
+        return fp.read()
+
+def get_blurb(component, license_info):
+    tokens = component.split(':')
+    artifact_id = tokens[1]
+    version = tokens[3]
+    return "This product bundles " + artifact_id + " " + version + ", which is available under a \"" + license_info['license'] + "\" license.  " + "For details, see " + license_info['url']
+
+if __name__ == '__main__':
+    components = read_component(sys.argv[1])
+    license = read_license(sys.argv[2]) 
+    for line in sys.stdin:
+        component = line.strip() 
+        if len(component) == 0 or component == 'none' or component not in components:
+            continue
+        else:
+            license_info = components[component]
+            if license_info['license'] in category_a_licenses:
+                license = license + "\n" +  get_blurb(component, license_info)
+            continue
+    print license
diff --git a/build_utils/list_dependencies.sh b/build_utils/list_dependencies.sh
@@ -0,0 +1,19 @@
+#!/bin/bash
+#
+#  Licensed to the Apache Software Foundation (ASF) under one or more
+#  contributor license agreements.  See the NOTICE file distributed with
+#  this work for additional information regarding copyright ownership.
+#  The ASF licenses this file to You under the Apache License, Version 2.0
+#  (the "License"); you may not use this file except in compliance with
+#  the License.  You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+#
+
+mvn dependency:list | grep "^\[INFO\]   " | awk '{print $2}' | grep -v "org.apache" | grep -v "test" | grep -v "provided" | grep -v "runtime" | grep -v ":system" |  sort | uniq 
diff --git a/build_utils/verify_license.py b/build_utils/verify_license.py
@@ -0,0 +1,41 @@
+#
+#  Licensed to the Apache Software Foundation (ASF) under one or more
+#  contributor license agreements.  See the NOTICE file distributed with
+#  this work for additional information regarding copyright ownership.
+#  The ASF licenses this file to You under the Apache License, Version 2.0
+#  (the "License"); you may not use this file except in compliance with
+#  the License.  You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+#
+
+import sets
+import sys
+
+def read_component(i):
+    with open(i, 'r') as fp:
+        component_lines = fp.readlines()
+        ret = []
+        for line in component_lines:
+            if len(line) > 0:
+                l = line.split(',')[0].strip()
+                ret.append(l)
+        return sets.Set(ret)
+
+if __name__ == '__main__':
+    components = read_component(sys.argv[1])
+    for line in sys.stdin:
+        component = line.strip() 
+        if len(component) == 0 or component == 'none' or component in components:
+            continue
+        else:
+            if len(sys.argv) > 2:
+                print component
+            else:
+                raise ValueError("Unable to find " + component + " in acceptable list of components: " + sys.argv[1])
diff --git a/build_utils/verify_licenses.sh b/build_utils/verify_licenses.sh
@@ -0,0 +1,19 @@
+#!/bin/bash
+#
+#  Licensed to the Apache Software Foundation (ASF) under one or more
+#  contributor license agreements.  See the NOTICE file distributed with
+#  this work for additional information regarding copyright ownership.
+#  The ASF licenses this file to You under the Apache License, Version 2.0
+#  (the "License"); you may not use this file except in compliance with
+#  the License.  You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+#
+
+build_utils/list_dependencies.sh | python build_utils/verify_license.py ./dependencies_with_url.csv