Internal: third party notices generator

docs/readme.md

@@ -14,3 +14,4 @@
 
 # Repo management
 * [Make a new release](new-release.md)
+* [Third party dependencies](third-party-dependencies.md)

docs/third-party-dependencies.md

@@ -0,0 +1,27 @@
+# Third party dependencies
+
+Each dependencies shipped with batchlabs must go through Microsoft open source approval. This include every new version of the same dependency. That mean every time you update a dependency you must request [here](https://ossmsft.visualstudio.com/DefaultCollection/_oss?searchText=p%3A%22Batchlabs%22&_a=existing)
+
+
+## Which dependencies needs to be approved
+Any dependencies in the dependency list of the `package.json`. devDependencies are not required as they are not shipped.
+It also include non dev dependencies in `python/requirements.txt`
+
+There is also 2 more dependency which depends on what `travis` and `appveyor` build definition use(Don't ask for your local version). You must request approval accordingly
+* `node`
+* `python`
+
+## ThirdPartyNotices.txt
+
+Each of those depenencies needs to be referenced in the `ThirdPartyNotices.txt` with their own license file appened.
+Fortunately there is a tool that will generate this file for you.
+
+To generate the file run
+```
+npm run ts scripts/lca/generate-third-party
+```
+
+To check the current file is up to date(This is run on travis before mergin to stable)
+```
+npm run ts scripts/lca/generate-third-party -- --check
+```

package.json

@@ -113,6 +113,7 @@
     "angular2-template-loader": "~0.6.2",
     "awesome-typescript-loader": "^3.2.1",
     "codelyzer": "~3.1.1",
+    "commander": "~2.11.0",
     "concurrently": "^3.0.0",
     "copy-webpack-plugin": "^4.0.1",
     "cross-env": "~5.0.1",
@@ -164,6 +165,7 @@
     "@angular/platform-browser-dynamic": "4.3.5",
     "@angular/router": "4.3.5",
     "@angular/tsc-wrapped": "4.3.5",
+    "@types/node-fetch": "~1.6.7",
     "azure-batch": "~2.1.0-preview",
     "azure-storage": "^2.1.0",
     "bunyan": "^1.8.4",
@@ -183,11 +185,11 @@
     "moment": "^2.15.2",
     "moment-duration-format": "^1.3.0",
     "mousetrap": "^1.6.0",
+    "node-fetch": "~1.7.3",
     "reflect-metadata": "^0.1.9",
     "roboto-fontface": "^0.7.0",
     "rxjs": "^5.4.2",
     "strip-json-comments": "~2.0.1",
-    "stylint": "~1.5.9",
     "zone.js": "0.8.10"
   }
 }

scripts/lca/default-licenses/bsd-2-clause.txt

@@ -0,0 +1,8 @@
+Copyright (c) <year> <owner> All rights reserved.
+Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:
+
+1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
+
+2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

scripts/lca/default-licenses/ofl-1.1-and-mit.txt

@@ -0,0 +1,52 @@
+SIL OPEN FONT LICENSE
+Version 1.1 - 26 February 2007
+
+PREAMBLE
+
+The goals of the Open Font License (OFL) are to stimulate worldwide development of collaborative font projects, to support the font creation efforts of academic and linguistic communities, and to provide a free and open framework in which fonts may be shared and improved in partnership with others.
+
+The OFL allows the licensed fonts to be used, studied, modified and redistributed freely as long as they are not sold by themselves. The fonts, including any derivative works, can be bundled, embedded, redistributed and/or sold with any software provided that any reserved names are not used by derivative works. The fonts and derivatives, however, cannot be released under any other type of license. The requirement for fonts to remain under this license does not apply to any document created using the fonts or their derivatives.
+
+DEFINITIONS
+
+"Font Software" refers to the set of files released by the Copyright Holder(s) under this license and clearly marked as such. This may include source files, build scripts and documentation.
+
+"Reserved Font Name" refers to any names specified as such after the copyright statement(s).
+
+"Original Version" refers to the collection of Font Software components as distributed by the Copyright Holder(s).
+
+"Modified Version" refers to any derivative made by adding to, deleting, or substituting — in part or in whole — any of the components of the Original Version, by changing formats or by porting the Font Software to a new environment.
+
+"Author" refers to any designer, engineer, programmer, technical writer or other person who contributed to the Font Software.
+
+PERMISSION & CONDITIONS
+
+Permission is hereby granted, free of charge, to any person obtaining a copy of the Font Software, to use, study, copy, merge, embed, modify, redistribute, and sell modified and unmodified copies of the Font Software, subject to the following conditions:
+
+1) Neither the Font Software nor any of its individual components, in Original or Modified Versions, may be sold by itself.
+
+2) Original or Modified Versions of the Font Software may be bundled, redistributed and/or sold with any software, provided that each copy contains the above copyright notice and this license. These can be included either as stand-alone text files, human-readable headers or in the appropriate machine-readable metadata fields within text or binary files as long as those fields can be easily viewed by the user.
+
+3) No Modified Version of the Font Software may use the Reserved Font Name(s) unless explicit written permission is granted by the corresponding Copyright Holder. This restriction only applies to the primary font name as presented to the users.
+
+4) The name(s) of the Copyright Holder(s) or the Author(s) of the Font Software shall not be used to promote, endorse or advertise any Modified Version, except to acknowledge the contribution(s) of the Copyright Holder(s) and the Author(s) or with their explicit written permission.
+
+5) The Font Software, modified or unmodified, in part or in whole, must be distributed entirely under this license, and must not be distributed under any other license. The requirement for fonts to remain under this license does not apply to any document created using the Font Software.
+
+TERMINATION
+
+This license becomes null and void if any of the above conditions are not met.
+
+DISCLAIMER
+
+THE FONT SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT OF COPYRIGHT, PATENT, TRADEMARK, OR OTHER RIGHT. IN NO EVENT SHALL THE COPYRIGHT HOLDER BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, INCLUDING ANY GENERAL, SPECIAL, INDIRECT, INCIDENTAL, OR CONSEQUENTIAL DAMAGES, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF THE USE OR INABILITY TO USE THE FONT SOFTWARE OR FROM OTHER DEALINGS IN THE FONT SOFTWARE.
+
+
+MIT License
+Copyright (c) <year> <copyright holders>
+
+Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

scripts/lca/generate-third-party.ts

@@ -0,0 +1,200 @@
+import * as commander from "commander";
+import * as fs from "fs";
+import fetch from "node-fetch";
+import * as path from "path";
+import { Writable } from "stream";
+import * as util from "util";
+import { Constants } from "../../src/client/client-constants";
+
+export interface ThirdPartyNoticeOptions {
+    check?: boolean;
+}
+
+const defaultThirdPartyNoticeOptions: ThirdPartyNoticeOptions = {
+    check: false,
+};
+
+const thirdPartyNoticeFile = path.join(Constants.root, "ThirdPartyNotices.txt");
+
+const output = [];
+const gitUrlRegex = /(?:git|ssh|https?|git@[-\w.]+):(\/\/[-\w.]+\/)?(.*?)(\.git)(\/?|\#[-\d\w._]+?)$/;
+const repoNameRegex = /https?:\/\/github\.com\/(.*)/;
+const innerSeparator = "-".repeat(60);
+const outerSeparator = "=".repeat(60);
+
+const defaultLicenseRoot = path.join(Constants.root, "scripts/lca/default-licenses");
+
+// tslint:disable:no-console
+const defaultLicenses = {
+    "mit": fs.readFileSync(path.join(defaultLicenseRoot, "mit.txt")).toString(),
+    "bsd-2-clause": fs.readFileSync(path.join(defaultLicenseRoot, "bsd-2-clause.txt")).toString(),
+    "(ofl-1.1 and mit)": fs.readFileSync(path.join(defaultLicenseRoot, "ofl-1.1-and-mit.txt")).toString(),
+};
+
+const additionalDependencies = [
+    {
+        name: "websockets",
+        version: "3.3",
+        url: "https://github.com/aaugustin/websockets",
+        repoUrl: "https://github.com/aaugustin/websockets",
+        licenseType: "BSD-3-Clause",
+    },
+    {
+        name: "azure-batch-cli-extensions",
+        version: "0.2.0",
+        url: "https://github.com/Azure/azure-batch-cli-extensions",
+        repoUrl: "https://github.com/Azure/azure-batch-cli-extensions",
+        licenseType: "MIT",
+    },
+];
+
+function listDependencies(): string[] {
+
+    const packageJsonPath = path.join(Constants.root, "package.json");
+
+    const batchLabsPackage = JSON.parse(fs.readFileSync(packageJsonPath).toString());
+    const dependencies: string[] = Object.keys(batchLabsPackage.dependencies);
+
+    return dependencies.sort((a, b) => {
+        if (a < b) { return -1; }
+        if (a > b) { return 1; }
+        return 0;
+    });
+}
+
+function loadDependency(name: string) {
+    const contents = fs.readFileSync(`node_modules/${name}/package.json`).toString();
+    const dependency = JSON.parse(contents);
+
+    const repoUrl = getRepoUrl(dependency);
+    const url = dependency.homepage || repoUrl;
+    return {
+        name: dependency.name,
+        version: dependency.version,
+        url: url,
+        repoUrl: repoUrl,
+        licenseType: dependency.license,
+    };
+}
+
+function getRepoUrl(dependency) {
+    const repo = dependency.repository;
+    if (typeof repo === "string") {
+        return `https://github.com/${repo}`;
+    }
+    const match = gitUrlRegex.exec(repo.url);
+    if (!match) { return null; }
+    return `https://github.com/${match[2]}`;
+}
+
+function getRepoName(repoUrl: string): string {
+    const match = repoNameRegex.exec(repoUrl);
+    if (!match) {
+        console.error("Couldn't get repo name for ", repoUrl);
+    }
+    const value = match[1];
+    return value.split("/").slice(0, 2).join("/");
+}
+
+function loadLicense(repoUrl: string): Promise<any> {
+    const repoName = getRepoName(repoUrl);
+    return fetch(`https://api.github.com/repos/${repoName}/license`, {
+        headers: {
+            Authorization: `token ${process.env.GH_TOKEN}`,
+        },
+    }).then((res) => {
+        return res.json();
+    }).catch((error) => {
+        console.error(`Error loading license for ${repoName}`, error);
+    });
+}
+
+function decode64(content: string) {
+    return Buffer.from(content, "base64").toString();
+}
+
+function getHeader() {
+    return fs.readFileSync(path.join(Constants.root, "scripts/lca/header.txt")).toString();
+}
+
+function getLicenseContent(dependency, license) {
+    if (!license.content) {
+        const licenseType = dependency.licenseType.toLowerCase();
+        if (licenseType in defaultLicenses) {
+            return defaultLicenses[licenseType];
+        } else {
+            console.warn(`Repo ${dependency.name} doesn't have a license file`
+                + ` for ${licenseType} and no default provided`);
+            return null;
+        }
+    } else {
+        return decode64(license.content);
+    }
+}
+
+function checkNoticeUpToDate(notices: string) {
+    const existingNotices = fs.readFileSync(thirdPartyNoticeFile).toString();
+    if (existingNotices === notices) {
+        console.log("ThirdPartyNotice.txt is up to date.");
+        process.exit(0);
+    } else {
+        console.error("ThirdPartyNotice.txt is not up to date."
+            + " Please run 'npm run ts scripts/lca/generate-third-party'");
+        process.exit(1);
+    }
+}
+
+function run(options: ThirdPartyNoticeOptions = {}) {
+    options = { ...defaultThirdPartyNoticeOptions, ...options };
+    output.push(getHeader());
+    output.push("");
+
+    const depenencyNames = listDependencies();
+    const dependencies = depenencyNames.map((dep) => {
+        return loadDependency(dep);
+    }).concat(additionalDependencies);
+    console.log("Loading dependencies...");
+
+    let toc = dependencies.map((dependency, index) => {
+        return `${index}. ${dependency.name}(${dependency.url}) - ${dependency.licenseType}`;
+    });
+    output.push(toc.join("\n"));
+    output.push("");
+
+    const licensePromises = dependencies.map((dependency, index) => {
+        return loadLicense(dependency.repoUrl);
+    });
+    console.log("Loading licenses...");
+
+    Promise.all(licensePromises).then((licenses) => {
+        for (const [i, license] of licenses.entries()) {
+            const dependency = dependencies[i];
+            output.push(outerSeparator);
+            output.push(`    Start license for ${dependency.name}`);
+            output.push(innerSeparator);
+
+            const licenseContent = getLicenseContent(dependency, license);
+            if (!licenseContent) { continue; }
+
+            output.push(licenseContent);
+            output.push(innerSeparator);
+            output.push(`    End license for ${dependency.name}`);
+            output.push(outerSeparator);
+            output.push("");
+        }
+        const notices = output.join("\n");
+
+        if (options.check) {
+            checkNoticeUpToDate(notices);
+        } else {
+            fs.writeFileSync(thirdPartyNoticeFile, notices);
+            console.log(`Generated third party notice file at ${thirdPartyNoticeFile}`);
+        }
+    });
+}
+
+const options = commander
+    .option("-c, --check", "Check the current third party notice file is valid.")
+    .parse(process.argv);
+
+run(options);

scripts/lca/header.txt

@@ -0,0 +1,7 @@
+Do Not Translate or Localize
+
+This file is based on or incorporates material from the projects listed below (Third Party IP).
+The original copyright notice and the license under which Microsoft received such Third Party IP, are set forth below.
+Such licenses and notices are provided for informational purposes only.
+Microsoft licenses the Third Party IP to you under the licensing terms for the Microsoft product.
+Microsoft reserves all other rights not expressly granted under this agreement, whether by implication, estoppel or otherwise.

scripts/travis/build-and-test.sh

@@ -11,6 +11,7 @@ npm run build -s
 
 # Only run prod build if on a branch build or PR for stable
 if [ "${TRAVIS_PULL_REQUEST}" = "false" ] || [ "${TRAVIS_BRANCH}" = "stable" ]; then
+    npm run ts -s scripts/lca/generate-third-party -- --check
 	npm run build:prod
 fi