feat(applets): integrate into toolkit (#1039)

Integrate the Applet runner into the toolkit. When the --app argument
points to a .yml or .yaml file, the target will be assumed to be a
JavaScript applet, and run through the applet interpreter.

Enhancements to JS applet interpreter:

- Allow using non-Stack constructs.
- Multiple Stacks in one applet file by supplying an array.
- Allow referencing packages directly from NPM.

Non-Stack constructs can now be used as applets. If a non-Stack applet
is selected, a Stack will be constructed around it. This makes it
possible to reuse Applet constructs in regular CDK applications (if they
meet the requirements).

BREAKING CHANGE: The applet schema has changed to allow 
Multiple applets can be define in one file by structuring the files like this: 
BREAKING CHANGE: The applet schema has changed to allow definition
of multiple applets in the same file.

The schema now looks like this:

    applets:
      MyApplet:
        type: ./my-applet-file
        properties:
          property1: value
          ...
By starting an applet specifier with npm://, applet modules can
directly be referenced in NPM. You can include a version specifier
(@1.2.3) to reference specific versions.

Fixes #849, #342, #291.

Author: rix0rrr

docs/src/applets.rst

@@ -16,61 +16,71 @@ Applets
 
 .. note:: Currently the |cdk| only supports applets published as JavaScript modules.
 
-Applets are files in the YAML or JSON format that have the following root attribute,
-where MODULE can represent
-a local file, such as :code:`./my-module`,
-a local dependency, such as :code:`my-dependency`,
-or a global module, such as :code:`@aws-cdk/aws-s3`
-and CLASS is the name of a class exported by the module.
+Applets are files in the YAML format that instantiate constructs directly,
+without writing any code. The structure of an applet file looks like this:
 
 .. code:: js
 
-   applet: MODULE[:CLASS]
+   applets:
+     Applet1:
+       type: MODULE[:CLASS]
+       properties:
+         property1: value1
+         property2: value2
+         ...
+     Applet2:
+       type: MODULE[:CLASS]
+       properties:
+         ...
 
-If CLASS is not specified, :code:`Applet` is used as the default class name.
-Therefore, you need only refer to |cdk| construct libraries that export
-an :code:`Applet` class by their library name.
+Every applet will be synthesized to its own stack, named after the key used
+in the applet definition.
 
-The rest of the YAML file is applet-dependent.
-The object is passed as :code:`props` when the applet object is instantiated
-and added to an |cdk| app created by **cdk-applet-js**.
+Specifying the applet to load
+=============================
 
-Use **cdk-applet-js** *applet* to run the applet, create an |cdk| app,
-and use that with the |cdk| tools, as shown in the following example.
+An applet ``type`` specification looks like this:
 
-.. code-block:: sh
+.. code:: js
 
-   cdk --app "cdk-applet-js ./my-applet.yaml" synth
+   applet: MODULE[:CLASS]
 
-To make the applet file executable and use the host as a shebang
-on Unix-based systems, such as Linux, MacOS, or Windows Bash shell,
-create a script similar to the following.
+**MODULE** can be used to indicate:
 
-.. code-block:: sh
+* A local file, such as ``./my-module`` (expects ``my-module.js`` in the same
+  directory).
+* A local module such as ``my-dependency`` (expects an NPM package at
+  ``node_modules/my-dependency``).
+* A global module, such as ``@aws-cdk/aws-s3`` (expects the package to have been
+  globally installed using NPM).
+* An NPM package, such as ``npm://some-package@1.2.3`` (the version specifier
+  may be omitted to refer to the latest version of the package).
+
+**CLASS** should reference the name of a class exported by the indicated module.
+If the class name is omitted, ``Applet`` is used as the default class name.
+
+Properties
+==========
 
-   #!/usr/bin/env cdk-applet-js
+Pass properties to the applet by specifying them in the ``properties`` object.
+The properties will be passed to the instantiation of the class in the ``type``
+parameter.
 
-   applet: aws-cdk-codebuild
-   source: arn:aws:codecommit:::my-repository
-   image: node:8.9.4
-   compute: large
-   build:
-      - npm install --unsafe-perm
-      - npm test
-      - npm pack --unsafe-perm
+Running
+=======
 
-To execute the applet and synthesize an |CFN| template,
-use the following command.
+To run an applet, pass its YAML file directly as the ``--app`` argument to a
+``cdk`` invocation:
 
 .. code-block:: sh
 
-   cdk synth --app "./build.yaml"
+   cdk --app ./my-applet.yaml deploy
 
-To avoid needing **--app** for every invocation,
-add the following entry to *cdk.json*.
+To avoid needing to specify ``--app`` for every invocation, make a ``cdk.json``
+file and add in the application in the config as usual:
 
 .. code-block:: json
 
    {
-      "app": "./build.yaml"
+      "app": "./my-applet.yaml"
    }

packages/@aws-cdk/applet-js/bin/cdk-applet-js.ts

@@ -2,8 +2,11 @@
 import 'source-map-support/register';
 
 import cdk = require('@aws-cdk/cdk');
+import child_process = require('child_process');
 import fs = require('fs-extra');
+import os = require('os');
 import path = require('path');
+import { isStackConstructor, parseApplet } from '../lib/applet-helpers';
 
 // tslint:disable-next-line:no-var-requires
 const YAML = require('js-yaml');
@@ -19,65 +22,84 @@ async function main() {
 
   const appletFile = process.argv[2];
   if (!appletFile) {
-    throw new Error(`Usage: ${progname}| <applet.yaml>`);
+    throw new Error(`Usage: ${progname} <applet.yaml>`);
   }
 
-  // read applet properties from the provided file
-  const props = YAML.safeLoad(await fs.readFile(appletFile, { encoding: 'utf-8' }));
+  // read applet(s) properties from the provided file
+  const fileContents = YAML.safeLoad(await fs.readFile(appletFile, { encoding: 'utf-8' }));
+  if (typeof fileContents !== 'object') {
+    throw new Error(`${appletFile}: should contain a YAML object`);
+  }
+  const appletMap = fileContents.applets;
+  if (!appletMap) {
+    throw new Error(`${appletFile}: must have an 'applets' key`);
+  }
+
+  const searchDir = path.dirname(appletFile);
+  const tempDir = await fs.mkdtemp(path.join(os.tmpdir(), 'cdkapplet'));
+  try {
+    const app = new cdk.App();
+
+    for (const [name, definition] of Object.entries(appletMap)) {
+      await constructStack(app, searchDir, tempDir, name, definition);
+    }
+    app.run();
+  } finally {
+    await fs.remove(tempDir);
+  }
+}
 
+/**
+ * Construct a stack from the given props
+ * @param props Const
+ */
+async function constructStack(app: cdk.App, searchDir: string, tempDir: string, stackName: string, spec: any) {
   // the 'applet' attribute tells us how to load the applet. in the javascript case
   // it will be in the format <module>:<class> where <module> is technically passed to "require"
   // and <class> is expected to be exported from the module.
-  const applet: string = props.applet;
-  if (!applet) {
-    throw new Error('Applet file missing "applet" attribute');
+  const appletSpec: string | undefined = spec.type;
+  if (!appletSpec) {
+    throw new Error(`Applet ${stackName} missing "type" attribute`);
   }
 
-  const { moduleName, className } = parseApplet(applet);
-
-  // remove the 'applet' attribute as we pass it along to the applet class.
-  delete props.applet;
+  const applet = parseApplet(appletSpec);
+
+  const props = spec.properties || {};
+
+  if (applet.npmPackage) {
+    // tslint:disable-next-line:no-console
+    console.error(`Installing NPM package ${applet.npmPackage}`);
+    // Magic marker to download this package directly off of NPM
+    // We're going to run NPM as a shell (since programmatic usage is not stable
+    // by their own admission) and we're installing into a temporary directory.
+    // (Installing into a permanent directory is useless since NPM doesn't do
+    // any real caching anyway).
+    child_process.execFileSync('npm', ['install', '--prefix', tempDir, '--global', applet.npmPackage], {
+      stdio: 'inherit'
+    });
+    searchDir = path.join(tempDir, 'lib');
+  }
 
   // we need to resolve the module name relatively to where the applet file is
   // and not relative to this module or cwd.
-  const resolve = require.resolve as any; // escapse type-checking since { paths } is not defined
-  const modulePath = resolve(moduleName, { paths: [ path.dirname(appletFile) ] });
+  const modulePath = require.resolve(applet.moduleName, { paths: [ searchDir ] });
 
   // load the module
   const pkg = require(modulePath);
 
   // find the applet class within the package
   // tslint:disable-next-line:variable-name
-  const AppletStack = pkg[className];
-  if (!AppletStack) {
-    throw new Error(`Cannot find applet class "${className}" in module "${moduleName}"`);
-  }
-
-  // create the CDK app
-  const app = new cdk.App();
-
-  const constructName = props.name || className;
-
-  // add the applet stack into the app.
-  new AppletStack(app, constructName, props);
-
-  // transfer control to the app
-  app.run();
-}
-
-function parseApplet(applet: string) {
-  const components = applet.split(':');
-  // tslint:disable-next-line:prefer-const
-  let [ moduleName, className ] = components;
-
-  if (components.length > 2 || !moduleName) {
-    throw new Error(`"applet" value is "${applet}" but it must be in the form "<js-module>[:<applet-class>]".
-      If <applet-class> is not specified, "Applet" is the default`);
+  const appletConstructor = pkg[applet.className];
+  if (!appletConstructor) {
+    throw new Error(`Cannot find applet class "${applet.className}" in module "${applet.moduleName}"`);
   }
 
-  if (!className) {
-    className = 'Applet';
+  if (isStackConstructor(appletConstructor)) {
+    // add the applet stack into the app.
+    new appletConstructor(app, stackName, props);
+  } else {
+    // Make a stack THEN add it in
+    const stack = new cdk.Stack(app, stackName, props);
+    new appletConstructor(stack, 'Default', props);
   }
-
-  return { moduleName, className };
 }

packages/@aws-cdk/applet-js/lib/applet-helpers.ts

@@ -0,0 +1,52 @@
+/**
+ * Determine whether this constructorFunction is going to create an object that inherits from Stack
+ *
+ * We do structural typing.
+ */
+export function isStackConstructor(constructorFn: any) {
+  // Test for a public method that Stack has
+  return constructorFn.prototype.findResource !== undefined;
+}
+
+/**
+ * Extract module name from a NPM package specification
+ */
+export function extractModuleName(packageSpec: string) {
+    const m = /^((?:@[a-zA-Z-]+\/)?[a-zA-Z-]+)/i.exec(packageSpec);
+    if (!m) { throw new Error(`Could not find package name in ${packageSpec}`); }
+    return m[1];
+}
+
+export function parseApplet(applet: string): AppletSpec {
+  const m = /^(npm:\/\/)?([a-z0-9_@./-]+)(:[a-z_0-9]+)?$/i.exec(applet);
+  if (!m) {
+    throw new Error(`"applet" value is "${applet}" but it must be in the form "[npm://]<js-module>[:<applet-class>]".
+      If <applet-class> is not specified, "Applet" is the default`);
+  }
+
+  if (m[1] === 'npm://') {
+    return {
+      npmPackage: m[2],
+      moduleName: extractModuleName(m[2]),
+      className: className(m[3]),
+    };
+  } else {
+    return {
+      moduleName: m[2],
+      className: className(m[3]),
+    };
+  }
+
+  function className(s: string | undefined) {
+    if (s) {
+      return s.substr(1);
+    }
+    return 'Applet';
+  }
+}
+
+export interface AppletSpec {
+  npmPackage?: string;
+  moduleName: string;
+  className: string;
+}