Merge from awslabs/aws-sam-cli/develop

appveyor-windows.yml

@@ -40,6 +40,9 @@ cache:
 
 install:
 
+    # setup make
+    - "choco install make"
+
     # Make sure the temp directory exists for Python to use.
     - ps: "mkdir -Force D:\\tmp"
     - "SET PATH=%PYTHON_HOME%;%PATH%"

designs/build_for_provided_runtimes.md

@@ -0,0 +1,139 @@
+What is the problem?
+--------------------
+* sam build does not support building for `provided` runtimes.
+* sam build also does not allow customization of the build process for the supported lambda runtimes.
+
+What will be changed?
+---------------------
+
+Serverless Function resources can now have a Metadata Resource Attribute which specifies a `BuildMethod`. 
+`BuildMethod` will either be the official lambda runtime identifiers such as `python3.8`, `nodejs12.x` etc or `makefile`.
+If `BuildMethod` is specified to be `makefile`, the build targets that are present in the `Makefile` which take the form of
+
+`build-{resource_logical_id}` will be executed.
+
+More details can also be found at: [CustomMakeBuildWorkflow](https://github.com/awslabs/aws-lambda-builders/blob/develop/aws_lambda_builders/workflows/custom_make/DESIGN.md)
+
+This enables following usecases:
+
+* build for `provided` runtimes.
+* user specified build steps for official lambda supported runtimes instead of what `sam build` natively offers.
+
+Success criteria for the change
+-------------------------------
+
+* Users are able to build for `provided` runtimes through sam build directly.
+* Users are able to bring their own build steps for even natively supported lambda runtimes.
+
+User Experience Walkthrough
+---------------------------
+
+#### Provided runtimes
+
+Template
+
+```yaml
+Resources:
+  HelloRustFunction:
+    Type: AWS::Serverless::Function
+    Properties:
+      Handler: bootstrap.is.real.handler
+      Runtime: provided
+      MemorySize: 512
+      CodeUri: .
+    Metadata:
+      BuildMethod: makefile
+```
+
+Makefile
+
+```
+build-HelloRustFunction:
+	cargo build --release --target x86_64-unknown-linux-musl
+	cp ./target/x86_64-unknown-linux-musl/release/bootstrap $(ARTIFACTS_DIR)
+```
+
+#### Makefile builder for lambda runtimes
+
+Template
+
+```yaml
+Resources:
+  HelloWorldFunction:
+    Type: AWS::Serverless::Function
+    Properties:
+      CodeUri: hello_world/
+      Handler: app.lambda_handler
+      Runtime: python3.7
+    Metadata:
+      BuildMethod: makefile
+```
+
+Makefile
+
+```
+build-HelloWorldFunction:
+    cp *.py $(ARTIFACTS_DIR)
+    cp requirements.txt $(ARTIFACTS_DIR)
+    python -m pip install -r requirements.txt -t $(ARTIFACTS_DIR)
+    rm -rf $(ARTIFACTS_DIR)/bin
+```
+
+Implementation
+==============
+### Proposal
+
+* Selection of the build workflow within sam cli will now have additional logic to select the correct build workflow from user input in the template.
+Currently, a build workflow is chosen based on the lambda runtime and the manifest file alone.
+
+FAQs
+----
+1. Can a user specify the BuildMethod to be `python3.8` for a `python3.8` runtime?
+    * Yes, this will just select the native python workflow that is used from `aws-lambda-builders` 
+2. Can a user specify BuildMethod to be `ruby2.7` for a `python3.8` runtime?
+    * Theoretically yes, But the build will just fail.
+3. Can a user specify the BuildMethod to be `python3.7` for a `python3.8` runtime?
+    * Theoretically yes, However If the user is just using the builder that samcli already provides, 
+    its best not to provide any `BuildMethod` at all.
+
+
+CLI Changes
+-----------
+
+No changes in CLI interface.
+
+### Breaking Change
+
+No breaking changes.
+
+
+What is your Testing Plan (QA)?
+===============================
+
+* Unit and Integration testing
+
+Goal
+----
+
+* Coverage of usecases such as:
+    * build for `provided` runtimes
+    * build for official lambda runtimes through the `makefile` construct
+    * build within containers for the `makefile` construct
+
+
+Expected Results
+----------------
+* Integration tests to pass that covers usecases listed in the goal.
+
+
+Task Breakdown
+==============
+
+-   \[x\] Send a Pull Request with this design document
+-   \[ \] Build the command line interface
+-   \[ \] Build the underlying library
+-   \[ \] Unit tests
+-   \[ \] Functional Tests
+-   \[ \] Integration tests
+-   \[ \] Run all tests on Windows
+-   \[ \] Update documentation

samcli/__init__.py

@@ -2,4 +2,4 @@
 SAM CLI version
 """
 
-__version__ = "0.50.0"
+__version__ = "0.51.0"

samcli/lib/build/app_builder.py

@@ -113,15 +113,17 @@ def build(self):
             result[function.name] = self._build_function(function.name,
                                                          function.codeuri,
                                                          function.runtime,
-                                                         function.handler)
+                                                         function.handler,
+                                                         function.metadata)
         for layer in self._resources_to_build.layers:
             LOG.info("Building layer '%s'", layer.name)
             if layer.build_method is None:
                 raise MissingBuildMethodException(
                     f"Layer {layer.name} cannot be build without BuildMethod. Please provide BuildMethod in Metadata.")
             result[layer.name] = self._build_layer(layer.name,
                                                    layer.codeuri,
-                                                   layer.build_method)
+                                                   layer.build_method,
+                                                   layer.compatible_runtimes)
 
         return result
 
@@ -171,13 +173,13 @@ def update_template(self, template_dict, original_template_path, built_artifacts
 
         return template_dict
 
-    def _build_layer(self, layer_name, codeuri, runtime):
+    def _build_layer(self, layer_name, codeuri, specified_workflow, compatible_runtimes):
         # Create the arguments to pass to the builder
         # Code is always relative to the given base directory.
         code_dir = str(pathlib.Path(self._base_dir, codeuri).resolve())
 
-        config = get_workflow_config(runtime, code_dir, self._base_dir)
-        subfolder = get_layer_subfolder(runtime)
+        config = get_workflow_config(None, code_dir, self._base_dir, specified_workflow)
+        subfolder = get_layer_subfolder(specified_workflow)
 
         # artifacts directory will be created by the builder
         artifacts_dir = str(pathlib.Path(self._build_dir, layer_name, subfolder))
@@ -186,21 +188,28 @@ def _build_layer(self, layer_name, codeuri, runtime):
             manifest_path = self._manifest_path_override or os.path.join(code_dir, config.manifest_name)
 
             # By default prefer to build in-process for speed
+            build_runtime = specified_workflow
             build_method = self._build_function_in_process
             if self._container_manager:
-                build_method = self._build_function_on_container
+                build_method = self._build_function_in_process
+                if config.language == "provided":
+                    LOG.warning(
+                        "For container layer build, first compatible runtime is chosen as build target for container.")
+                    # Only set to this value if specified workflow is makefile which will result in config language as provided
+                    build_runtime = compatible_runtimes[0]
+            options = ApplicationBuilder._get_build_options(layer_name, config.language, None)
 
             build_method(config,
                          code_dir,
                          artifacts_dir,
                          scratch_dir,
                          manifest_path,
-                         runtime,
-                         None)
+                         build_runtime,
+                         options)
             # Not including subfolder in return so that we copy subfolder, instead of copying artifacts inside it.
             return str(pathlib.Path(self._build_dir, layer_name))
 
-    def _build_function(self, function_name, codeuri, runtime, handler):
+    def _build_function(self, function_name, codeuri, runtime, handler, metadata=None):
         """
         Given the function information, this method will build the Lambda function. Depending on the configuration
         it will either build the function in process or by spinning up a Docker container.
@@ -216,6 +225,9 @@ def _build_function(self, function_name, codeuri, runtime, handler):
         runtime : str
             AWS Lambda function runtime
 
+        metadata : dict
+            AWS Lambda function metadata
+
         Returns
         -------
         str
@@ -232,7 +244,10 @@ def _build_function(self, function_name, codeuri, runtime, handler):
         # Code is always relative to the given base directory.
         code_dir = str(pathlib.Path(self._base_dir, codeuri).resolve())
 
-        config = get_workflow_config(runtime, code_dir, self._base_dir)
+        # Determine if there was a build workflow that was specified directly in the template.
+        specified_build_workflow = metadata.get("BuildMethod", None) if metadata else None
+
+        config = get_workflow_config(runtime, code_dir, self._base_dir, specified_workflow=specified_build_workflow)
 
         # artifacts directory will be created by the builder
         artifacts_dir = str(pathlib.Path(self._build_dir, function_name))
@@ -245,7 +260,7 @@ def _build_function(self, function_name, codeuri, runtime, handler):
             if self._container_manager:
                 build_method = self._build_function_on_container
 
-            options = ApplicationBuilder._get_build_options(config.language, handler)
+            options = ApplicationBuilder._get_build_options(function_name, config.language, handler)
 
             return build_method(config,
                                 code_dir,
@@ -256,20 +271,27 @@ def _build_function(self, function_name, codeuri, runtime, handler):
                                 options)
 
     @staticmethod
-    def _get_build_options(language, handler):
+    def _get_build_options(function_name, language, handler):
         """
         Parameters
         ----------
+        function_name str
+            currrent function resource name
         language str
-            Language of the runtime
+            language of the runtime
         handler str
             Handler value of the Lambda Function Resource
         Returns
         -------
         dict
             Dictionary that represents the options to pass to the builder workflow or None if options are not needed
         """
-        return {'artifact_executable_name': handler} if language == 'go' else None
+
+        _build_options = {
+            'go': {'artifact_executable_name': handler},
+            'provided': {'build_logical_id': function_name}
+        }
+        return _build_options.get(language, None)
 
     def _build_function_in_process(self,
                                    config,