Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Ruby to Supported Runtime Enum #866

Merged
merged 9 commits into from
Dec 21, 2018
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions requirements/base.txt
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,6 @@ docker>=3.3.0
dateparser~=0.7
python-dateutil~=2.6
pathlib2~=2.3.2; python_version<"3.4"
requests~=2.20.0
aws_lambda_builders==0.0.4
requests==2.20.1
aws_lambda_builders==0.0.5
serverlessrepo==0.1.5
1 change: 0 additions & 1 deletion requirements/dev.txt
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,6 @@ pylint==1.7.2
pytest==3.0.7
py==1.4.33
mock==2.0.0
requests==2.20.0
parameterized==0.6.1
pathlib2==2.3.2; python_version<"3.4"
futures==3.2.0; python_version<"3.2.3"
Expand Down
6 changes: 6 additions & 0 deletions samcli/lib/build/app_builder.py
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,12 @@ def _get_workflow_config(runtime):
dependency_manager="npm",
application_framework=None,
manifest_name="package.json")
elif runtime.startswith("ruby"):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Woot! sam build for ruby! :)

return config(
language="ruby",
dependency_manager="bundler",
application_framework=None,
manifest_name="Gemfile")
else:
raise UnsupportedRuntimeException("'{}' runtime is not supported".format(runtime))

Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Cookiecutter SAM for Ruby Lambda functions

This is a [Cookiecutter](https://github.com/audreyr/cookiecutter) template to create a Serverless Hello World App based on Serverless Application Model (SAM) and Python.
This is a [Cookiecutter](https://github.com/audreyr/cookiecutter) template to create a Serverless Hello World App based on Serverless Application Model (SAM) and Ruby.

## Recommendations

Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,8 @@
source "https://rubygems.org"

gem "httparty"

group :test do
gem "test-unit"
gem "mocha"
end
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,8 @@ This is a sample template for {{ cookiecutter.project_name }} - Below is a brief
├── README.md <-- This instructions file
├── hello_world <-- Source code for a lambda function
│ ├── app.rb <-- Lambda function code
├── Gemfile <-- Ruby dependencies
│ ├── Gemfile <-- Ruby function dependencies
├── Gemfile <-- Ruby test/documentation dependencies
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not a huge fan of the Double Gemfile. How do I know or remember which one to update? Is this absolutely needed? For Node, we had tests under the function directory and excluded them through .npmignore

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So...I'm open to ways we can support this through sam build. If we had to pick one, we would not include the global test/docs Gemfile, but then the way the example code does testing wouldn't work easily. I don't know how to get SAM CLI to use a Gemfile in the root directory to build vendored dependencies into each function directory in a single command.

It's actually very possible that multiple Gemfiles may be a reality, and that we just need to teach the paradigm. Why?

  1. Layers will eventually be a part of this build story, and you only want to specify those dependencies within the layer itself. In fact, we may want builds where we say "don't fetch all dependencies I need, expect some to be resolved in a layer" down the road.
  2. I may not want all of my dependencies in every function.
  3. I may want global dependencies in my build structure.

There is actually precedent for this, the Ruby SDK repo itself: https://github.com/aws/aws-sdk-ruby/

We have a global Gemfile for testing, building, and documentation generation. Within each modular package, we have its own gemspec for only the required dependencies. This may be the correct pattern for modular Lambda functions.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can image an appropriate alternative approach, perhaps where we use "groups" per function to keep dependency versions in sync in a global Gemfile. In an ideal world, we could provide options to the user.

├── template.yaml <-- SAM template
└── tests <-- Unit tests
└── unit
Expand All @@ -31,27 +32,27 @@ Setup Ruby Version Manager from [Ruby Version Manager](http://rvm.io/)
Run following commands

```bash
rvm install ruby-2.5.0
rvm use ruby-2.5.0
rvm --default use 2.5.0
rvm install ruby-2.5.3
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does patch version need to be specified here?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's a best practice, yes. Ruby on AWS Lambda uses version 2.5.3 - while 2.5.0 will technically work for builds, there are security patches present on 2.5.3 that are not present on 2.5.0, so I would not intentionally direct users to use an older version in this case.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a way the newest just gets installed. I am worried this will be out of date and something we need to maintain.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do we need to provide instructions on how to use install Ruby for Ruby developers? I wonder if Requirements (ruby 2.5) suffices as we do with Python and Node init

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is how you install Ruby. Not sure what the question is @heitorlessa

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure, let me try once more and let me know if it's clearer:

  1. Do we need to tell Ruby developers how they should install Ruby?
  2. Customers trying Ruby quickstart should have Ruby installed, shouldn't we assume that and cut the Installation part?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wouldn't add install instructions. People who use languages like ruby, go etc will have their tooling in place.

It would be like every pkg internally saying start off by installing the Amazon builder tools.

rvm use ruby-2.5.3
rvm --default use 2.5.3
```

### Building the Project

### Installing dependencies
```sam-app``` comes with a Gemfile that defines the requirements and manages installing them. The `sam build` command will install the dependencies in your function Gemfile and vendor it for deployment.

```sam-app``` comes with a Gemfile that defines the requirements and manages installing them.

```bash
gem install bundler
bundle install
bundle install --deployment --path hello_world/vendor/bundle
```
sam build
```

* Step 1 installs ```bundler```which provides a consistent environment for Ruby projects by tracking and installing the exact gems and versions that are needed.
* Step 2 creates a Gemfile.lock that locks down the versions and creates the full dependency closure.
* Step 3 installs the gems to ```hello_world/vendor/bundle```.
If your dependencies contain native modules that need to be compiled specifically for the operating system running on AWS Lambda, use this command to build inside a Lambda-like Docker container instead:

**NOTE:** As you change your dependencies during development you'll need to make sure these steps are repeated in order to execute your Lambda and/or API Gateway locally.
```
sam build --use-container
```
By default, this command writes built artifacts to .aws-sam/build folder.

**NOTE:** As you change your dependencies during development you'll need to run `sam build` again in order to execute your Lambda and/or API Gateway locally.

### Local development

Expand Down Expand Up @@ -124,20 +125,18 @@ aws cloudformation describe-stacks \

## Testing

We use [Mocha](http://gofreerange.com/mocha/docs) for testing our code and you can install it using gem: ``gem install mocha``

Next, we run our initial unit tests:
Run our initial unit tests:

```bash
ruby tests/unit/test_hello.rb
ruby tests/unit/test_handler.rb
```

**NOTE**: It is recommended to use a Ruby Version Manager to manage, and work with multiple ruby environments from interpreters to sets of gems
# Appendix

## AWS CLI commands

AWS CLI commands to package, deploy and describe outputs defined within the cloudformation stack:
AWS CLI commands to package, deploy and describe outputs defined within the cloudformation stack after building:

```bash
sam package \
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
source "https://rubygems.org"

gem "httparty"
Original file line number Diff line number Diff line change
Expand Up @@ -73,11 +73,11 @@ def lambda_handler(event:, context:)
raise error
end

return {
:statusCode => response.code,
:body => {
:message => "Hello World!",
:location => response.body
{
statusCode: response.code,
body: {
message: "Hello World!",
location: response.body
}.to_json
}
end
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ Description: >
{{ cookiecutter.project_name }}

Sample SAM Template for {{ cookiecutter.project_name }}

# More info about Globals: https://github.com/awslabs/serverless-application-model/blob/master/docs/globals.rst
Globals:
Function:
Expand Down
Original file line number Diff line number Diff line change
@@ -1,12 +1,12 @@
require 'json'
require 'test/unit'
require 'mocha/test_unit'

require_relative '../../hello_world/app'

class HelloWorldTest < Test::Unit::TestCase

def setup
@event = {
def event
{
body: 'eyJ0ZXN0IjoiYm9keSJ9',
resource: '/{proxy+}',
path: '/path/to/resource',
Expand All @@ -22,24 +22,24 @@ def setup
baz: 'qux'
},
headers: {
Accept: 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8',
Accept-Encoding: 'gzip, deflate, sdch',
Accept-Language: 'en-US,en;q=0.8',
Cache-Control: 'max-age=0',
CloudFront-Forwarded-Proto: 'https',
CloudFront-Is-Desktop-Viewer: 'true',
CloudFront-Is-Mobile-Viewer: 'false',
CloudFront-Is-SmartTV-Viewer: 'false',
CloudFront-Is-Tablet-Viewer: 'false',
CloudFront-Viewer-Country: 'US',
Host: '1234567890.execute-api.us-east-1.amazonaws.com',
Upgrade-Insecure-Requests: '1',
User-Agent: 'Custom User Agent String',
Via: '1.1 08f323deadbeefa7af34d5feb414ce27.cloudfront.net (CloudFront)',
X-Amz-Cf-Id: 'cDehVQoZnx43VYQb9j2-nvCh-9z396Uhbp027Y2JvkCPNLmGJHqlaA==',
X-Forwarded-For: '127.0.0.1, 127.0.0.2',
X-Forwarded-Port: '443',
X-Forwarded-Proto: 'https'
'Accept' => 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8',
'Accept-Encoding' => 'gzip, deflate, sdch',
'Accept-Language' => 'en-US,en;q=0.8',
'Cache-Control' => 'max-age=0',
'CloudFront-Forwarded-Proto' => 'https',
'CloudFront-Is-Desktop-Viewer' => 'true',
'CloudFront-Is-Mobile-Viewer' => 'false',
'CloudFront-Is-SmartTV-Viewer' => 'false',
'CloudFront-Is-Tablet-Viewer' => 'false',
'CloudFront-Viewer-Country' => 'US',
'Host' => '1234567890.execute-api.us-east-1.amazonaws.com',
'Upgrade-Insecure-Requests' => '1',
'User-Agent' => 'Custom User Agent String',
'Via' => '1.1 08f323deadbeefa7af34d5feb414ce27.cloudfront.net (CloudFront)',
'X-Amz-Cf-Id' => 'cDehVQoZnx43VYQb9j2-nvCh-9z396Uhbp027Y2JvkCPNLmGJHqlaA==',
'X-Forwarded-For' => '127.0.0.1, 127.0.0.2',
'X-Forwarded-Port' => '443',
'X-Forwarded-Proto' => 'https'
},
requestContext: {
accountId: '123456789012',
Expand All @@ -49,17 +49,17 @@ def setup
requestTime: '09/Apr/2015:12:34:56 +0000',
requestTimeEpoch: 1428582896000,
identity: {
cognitoIdentityPoolId: "null",
accountId: "null",
cognitoIdentityId: "null",
caller: "null",
accessKey: "null",
cognitoIdentityPoolId: 'null',
accountId: 'null',
cognitoIdentityId: 'null',
caller: 'null',
accessKey: 'null',
sourceIp: '127.0.0.1',
cognitoAuthenticationType: "null",
cognitoAuthenticationProvider: "null",
userArn: "null",
cognitoAuthenticationType: 'null',
cognitoAuthenticationProvider: 'null',
userArn: 'null',
userAgent: 'Custom User Agent String',
user: "null"
user: 'null'
},
path: '/prod/path/to/resource',
resourcePath: '/{proxy+}',
Expand All @@ -68,24 +68,27 @@ def setup
protocol: 'HTTP/1.1'
}
}
end

@mock_response = {
:statusCode => 200,
:body => {
message: "Hello World!",
location: "1.1.1.1"
def mock_response
Object.new.tap do |mock|
mock.expects(:code).returns(200)
mock.expects(:body).returns('1.1.1.1')
end
end

def expected_result
{
statusCode: 200,
body: {
message: 'Hello World!',
location: '1.1.1.1'
}.to_json
}

end

def test_lambda_handler
expects(:lambda_handler).with(event:@event, context:"").returns(@mock_response)
response = lambda_handler(event:@event, context:"")
json_body = JSON.parse(response[:body])

assert_equal(200, response[:statusCode])
assert_equal("Hello World!", json_body["message"])
assert_equal("1.1.1.1", json_body["location"])
HTTParty.expects(:get).with('http://checkip.amazonaws.com/').returns(mock_response)
assert_equal(lambda_handler(event: event, context: ''), expected_result)
end
end
2 changes: 1 addition & 1 deletion tests/functional/local/apigw/test_local_apigw_service.py
Original file line number Diff line number Diff line change
Expand Up @@ -630,7 +630,7 @@ def make_service(list_of_routes, function_provider, cwd):

def make_service_response(port, method, scheme, resourcePath, resolvedResourcePath, pathParameters=None,
body=None, headers=None, queryParams=None, isBase64Encoded=False):
response_str = '{"httpMethod": "GET", "body": null, "resource": "/something/{event}", "requestContext": {"resourceId": "123456", "apiId": "1234567890", "resourcePath": "/something/{event}", "httpMethod": "GET", "requestId": "c6af9ac6-7b61-11e6-9a41-93e8deadbeef", "accountId": "123456789012", "stage": "prod", "identity": {"apiKey": null, "userArn": null, "cognitoAuthenticationType": null, "caller": null, "userAgent": "Custom User Agent String", "user": null, "cognitoIdentityPoolId": null, "cognitoAuthenticationProvider": null, "sourceIp": "127.0.0.1", "accountId": null}, "extendedRequestId": null, "path": "/something/{event}"}, "queryStringParameters": null, "headers": {"Host": "0.0.0.0:33651", "User-Agent": "python-requests/2.20.0", "Accept-Encoding": "gzip, deflate", "Accept": "*/*", "Connection": "keep-alive"}, "pathParameters": {"event": "event1"}, "stageVariables": null, "path": "/something/event1", "isBase64Encoded": false}' # NOQA
response_str = '{"httpMethod": "GET", "body": null, "resource": "/something/{event}", "requestContext": {"resourceId": "123456", "apiId": "1234567890", "resourcePath": "/something/{event}", "httpMethod": "GET", "requestId": "c6af9ac6-7b61-11e6-9a41-93e8deadbeef", "accountId": "123456789012", "stage": "prod", "identity": {"apiKey": null, "userArn": null, "cognitoAuthenticationType": null, "caller": null, "userAgent": "Custom User Agent String", "user": null, "cognitoIdentityPoolId": null, "cognitoAuthenticationProvider": null, "sourceIp": "127.0.0.1", "accountId": null}, "extendedRequestId": null, "path": "/something/{event}"}, "queryStringParameters": null, "headers": {"Host": "0.0.0.0:33651", "User-Agent": "python-requests/2.20.1", "Accept-Encoding": "gzip, deflate", "Accept": "*/*", "Connection": "keep-alive"}, "pathParameters": {"event": "event1"}, "stageVariables": null, "path": "/something/event1", "isBase64Encoded": false}' # NOQA
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

grrr.. let's just pin requests. We have installers now, so pinning isn't as dangerous. If customers run into conflicts, this gives them a reason to get onto our installers and out of pip (which is what we want/recommend anyways)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could remove requests from test requirements (dev.txt) , and pin the requests in the base.txt.

But if we pin, we would periodically need to bump up requests as when there is a release of requests.

response = json.loads(response_str)

if body:
Expand Down
76 changes: 76 additions & 0 deletions tests/integration/buildcmd/test_build_cmd.py
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,10 @@
import json
import logging

try:
from pathlib import Path
except ImportError:
from pathlib2 import Path
from parameterized import parameterized

from samcli.yamlhelper import yaml_parse
Expand Down Expand Up @@ -192,3 +196,75 @@ def _verify_resource_property(self, template_path, logical_id, property, expecte
with open(template_path, 'r') as fp:
template_dict = yaml_parse(fp.read())
self.assertEquals(expected_value, template_dict["Resources"][logical_id]["Properties"][property])


class TestBuildCommand_RubyFunctions(BuildIntegBase):

EXPECTED_FILES_GLOBAL_MANIFEST = set()
EXPECTED_FILES_PROJECT_MANIFEST = {'app.rb'}
EXPECTED_RUBY_GEM = 'httparty'

FUNCTION_LOGICAL_ID = "Function"

@parameterized.expand([
("ruby2.5", False),
("ruby2.5", "use_container")
])
def test_with_default_gemfile(self, runtime, use_container):
overrides = {"Runtime": runtime, "CodeUri": "Ruby"}
cmdlist = self.get_command_list(use_container=use_container,
parameter_overrides=overrides)

LOG.info("Running Command: {}".format(cmdlist))
process = subprocess.Popen(cmdlist, cwd=self.working_dir)
process.wait()

self._verify_built_artifact(self.default_build_dir, self.FUNCTION_LOGICAL_ID,
self.EXPECTED_FILES_PROJECT_MANIFEST, self.EXPECTED_RUBY_GEM)

self._verify_resource_property(str(self.built_template),
"OtherRelativePathResource",
"BodyS3Location",
os.path.relpath(
os.path.normpath(os.path.join(str(self.test_data_path), "SomeRelativePath")),
str(self.default_build_dir))
)

def _verify_built_artifact(self, build_dir, function_logical_id, expected_files, expected_modules):

self.assertTrue(build_dir.exists(), "Build directory should be created")

build_dir_files = os.listdir(str(build_dir))
self.assertIn("template.yaml", build_dir_files)
self.assertIn(function_logical_id, build_dir_files)

template_path = build_dir.joinpath("template.yaml")
resource_artifact_dir = build_dir.joinpath(function_logical_id)

# Make sure the template has correct CodeUri for resource
self._verify_resource_property(str(template_path),
function_logical_id,
"CodeUri",
function_logical_id)

all_artifacts = set(os.listdir(str(resource_artifact_dir)))
actual_files = all_artifacts.intersection(expected_files)
self.assertEquals(actual_files, expected_files)

ruby_version = None
ruby_bundled_path = None

# Walk through ruby version to get to the gem path
for dirpath, dirname, _ in os.walk(str(resource_artifact_dir.joinpath('vendor', 'bundle', 'ruby'))):
ruby_version = dirname
ruby_bundled_path = Path(dirpath)
break
gem_path = ruby_bundled_path.joinpath(ruby_version[0], 'gems')

self.assertTrue(any([True if self.EXPECTED_RUBY_GEM in gem else False for gem in os.listdir(str(gem_path))]))

def _verify_resource_property(self, template_path, logical_id, property, expected_value):

with open(template_path, 'r') as fp:
template_dict = yaml_parse(fp.read())
self.assertEquals(expected_value, template_dict["Resources"][logical_id]["Properties"][property])
3 changes: 3 additions & 0 deletions tests/integration/testdata/buildcmd/Ruby/Gemfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
source "https://rubygems.org"

gem "httparty"
Empty file.