Skip to content
URL Shortener using: Node.js, Serverless, AWS Lambda, AWS DynamoDB
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
.dynamodb
build
seed
.eslintrc.js
.gitignore
LICENSE
README.md
dynamodb.js
handler.js
package.json
secrets.example.json
serverless.yml
yarn.lock fix: arn for index Feb 4, 2019

README.md

URL Shortener

Goal of this project:

  • Create a URL Shortener
  • Keep the cost of running it cheap
  • Experiment with AWS Lambda and DynamoDB for my own learning
  • Create an example that other people can learn from
  • Really short URLs
    • Achieving this via using base58 on incremental ids
    • Incremental IDs
    • I think this is OK because I am not planning to have that many urls
  • Reserve shortest URLs
    • Since using incremental ids, we can start from an offset
  • Authentication using AWS IAM

Cannot use Go lang because you cannot invoke functions locally, though I didn't really use that for this.

Warnings, Disclaimer, Gotchas

  • I am new to all this serverless infrastructure. Keeping notes at:
  • This code isn't perfect. It's been cobbled from many different examples and sources you can find in the "Resources" section below.
  • Not familiar with the best optimizations
  • Serverless (as in the tool) will drop your tables if you change the name of the tables. You have been warned.
  • I don't fully understand the limitations of DynamoDB, Lamabda, Serverless
  • Possible DyanmoDB scaling problems:
    • We are using a singular key to keep track of the counter
  • Investigate why deploying to a different stage creates a new API Gateway
    • See Deploy section for more notes.

TODOs, Bugs, Improvements, Features

  • Fix naming
  • Features
    1. Consider authentication using Cognito
    • Cons: More infra to manage and understand
    1. Batch operation. Support submitting multiple urls
  • Speed Improvement
    1. Hash URL to speed upfetch or create. I haven't tested lookup speeds using just URL. But I assume if I use a checksum of the URL I could get a minor speed improvement.
  • Learn more about what I am doing
    • I really have no clue; but it works!

Stack

  • Serverless
  • Node.js
  • AWS IAM
  • AWS Lambda
  • AWS DynamoDB

Requirements

Java

On Mac:

brew cask install java

Install

Optional: Configure secrets.json (copy from secrets.example.json.

yarn install
sls dynamodb install --stage dev

Start Offline

First run:

sls offline start --migrate --seed --stage dev --region ap-southeast-1

Subsequent runs:

sls offline start --stage dev --region ap-southeast-1

Sometimes it fails to start, what helps is reinstalling dynamodb:

rm .dynamodb/shared-local-instance.db
sls dynamodb remove
sls dynamodb install --stage dev

Testing

curl -v http://localhost:3000/

Create Short URL (Without Authentication)

serverless-offline doesn't support aws_iam authentication, so feel free to ignore if testing offline.

To disable authentication, disable/delete the authorizer: aws_iam line in serverless.yml

To test, call:

curl -v -H "Content-Type:application/json" http://localhost:3000/ --data "{ \"url\": \"http://example.com/$RANDOM\" }"

Create Short URL (With Authentication)

Enable/Add the authorizer: aws_iam line in serverless.yml.

Because we need to authenticate in order to create a short url see (test.js is not yet implemented):

./test.js --region ap-southeast-1 http://localhost:3000/ http://example.com/$RANDOM

Example on how to call it from your code.

Deploy

SLS_DEBUG=* sls deploy --stage dev --region ap-southeast-1

Notes on deploying to a different stage:

  • Createa a new CloudFormation Stack
    • ie: url-shortener-dev vs url-shortener-prod
  • Creates a new API Gateway
    • I would assume it would create a new stage in API Gateway

With Custom Domain Name

Configure ./secrets.json:

SLS_DEPLOY=* sls create_domain --stage dev --region ap-southeast-1
SLS_DEPLOY=* sls deploy --stage dev --region ap-southeast-1

Create IAM User

In order to use authentication you have to create a new user with the correct policy.

Visit https://console.aws.amazon.com/iam/home?#/users and create a user, I call it url-shortener.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "execute-api:Invoke",
                "execute-api:InvalidateCache"
            ],
            "Resource": [
                "arn:aws:execute-api:*:*:$DOMAIN/$STAGE/POST"
            ]
        }
    ]
}

Replace $DOMAIN and $STAGE.

Use the Access Key ID and Secret Access Key to make requests to your endpoint.

https://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-iam-policy-examples-for-api-execution.html

Note: This may take several minutes to be enabled.

Custom Domain

I couldn't use Namecheap because, AWS Custom Domain requires the A Record to support an Alias Target. So I pointed my Nameservers to AWS Route 53.

AWS Route 53 doesn't support .app domains, so I couldn't transfer the domain over.

Command line to check NS:

host -t ns example.com

In parameters.yml

custom:
  customDomain:
    domainName: <registered_domain_name>
    basePath: ''
    stage: ${opt:stage}
    createRoute53Record: true

Manual Setup

DynamoDB Shell

http://localhost:8000/shell/

List Tables:

var params = {
    Limit: 5, // optional (to further limit the number of table names returned per page)
};
dynamodb.listTables(params, function(err, data) {
    if (err) ppJson(err); // an error occurred
    else ppJson(data); // successful response
});

Scan Tables:

var params = {
    TableName: 'counters-dev'
}
dynamodb.scan(params, function(err, data) {
    if (err) ppJson(err); // an error occurred
    else ppJson(data); // successful response
});

Put Item:

var params = {
    TableName: 'counters-dev',
    Item: {
        name: "url-shortener",
        value: -1,
    },
    ReturnValues: "ALL_OLD",
}
docClient.put(params, function(err, data) {
    if (err) ppJson(err); // an error occurred
    else ppJson(data); // successful response
});

Get Item:

var params = {
    TableName: 'url-shortener-dev',
    Key: { // a map of attribute name to AttributeValue
        uuid: "1",
    },
};
docClient.get(params, function(err, data) {
    if (err) ppJson(err); // an error occurred
    else ppJson(data); // successful response
});

AWS Console Links

Resources

Serverless Plugins

Base58 Decode

  • https://en.wikipedia.org/wiki/Base58
  • Using initial offset of asa or 31793
  • Using asa so I start with using the left side of the keyboard
  • I want to start with an offset to reserve shorter urls

Developer Notes

Why JavaScript?

  • I originally wanted to use Go Lang because:
    • I like the language
    • Typesafe
    • Reusable skills for building other backend/frontend systems
    • No Promises, or indent hell
    • gofmt for consistency
  • Unfortunately "serverless-offline" doesn't support Go Lang
  • I was already familiar with JavaScript

Naming conventions:

  • ${stage} as a prefix
    • API Gateway uses it as the basepath, so I figure we should just use it in the beginning
    • Visually groups the same stage together

Should the default be to reuse a shorturl or create a new one?

  • Currently opting to create a new one

Why did you not use Cognito?

  • I didn't want to manage yet another service
  • I didn't want to learn more
  • I didn't want to increase complexity
  • Focusing on MVP
You can’t perform that action at this time.