Localstack API Gateway does not understand request-parameters

[clicktravel-steffan]

I have successfully created, deployed and invoked a JavaScript Lambda function through a REST API created in API Gateway, all in Localstack. This is using the Lambda Proxy Integration as described here : http://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-set-up-simple-proxy.html#api-gateway-set-up-lambda-proxy-integration-on-proxy-resource

However, if I introduce a path parameter to the resource, localstack fails to route the HTTP request and instead complains that the API Gateway endpoint for my method cannot be found. It would appear that localstack's API Gateway implementation does not process request-parameters, as described here : http://docs.aws.amazon.com/cli/latest/reference/apigateway/put-method.html

Below I give full steps to reproduce the error.
Note: This was produced using the latest localstack Docker image:

$ docker images
REPOSITORY              TAG                 IMAGE ID            CREATED             SIZE
localstack/localstack   latest              c47a0b222d28        4 hours ago         935MB

Create a file index.js with these contents:

const apiTestHandler = (payload, context, callback) => {
  console.log (`Function apiTestHandler called with payload ${JSON.stringify(payload)}`);
  callback(null,
           {
             statusCode : 201,
             body : JSON.stringify({ somethingId : payload.pathParameters.somethingId }),
             headers : { "X-Click-Header" : "abc" }
           });
}

module.exports = {
  apiTestHandler,
}

Then use awslocal to create a Lambda function, create a REST API and deploy it:

$ zip apiTestHandler.zip index.js

$ awslocal lambda create-function \
    --region us-east-1 \
    --function-name api-test-handler \
    --runtime nodejs6.10 \
    --handler index.apiTestHandler \
    --memory-size 128 \
    --zip-file fileb://apiTestHandler.zip \
    --role arn:aws:iam::123456:role/role-name

$ awslocal apigateway create-rest-api --region us-east-1 --name 'API Test handler'

{
    "name": "API Test handler", 
    "id": "35937034A-Z3", 
    "createdDate": "2017-08-02T15:07:46.540Z"
}

$ awslocal apigateway get-resources --region us-east-1 --rest-api-id 35937034A-Z3

{
    "items": [
        {
            "path": "/", 
            "id": "782A-ZA-Z66970", 
            "resourceMethods": {
                "GET": {}
            }
        }
    ]
}

$ awslocal apigateway create-resource \
    --region us-east-1 \
    --rest-api-id 35937034A-Z3 \
    --parent-id 782A-ZA-Z66970 \
    --path-part "{somethingId}"

{
    "resourceMethods": {
        "GET": {}
    }, 
    "pathPart": "{somethingId}", 
    "parentId": "782A-ZA-Z66970", 
    "path": "/{somethingId}", 
    "id": "240A-Z01566A-Z"
}

$ awslocal apigateway put-method \
    --region us-east-1 \
    --rest-api-id 35937034A-Z3 \
    --resource-id 240A-Z01566A-Z \
    --http-method GET \
    --request-parameters "method.request.path.somethingId=true" \
    --authorization-type "NONE"

$ awslocal apigateway put-integration \
    --region us-east-1 \
    --rest-api-id 35937034A-Z3 \
    --resource-id 240A-Z01566A-Z \
    --http-method GET \
    --type AWS_PROXY \
    --integration-http-method POST \
    --uri arn:aws:apigateway:us-east-1:lambda:path/2015-03-31/functions/arn:aws:lambda:us-east-1:000000000000:function:api-test-handler/invocations \
    --passthrough-behavior WHEN_NO_MATCH

$ awslocal apigateway create-deployment \
    --region us-east-1 \
    --rest-api-id 35937034A-Z3 \
    --stage-name test

The above should deploy an endpoint GET /{somethingId} which responds with a body containing the path parameter somethingId. However, localstack fails to find the resource method entirely:

$ curl http://localhost:4567/restapis/35937034A-Z3/test/_user_request_/u-1234

{"message": "API Gateway endpoint \"/u-1234\" for method \"GET\" not found"}

The log for localstack shows this error :

127.0.0.1 - - [02/Aug/2017 15:54:10] "GET /restapis/35937034A-Z3/resources?limit=100 HTTP/1.1" 200 -
WARNING:localstack.services.apigateway.apigateway_listener:API Gateway endpoint "/u-1234" for method "GET" not found

Doing similar on real AWS gives the expected result:

$ curl https://xxxxxxxxx.execute-api.eu-west-1.amazonaws.com/test/u-1234

{"somethingId":"u-1234"}

[sshogunn]

Is it planned to fixe it any time soon?

[KamuelaFranco]

@sshogunn I am presently working on this. Don't know how fast progress will be though.

[KamuelaFranco]

@whummer Where might I start looking? I can't seem to find the code that handles API Gateway's put-method, http://docs.aws.amazon.com/cli/latest/reference/apigateway/put-method.html

Am I right in assuming that this functionality has not yet been implemented?

[whummer]

@KamuelaFranco For the basic "CRUD" functionality of API Gateway we're using moto under the covers, and LocalStack adds a bunch of integrations on top of it. See moto's add_method function here: https://github.com/spulec/moto/blob/master/moto/apigateway/models.py#L146

The way it works is that we're using moto_server in the background, and start up an HTTP proxy which intercepts all invocations and forwards requests to the backend. This allows us to add extended functionality without having to change the backend service.

 --------      -------------      -------------
| Client | -> |    Proxy    | -> |   Backend   |
|        |    | (port 4567) |    | (port 4566) |
 --------      -------------      -------------

(The default ports can be found in https://github.com/localstack/localstack/blob/master/localstack/constants.py#L18-L43 ) The proxy follows a simple protocol by implementing 2 methods: forward_request which is called before a request is forwarded to the backend, and return_response which is called after a response has been received from the backend: https://github.com/localstack/localstack/blob/master/localstack/services/generic_proxy.py#L38-L68

The proxy implementation for API Gateway can be found here: https://github.com/localstack/localstack/blob/master/localstack/services/apigateway/apigateway_listener.py#L81

Given that your planned changes affect mostly the integration part (rather than the CRUD part), it probably makes more sense to add this functionality to apigateway_listener.py. However, depending on the type of changes required, you may want (or need) to extend moto as well. Moto has a relatively slow/irregular release cycle, hence we're using a forked version here (branch localstack-fixes) which is published as a separate moto-ext pip package: https://github.com/whummer/moto/tree/localstack-fixes . If you decide to extend moto, you can either raise a PR against that repo, or against the main repo spulec/moto (I'll take care of cross-merging and releasing).

EDIT: Actually, I feel like adding request parameters has some similarities with mapping templates: http://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-mapping-template-reference.html. They both store a static string with variable placeholders which are then replaced with actual parameter values at invocation time. An example for how we apply/transform mapping templates can be found here: https://github.com/localstack/localstack/blob/master/localstack/services/apigateway/apigateway_listener.py#L100

Hope that helps as a starting point. Please keep me in the loop and let me know if there are any questions. Thanks

(Btw, we've been planning for a long time to write up some developer documentation, the above may serve as a good starting point. I'm going to start a wiki/README page on that.)

[KamuelaFranco]

@whummer The approach I'm working on so far is entirely encompassed in apigateway_listener.py. I'm trying to manage registering path parameters and handling requests involving them here. Sound like a reasonable if not under-engineered approach?

[whummer]

This should be fixed in the meantime. Closing this issue - please re-open if the problem persists. Thanks

[benanderton]

It seems that this issue is not fixed, please see #3448