CARTO Pass-Through API

[andrewbt]

• Determine approach for the pass-through API (test this? https://serverless.com/framework/docs/providers/aws/events/apigateway/#setting-an-http-proxy-on-api-gateway )
• do we want data.phila.gov/carto/, or /api/v2/sql, or another URL structure as a root for phl.carto.com/api/v2/sql?
• add the passthru API to the serverless.yaml

[timwis]

Now that I've actually read that doc page, it looks perfect!

[andrewbt]

I tried it today and couldn't get it to deploy. ``` andrewbt_1:~/serverless-testing $ sls deploy Serverless: Creating Stack... Serverless: Checking Stack create progress... ..... Serverless: Stack create finished... Serverless: Packaging service... Serverless: Uploading CloudFormation file to S3... Serverless: Uploading service .zip file to S3 (22 B)... Serverless: Updating Stack... Serverless Error --------------------------------------- Template format error: Unresolved resource dependencies [ApiGatewayRestApi] in the Resources block of the template Get Support -------------------------------------------- Docs: docs.serverless.com Bugs: github.com/serverless/serverless/issues Your Environment Information ----------------------------- OS: linux Node Version: 4.6.1 Serverless Version: 1.6.1 ``` Didn't have much time to debug, will be back at it later.

…

On Wed, Feb 8, 2017 at 7:11 PM, Tim Wisniewski ***@***.***> wrote: Now that I've actually read that doc page, it looks perfect! — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#19 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AB65c83u2ZZztx4Ir6zPE84xD8DiaEeRks5ralmogaJpZM4L7XRq> .

-- Andrew Thompson, Solutions Engineer *CARTO* | 201 Moore St, Brooklyn, NY, 11206 athompson@carto.com | T +1 508.333.7265 <+15083337265> Web carto.com <http://cartodb.com/> | Twitter @carto <http://twitter.com/carto> Predict Through Location Live sessions to learn CARTO <http://carto.com/webinars/>

[timwis]

This is almost working:

resources:
  Resources:
    ProxyResource:
      Type: AWS::ApiGateway::Resource
      Properties:
        ParentId:
          Fn::GetAtt:
            - ApiGatewayRestApi # our default Rest API logical ID
            - RootResourceId
        PathPart: carto # the endpoint in your API that is set as proxy
        RestApiId:
          Ref: ApiGatewayRestApi
    ProxyMethod:
      Type: AWS::ApiGateway::Method
      Properties:
        ResourceId:
          Ref: ProxyResource
        RestApiId:
          Ref: ApiGatewayRestApi
        AuthorizationType: NONE
        HttpMethod: GET # the method of your proxy. Is it GET or POST or ... ?
        MethodResponses:
          - StatusCode: 200
        Integration:
          IntegrationHttpMethod: GET
          Type: HTTP
          Uri: https://phl.carto.com/api/v2/sql # the URL you want to set a proxy to
          IntegrationResponses:
            - StatusCode: 200

The problem appears to be that API Gateway doesn't pass querystrings along by default, and it looks like the only way to enable them is to explicitly activate each one. Fortunately, with carto, we can activate q, format, and filename, though what if carto adds another parameter? And perhaps there are others that simply aren't in the sql api docs? Definitely not ideal :-/

Need someone familiar with API Gateway / cloudformation to weigh in. I've asked in freenode ##aws and the og-aws slack.

EDIT: Also, I think that ideally it would be data.phila.gov/api/v2/sql?q=xxxx for the URL schema, rather than /carto. Not super firm on it but it seems to make sense to me. Note that one of the socrata url structures is /api/views/.... but that shouldn't matter.

EDIT2: Apparently you can make the resource pathpart {proxy+} to use it like a "greedy parameter" and then use {proxy} in the method uri, like https://phl.carto.com/{proxy}. This may include querystrings. I haven't been able to get it to work. In particular, I was having trouble making the path something like /carto/{proxy+} or /api/v2/{proxy+}.

[timwis]

@andrewbt can you post the screenshots of the 3 configuration forms you got working when you get a chance? I can try to translate into cloudformation code this weekend.

[timwis]

Posted for help

[andrewbt]

Reposting screenshots from #37 :

Testable here: https://zsp6egcjhf.execute-api.us-east-1.amazonaws.com/dev/carto/v2/sql?q=select * from indego_stations where ST_DWITHIN(the_geom::geography, ST_GeographyFromText('POINT(-75.1636 39.9524)'),500)

Screenshots:

[andrewbt]

API Gateway allows you to “export” an API (under Stages) as a Swagger YAML or JSON document. One of the options for creating an API is through importing a Swagger document.

If we can’t get serverless.yaml to work with the HTTP-proxy integration and other options we need there for the pass-through API, Swagger seems like a good alternative and will still let us keep “deployment in code” rather than through the AWS console UI.

[timwis]

https://blog.jayway.com/2016/09/18/introduction-swagger-cloudformation-api-gateway/ 👯

[timwis]

We bloody cracked it!!

I got swagger to work, but it ended up overwriting all the serverless functions. But then I noticed that the swagger itself was basically cloudformation. So that told me what pieces were missing from the original cloudformation. And it's working! :D

https://67qq7fn418.execute-api.us-east-1.amazonaws.com/dev/carto/api/v2/sql?q=select%20*%20from%20pol_incidents_part1_part2%20limit%2010

[andrewbt]

Hey! That's awesome!

…

On Tue, Feb 28, 2017 at 7:47 AM Tim Wisniewski ***@***.***> wrote: [image: giphy] <https://cloud.githubusercontent.com/assets/761444/23405829/0d08737a-fd8a-11e6-8bd9-dd3e584a4a10.gif> We bloody cracked it!! I got swagger to work, but it ended up overwriting all the serverless functions. But then I noticed that the swagger itself was *basically* cloudformation. So that told me what pieces were missing from the original cloudformation. And it's working! :D https://67qq7fn418.execute-api.us-east-1.amazonaws.com/dev/carto/api/v2/sql?q=select%20*%20from%20pol_incidents_part1_part2%20limit%2010 — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#19 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AB65c5RZFDTModIV7f7Bb60e5EdLfNucks5rhBd1gaJpZM4L7XRq> .