Use App Mesh for ingress routing

[jamsajones]

No description provided.

[zbintliff]

Finding this issue saved me some time. Was deep diving documentation to see if we could use it for ingress. Currently planning on using Envoy for that as well, but missing the control plane piece.

[coopr]

@zbintliff if you didn't find it in the spot in the docs you'd expect to see it, feel free to PR an improvement to the docs 😄 - TIA!

[zbintliff]

Definitely. Does that mean my understanding from above correct?

…

On Fri, Nov 30, 2018, 1:31 PM Cooper Marcus ***@***.***> wrote: @zbintliff <https://github.com/zbintliff> if you didn't find it in the spot in the docs you'd expect to see it, feel free to PR an improvement to the docs 😄 - TIA! — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <https://github.com/awslabs/aws-app-mesh-examples/issues/31#issuecomment-443296123>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AHLWl4LczGOD0S9bGPeG1KOyrHJsXer2ks5u0Xl7gaJpZM4Y4M2b> .

[coopr]

My understanding is that your understanding is correct @zbintliff - AWS App Mesh does not (yet) support ingress routing or typical "edge" API gateway functions. I'm happy to be corrected if I'm mistaken 😉

[jamsajones]

@zbintliff and @coopr you are correct.

Service to service communication has been our focus at this point. We do not do anything to keep customers from deploying an Envoy that would work as an ingress router, but we have not tested it ourselves, do not have any good examples for it and so there could be subtle gotchas. Your continued input here will help us understand and prioritize this feature. If you feel that you have any interesting use cases around this feature please let us know. Cheers!

[pawelprazak]

any plans for API Gateway integration?

[FractalJim]

Lack of ingress routing is the main thing holding us back from using AppMesh at the moment. Without ingress routing we need to maintain separately managed config for ingress routing as well as taking on AppMesh config, making the benefits over doing your own envoy deployment on EKS less clear. With ingress routing AppMesh is a clear winner.

[rverma-nikiai]

@jamsajones Does it meant that we can't make alb ingress communicate with virtual services/nodes.
Also is it possible to expose a service publically if that is deployed as part of app-mesh?

[mumoshu]

In case this "App Mesh for Ingress Routing" works like a managed Envoy cluster controllable via App Mesh API, I'd definitely love it.

I'll use it as an alternative to ALB, because it gives us gRPC load-balancing/routing which isn't possible with ALB.

[manikawnth-abg]

@jamsajones Does it meant that we can't make alb ingress communicate with virtual services/nodes.
Also is it possible to expose a service publically if that is deployed as part of app-mesh?

I think as long as you attach ALB while creating your ecs service, the target group can act as service registry for ALB ingress. This can be done along with your service discovery (rt 53) while creating ecs service, which is used for within mesh communication.
Checkout the colorteller demo of appmesh
Folks can correct me if I'm wrong.

[rverma-nikiai]

@manikawnth-abg I am not very clear on this. Anyways we are trying to run app-mesh on eks. Currently, we have target groups with health check behind ALB load balancer.
When I injected app-mesh sidecar to those containers the health check started failing. It's simply that now the load balancers are not able to connect to the pod by IP.

Can you please expand your answer in this direction?

[kiranmeduri]

@rverma-nikiai that seems like a misconfiguration or a bug. Can you please open a new issue where we can dive deep into it. Thanks

[rverma-nikiai]

@kiranmeduri just to ensure what I am doing is correct, I created an ingress (say hello.aws.com) using aws-alb-ingress-controller. Then created a virtual service(hello) and a virtual node(hello.aws.com with service discovery as DNS) and a virtual route along with standard kubernetes service & pod. Added egress filter as allow all for now.

Doing the above steps I observed two things.

1. ALB can't do health checks
2. ALB traffic is not coming to pod.

There is no concept of gateway unlike istio and I also couldn't find an example in eks which demonstrate how to expose a service over https publically.
Appreciate your help.

Also wondering if we compulsorily require cloudmap to work with app mesh, since all our services are within eks cluster and only outside resources are rds, dynamo, sqs, redis?

[kiranmeduri]

@rverma-nikiai When Envoy is injected into the pod, it intercepts all the incoming and outgoing traffic from the application container. This should be true for health-check too.

• Can you confirm that health-check using curl against a candidate pod is also failing?

• Next can you see that Envoy has correct cluster definition (distributed by appmesh via xDS)

kubectl exec -it $POD_ID -c envoy curl http://localhost:9901/clusters | cut -f1 -d":" | sort | uniq

Above command should show proper clusters configured.

• If you see clusters then can you share envoy access logs on how the health-checks are failing.

kubectl logs -f $POD_ID -c envoy 

Thanks

[tomaszdudek7]

+1, I need this badly.

As referenced in #119, this feature would enable to do canary deployments internally + externally at the same time. Right now ALB does not mix in nicely and we have to run our own Virtual Node(nginx, HAproxy, zuul or self-made ColorGateway in AWS ColorApp example) on the edge to do so.

That is both time consuming(configuring nginx or zuul, creating our own gateway), dangerous(needs to have HA etc.) and prone to human errors(misconfiguration nginx vs Virtual Routers).

I would love(and pay) AWS to take off this burden from my team

[rverma-nikiai]

I believe the best place for edge proxy is route53 and with its support for weighted routing. IWith Alb-ingress-controller(latest release) we can have two app-groups v1, v2. All the edge services mapped to virtual node as usual. The missing component is the external-dns support for weighted routing kubernetes-sigs/external-dns#196, but the PR branch work.

I am guessing this would be more performant as well. If anyone can foresee any issues with this approach would love to hear.

With this approach, there are two main advantages for us, WAF and Cognito :)

[mumoshu]

@rverma-nikiai The solution you proposed works, as long as you use HTTP only.

For HTTP/2 and gRPC(#13), DNS-level weighting doesn't work as they use persistent connections that can be much longer than the TTL of your DNS records.

Of course I'm not saying your solution doesn't work. It is just that we still need something that doesn't depend on DNS/Route 53 for h2 and grpc :)

[kiranmeduri]

Referencing API Gateway proposal #111

[eddgrant]

Hi folks,

If I've understood the approach correctly, each time a service is deployed in to the mesh it is necessary to add a new route to the VirtualGateway (assuming we want the new service to be accessible from outside of the mesh)?

If this is the case, it would be it would be really helpful if route management on the virtual gateways was idempotent. This would avoid us having to know whether we should be creating or updating a route each time a service is deployed. We could instead be completely stateless and just "attach" the route each time. Would be great if you could give some guidance as to whether this is something that has been considered yet?

Also, I appreciate this is still in preview, but was wondering if you're able to give any hints as to whether route management will end up getting CloudFormation support? We currently manage all our deployments in CloudFormation and it would be great to be able to attach routes within our CF stacks.

Cheers,

Edd

[bigdefect]

@eddgrant Thanks for your comment.

each time a service is deployed in to the mesh it is necessary to add a new route to the VirtualGateway

If I follow the rest of the message, deploy here refers to a net-new Virtual Service. In that case, a new route (or a modification of a route) would naturally be needed, or an existing route would need to be modified to point to the new VS. After that, unless you're changing routing, the gateway route shouldn't need to be modified for new revisions of that service.

it would be really helpful if route management on the virtual gateways was idempotent

The idea of an "attach" primitive is interesting. We've focused on providing declarative CRUD primitives and having Virtual Services serve as the "smart pointer". What is your use-case such that you'd need to update a gateway route on each service revision? Given that you're already using CloudFormation, what is missing, given that it will already decide on create vs update for you? Just looking for more information on what you would want your integration to look like.

Re: CloudFormation, we will indeed support Virtual Gateways and Gateway Routes via CloudFormation. They'll typically arrive shortly after the feature is fully released.

[eddgrant]

Hi @efe-selcuk ,

Thanks for taking the time to respond and apologies for taking so long to respond myself.

Great to hear that CloudFormation support will be coming. My reference to the "attach" primitive was really my thinking about how the various create and update API calls might be orchestrated, but thinking about it now, that's exactly what CloudFormation does already, so I think what you're proposing would actually work for us.

We're looking forward to the release of this in CloudFormation, cheers!

[herbertgoto]

AWS CLI 2.0 Support

Summary
Using AWS CLI 2.0 you cannot add the model for the preview channel.

Steps to Reproduce
Using AWS CLI 2.0.0, I execute the following command:

aws configure add-model \
    --service-name appmesh-preview \
    --service-model https://raw.githubusercontent.com/aws/aws-app-mesh-roadmap/master/appmesh-preview/service-model.json

It returns the following error:

Expecting value: line 1 column 1 (char 0)

Are you currently working around this issue?
Yes. By using AWS CLI 1.17.7

[herbertgoto]

I follow instructions in: https://docs.aws.amazon.com/app-mesh/latest/userguide/virtual_gateways.html

When I try to create the virtual-gateway I get an error. I have tried with:

aws appmesh-preview create-virtual-gateway --cli-input-json file://virtual-gateway.json

aws appmesh-preview create-virtual-gateway --mesh-name appmesh-service-discovery --virtual-gateway-name myVGW --spec file://virtual-gateway.json

The error I get is:

An error occurred (NotFoundException) when calling the CreateVirtualGateway operation: Mesh with name vgw is not found under account

If I call aws appmesh list-meshes, I can see the meshName I am using.

[bigdefect]

@herbertgoto Is it the first or second command that fails? In the second, you're passing in appmesh-service-discovery as the mesh name, which of course is not the mesh vgw in your error message.

Also, could you please report that CLI 2.0 problem as a separate issue?

[rajal-amzn]

@herbertgoto We have an open issue to track the support for AWS CLI 2.0 support.

When you create AppMesh resources using aws appmesh-preview, the resources are created against our preview environment whereas the resources created using aws appmesh are created against our prod environment. I guess, you are trying to access the prod resources from preview. If yes, it would not work. Please create a mesh using aws appmesh-preview create-mesh and then use it to create virtual gateway.

[herbertgoto]

@herbertgoto Is it the first or second command that fails? In the second, you're passing in appmesh-service-discovery as the mesh name, which of course is not the mesh vgw in your error message.

Also, could you please report that CLI 2.0 problem as a separate issue?

Both comments have been resolved by @rajal-amzn

[isaac-mj]

Hi guys - I have an upcoming project that could be a good candidate to use virtual gateways. However, I need this feature in production. Do you have an estimation about it? It would help me decide whether we should wait to include it or not.

[rajal-amzn]

@isaac-mj We cannot commit to a date as per policies but I can say that this is our top priority and we are actively working on releasing it in production. You can expect the release soon.

[nightmareze1]

+1 I wait a ingress gw for ECS! Because in kubernetes is easy but in ecs is very complex you.need code you personal api-gateway..

[rajal-amzn]

Hey all,
We will be making a minor change to the VirtualGateway API when we release this feature globally. Initially, we had designed to let customers define logging as part of the listeners object which might help the gateway owners to separate out the logs since Virtual Gateways serve traffic to multiple services. On further analysis from our end, we have decided to pull it out of the listeners and have one common logging per VirtualGateway.
So the updated API model will look as below:

{ 
    "type" : "AWS::AppMesh::VirtualGateway", 
    "properties" : 
    { 
        "meshName" : "SampleMesh", 
        "virtualGatewayName" : "IngressGateway",
        "spec" :
        {
            "backendDefaults" : {
                ...
            }, 
            "listeners" : [  
            { 
                "healthCheck" :
                {
                    ...
                },
                "portMapping" :
                {
                    ...
                },
                "tls":
                {
                    ...
                }
            }
            ],
            "logging" : 
            {
                "accessLog" : {
                    "file" : {
                        "path" : "/path/to/access.log"
                    }
                }
            }
        } 
    } 

We have updated our preview model as per this new API. This would be a breaking change to the models created in preview currently. So, if you have already created the access log, it would continue to work until we release this feature in GA. Post that, you will have to update your Virtual Gateway API to reflect the changes required, in order to get the access log changes working.

[benpettman]

What kind of authentication will this have? OR how can it be used to authenticate requests? Does this have to be done by some other mechanism?

[bigdefect]

@benpettman Initially, TLS and x.509 certificates are going to be the authentication mechanism from the gateway into your mesh. We're in research and/or design phase for Mutual TLS (#34) and external authorization (#140), both of which have implications for authentication at the gateway.

Can you tell us more about your use case?

[zaheershaik]

AWS CLI 2.0 Support

Summary
Using AWS CLI 2.0 you cannot add the model for the preview channel.

Steps to Reproduce
Using AWS CLI 2.0.0, I execute the following command:

aws configure add-model \
    --service-name appmesh-preview \
    --service-model https://raw.githubusercontent.com/aws/aws-app-mesh-roadmap/master/appmesh-preview/service-model.json

It returns the following error:

Expecting value: line 1 column 1 (char 0)

Are you currently working around this issue?
Yes. By using AWS CLI 1.17.7

I had the same issue. I was able to work around by downloading the service model json locally and by referring the file using file:// in the add-model command.

[rajal-amzn]

Hey all,
Thank to everyone for your feedback. Ingress Gateways for AppMesh is now generally available. Check out this blog post for more details: https://aws.amazon.com/about-aws/whats-new/2020/07/aws-app-mesh-launches-ingress-support-for-virtual-gateways/

NOTE: Cloudformation support for this is yet to be launched. We will update this issue once it is done.

[rajal-amzn]

Cloudformation support for Virtual Gateway are live. Check the docs for details.

Resolving this issue.