AWS Application Load Balance

[riqueps]

Hello there.
I'm deploying Epinio on AWS EKS.
But I need to use AWS WAF for Epinio and Apps endpoints for security reasons.
But when I deploy Epinio on EKS using traefik or nginx, it is just able to create Classic ELB, even when I add aws-loadbalancer annotations.

[jimmykarily]

I'm not sure I'm reading the right docs but, I suppose you are using the epinio's values to set an annotation:

    kubernetes.io/ingress.class: alb

as described here

You say you deploy Epinio with traefik or nginx but afaict, the alb load balancer is supposed to be the Ingress controller. Is the alb load balancer installed and running on your cluster? Can you try to deploy Epinio without Traefik or nginx and let alb implement the Ingress? I've never used AWS WAF before so I could be completely off here.

[riqueps]

@jimmykarily Epinio needs to be deployed as a nodePort in order for the ingress to be exposed using the aws load balancer controller. We modified the service helm chart but Epinio is stuck on a loop after changing it to NodePort. Do you know if Epinio supports node port deployment?

Epinio-server logs shows me the errors below:
2022/08/01 19:54:51 EpinioServer: "msg"="ERROR" "error"="Route not found" "requestId"="27f1f601-61f3-42f8-b787-f1ddad1ad0a9" "origin"="/" "error"="Route not found"

[jimmykarily]

I don't fully understand your setup but let's start with what I know. The error message you posted indicates that a request landed on the epinio API server for this path: / . Such a route is not defined in the router thus the error you see.

In the default Epinio setup (with the Ingress and the load balanced traefik service), the epinio.your-system-domain-here url points to the epinio-ui Ingress/service. It seems to me that in your setup this points to the epinio API server service and that's why you see this error.

The epinio-ui backend is the one making requests against the Epinio API server and it never asks the / path, because there is nothing served there. Paths requested usually start with /api/v1/.

To be able to help you I need to understand a bit more about your setup and what you are trying to achieve. Here are some of my questions:

• Epinio needs to be deployed as a nodePort : where does this requirement come from? Any AWS documentation links?
• What services are you exposing using NodePort? As far as I can tell you've changed the epinio-server and the epinio-ui services to be NodePort, not the Traefik svc pointing to the ingress controller. In other words you are completely bypassing the Ingress controller. If that's true, you should also setup the epinio.your-system-domain-here domain to point to the IP address of one of your Nodes and you should use a Port in the URL to access it (Probably in the range 30000-32767, docs here).
• The spinner you see indicates that the epinio-ui code was somehow loaded (which contradicts with my assumption above that the url points to the epinio API server). I suspect that there some useful error message in the browser's console. Can you check?

[riqueps]

Hello @jimmykarily
I will try to explain it more detailed.
I’m deploying Epinio on AWS EKS (EC2 instances) and for security reasons I need to deploy a WAF in front of the Epinio endpoint and the applications which I am supposed to deploy inside of it.
The big concern is: AWS just let me deploy a WAF with Application LB and Application LB just works with NodePort service type.

• AWS ALB documentation (*second blue Note)
• AWS WAF documentation

So after some hard working I was able to deploy Epinio on EKS with AWS Application LB + WAF just by editing some manifest files from epinio helm chart and using AWS-Loadbalancer-Controller as ingress controller:

• Switching from ClusterIP to NodePort
  • chart/epinio/templates/server.yaml
  • chart/epinio/templates/charts/epinio-ui-0.7.0.tgz
    • templates/service.yaml
• Passing AWS LB annotations through values.yaml
  • alb.ingress.kubernetes.io/scheme: internet-facing
  • alb.ingress.kubernetes.io/target-type: ip
  • alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm*****
  • alb.ingress.kubernetes.io/wafv2-acl-arn: arn:aws:wafv2:*****
  • alb.ingress.kubernetes.io/ssl-redirect: '443'
  • alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS":443}]'
  • alb.ingress.kubernetes.io/healthcheck-path: /ready
• Also by setting “--set ingress.ingressClassName=alb” during installation

After this configuration my Epinio get UP and Running on EKS:

• Access Epinio UI
• Deploy sample application (with error)
  

But there are some problems:

• Not able to connect through CLI
  
• Not able to access the sample application from public, because when I deploy the application is created with ClusterIP service type.
  

[jimmykarily]

Thanks for the detailed information @riqueps! The documentation link you shared also mentions "LoadBalancer":

Your Kubernetes service must specify the NodePort or "LoadBalancer" type to use this traffic mode.

but the examples they provide use NodePort so I can't tell if LoadBalancer would work. In any case, since you can access the dashboard, it seems that your load balancer implements the Ingress just fine.

So the 2 issues you have:

1. Unauthorized on the cli: Are you sure the password you provide is correct? Have you changed the default password? As described here the user should be created with a bcrypt-ed value. (I just realized this text in docs is wrong Below, cGFzc3dvcmQ= is the base64 encoded version of password., we are preparing a fix)

2. Not being able to access the application: your whole setup is supposed to use NodePort for services. The default application chart sets the type to ClusterIP. You would need to modify that, probably by creating a custom application chart.

Another issue, I see in your last screenshot: "failed to get replica details". This will go away if you install a metric server.

[jimmykarily]

(I just realized this text in docs is wrong Below, cGFzc3dvcmQ= is the base64 encoded version of password., we are preparing a fix)

My mistake, docs are fine.

[riqueps]

@jimmykarily, thanks for the information.
About the issues:

1. Unauthorized on the cli: the issue was caused by ingress rules order, I just put the epinio-ui a the last position than cli has just worked. I think this could be caused by AWS-LB behavior or limitation ingress pathTypes (aws ingress specification).
2. Application access: after create custom application chart and did some manually changes at app service and ingress, to get aws lb target group working, I was able to access it from internet.

There is another issue related to custom app chart:

3. the apps create into this custom chart are not getting ingress annotation, even when is hard coded at ingress.yaml.

[jimmykarily]

@riqueps happy to hear you are getting the issues resolved.

3. the apps create into this custom chart are not getting ingress annotation, even when is hard coded at ingress.yaml.

I don't see how this could happen. I mean, Epinio eventually does a helm install of the application chart. If the annotations are defined in the chart, they should be set on the created Ingresses too. Are you sure you are using the custom helm chart you created? Sorry, I'm out of ideas about this one.

[riqueps]

@jimmykarily
about this last issue, it was my fault, I forget save the ingress file before compress it.
So, now Epinio is completly running on EKS.

As a improvement, I think it would be nice if the Ingress pathtype and Service type were mapped to values.yaml.
With these, Epinio would enhance compatibility with cloud providers.

Also, ingress rules orders, I think would be a good idea put the epinio-ui at the last position, to avoid epinio-server access issues.

During the process of pushing an app, is there a way to pass some values, using values.yaml?

[jimmykarily]

@riqueps glad to hear everything was sorted out. On you recommendations/questions:

• pathType: exposing it is easy but what the right pathType is, partly depends on the Ingress controller and partly on Epinio itself. E.g. I think ImplementationSpecific is wrong here, it should have been Prefix. I don't see why someone would want anything but Prefix in this case. Maybe it makes sense to allow setting that field so that people can easily debug their issues. In the end, we may have missed some case.
• Service type: If someone sets the service type of the epinio-ui to, let's say, NodePort, this means they are completely bypassing the Ingress, no? In that case, the Ingress resources for those services shouldn't be created at all. If I'm right, simply changing the type of the service is not enough on it's own, for a proper setup at least. Again, I'm fine opening this setting through values.yaml but I'd like to understand the use case better.
• Ingress rule order: I think the precedence of Ingress rules depends on the Ingress controller. For example nginx orders them by path length. Maybe if we used the right pathType instead of ImplementationSpecific would be the right solution? (see bullet#1 above). Can you try that in your setup and see if we can make the order of rules irrelevant by setting the pathTypes correctly?
• application chart values: Allow for developer to set custom values for apps (based on what is provided by the chosen custom appchart) #1252 (already in the backlog)

[riqueps]

@jimmykarily.

• pathType: ok.
• service type: as I'm using aws-load balancer controller, is a prerequisite the service type be NodePort, because the aws-lb will create a route directly to POD IP.
• Ingress rule order: unfortunately not. I tested with aws-lb-controller and the only way it worked was change the rules order.
• application chart values: ok

Also, I'm just ran into another problem.
When I try to execute "epinio app exec" I'm getting this error:

• "error getting a shell to application: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Upgrade request required","reason":"BadRequest","code":400}"
  Is there some configuration I missed?
  I took a look at setting.yaml, and looks fine, but there is no certificate, is it normal?

[jimmykarily]

Not having a certificate in settings.yaml would be a problem. You should be able to fix it with epinio settings update-ca. But if that was your problem, I think the error would look something like this:

~/workspace/epinio/epinio (main)*$ epinio app exec sample

🚢  Executing a shell
Namespace: workspace
Application: sample
2022/08/08 13:02:26 EpinioApiClient: "msg"="request failed" "error"="Get \"https://epinio.172.18.0.2.omg.howdoi.website/api/v1/authtoken\": x509: certificate signed by unknown authority"

❌  error getting a shell to application: making the request: Get "https://epinio.172.18.0.2.omg.howdoi.website/api/v1/authtoken": x509: certificate signed by unknown authority

(I got that by removing the certs from my settings file).

Try to fix the certificate problem with epinio settings update-ca (to make sure it doesn't interfere with the actual problem). Then try again but fist run this to increase the server's trace log level:

kubectl set env -n epinio deployment/epinio-server TRACE_LEVEL=20

and then check logs of the epinio-server pod for anything relevant. We should send back a more descriptive error but let's find out which error we are hiding first.

[riqueps]

I tried to run epinio settings update-ca but it got failed status.

And the settings file remain without any certificate.

But I'm able to push an app, create service, bind service through CLI.

Server logs, with TRACE_LEVEL=20 enabled, during epinio app exec wordpress command.
2022/08/08 12:35:07 EpinioServer/AuthMiddleware: "level"=2 "msg"="Basic auth authentication" "requestId"="6a400a54-b12a-4a42-996c-d79cfdc94bfa" 2022/08/08 12:35:08 EpinioServer/AuthorizationMiddleware: "level"=1 "msg"="authorization request from user [admin] with role [admin] for [GET - /api/v1/authtoken]" "requestId"="6a400a54-b12a-4a42-996c-d79cfdc94bfa" 2022/08/08 12:35:08 EpinioServer/AuthorizationMiddleware/authorizeAdmin: "level"=2 "msg"="user admin is authorized" "requestId"="6a400a54-b12a-4a42-996c-d79cfdc94bfa" 2022/08/08 12:35:08 EpinioServer/AuthorizationMiddleware: "level"=1 "msg"="user [admin] with role [admin] authorized [true] for namespace []" "requestId"="6a400a54-b12a-4a42-996c-d79cfdc94bfa" 2022/08/08 12:35:08 EpinioServer: "level"=1 "msg"="OK" "requestId"="6a400a54-b12a-4a42-996c-d79cfdc94bfa" "origin"="/api/v1/authtoken" "returning"={"token":"eyJhbGciOiJSUzM4NCIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJlcGluaW8tc2VydmVyIiwiZXhwIjoxNjU5OTYyMTM4LCJ1c2VyIjoiYWRtaW4ifQ.ABnPdmhWDW9PAgFs8TpqI64wynoEYzk4OfhqeCe5SMcAeTyifPZ5abPt20A0JjiN9OxekaEJxSSAzwD0M5JcGjbeeV5Zou2bIieFAc0clnrZfqjLEiOeLnoTBuLH6LrMGtZ1GkVdgVase0u-vT_XczPRzWfXTJKHu36ycWLsI0MKKFxNSXXMnDMBqStPiCe8A3YcyCVnL1cPaUYJLrdBWHHmsjFfLMZxGGlNEzSwHXz_mb05gjyRjfjYEzyOl-qPPdilnPEgD7gL81EQoybyTKRyK9a-1iVoR8vd7qyJ1ZlN-Nuv4y4a2qYSGej7v_hVOzjroSqFVJHDXb5zznz8Iw"} 2022/08/08 12:35:08 EpinioServer: "level"=1 "msg"="/api/v1/authtoken" "status"=200 "method"="GET" "path"="/api/v1/authtoken" "query"="" "ip"="189.14.243.145" "user-agent"="Go-http-client/2.0" "time"="2022-08-08T12:35:08Z" "latency"="96.826597ms" 2022/08/08 12:35:08 EpinioServer/TokenAuthMiddleware: "level"=2 "msg"="Authtoken authentication" "requestId"="9fc8ba3d-0f94-4ab9-aa7f-8daa1cbb14ea" 2022/08/08 12:35:08 EpinioServer/AuthorizationMiddleware: "level"=1 "msg"="authorization request from user [admin] with role [admin] for [GET - /wapi/v1/namespaces/workspace/applications/wordpress/exec]" "requestId"="9fc8ba3d-0f94-4ab9-aa7f-8daa1cbb14ea" 2022/08/08 12:35:08 EpinioServer/AuthorizationMiddleware/authorizeAdmin: "level"=2 "msg"="user admin is authorized" "requestId"="9fc8ba3d-0f94-4ab9-aa7f-8daa1cbb14ea" 2022/08/08 12:35:08 EpinioServer/AuthorizationMiddleware: "level"=1 "msg"="user [admin] with role [admin] authorized [true] for namespace [workspace]" "requestId"="9fc8ba3d-0f94-4ab9-aa7f-8daa1cbb14ea" 2022/08/08 12:35:08 EpinioServer: "level"=1 "msg"="/wapi/v1/namespaces/workspace/applications/wordpress/exec" "status"=400 "method"="GET" "path"="/wapi/v1/namespaces/workspace/applications/wordpress/exec" "query"="authtoken=eyJhbGciOiJSUzM4NCIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJlcGluaW8tc2VydmVyIiwiZXhwIjoxNjU5OTYyMTM4LCJ1c2VyIjoiYWRtaW4ifQ.ABnPdmhWDW9PAgFs8TpqI64wynoEYzk4OfhqeCe5SMcAeTyifPZ5abPt20A0JjiN9OxekaEJxSSAzwD0M5JcGjbeeV5Zou2bIieFAc0clnrZfqjLEiOeLnoTBuLH6LrMGtZ1GkVdgVase0u-vT_XczPRzWfXTJKHu36ycWLsI0MKKFxNSXXMnDMBqStPiCe8A3YcyCVnL1cPaUYJLrdBWHHmsjFfLMZxGGlNEzSwHXz_mb05gjyRjfjYEzyOl-qPPdilnPEgD7gL81EQoybyTKRyK9a-1iVoR8vd7qyJ1ZlN-Nuv4y4a2qYSGej7v_hVOzjroSqFVJHDXb5zznz8Iw" "ip"="189.14.243.145" "user-agent"="Go-http-client/1.1" "time"="2022-08-08T12:35:08Z" "latency"="90.859933ms"

Client side:

[jimmykarily]

There seem to be 2 different issues here:

• The timeout when updating the certs in settings.yaml
• The 400 when epinio app exec is called

Maybe the have the same cause so let's stick to this issue until we clarify. Regarding the 400, Epinio's exec endpoint always sends back a text when returning a BadRequest response. It seems that the error is coming from Kubernetes after it's been proxied by Epinio. Googling around, I see other people facing similar issues with ALB:

rancher/rancher#27871 (linking to something like a fix)

Maybe it SPDY protocol is not supported by ALB? (it's used under the hood for kubectl exec and epinio app exec):

• SPDY is deprecated. Switch to HTTP/2. kubernetes/kubernetes#7452
• https://stackoverflow.com/questions/19983678/has-anyone-managed-to-get-spdy-to-work-behind-an-amazon-elb

Can you try to kubectl exec to the application Pod? Does that work?
Also, what version of Kubernetes are your running on?

[riqueps]

I'm just reading your inputs.

But in advance, answering your questions:

• kubectl exec: Yes it works.
  
• Kubernetes cluster version: v1.22.9-eks-810597c
• Kubectl version (n my pc): 1.22.9

[riqueps]

@jimmykarily
After follow these instructions, I just got epinio app exec working.
Tks a lot!

[jimmykarily]

Welcome @riqueps . Thanks for your persistence to solve these issues. We discussed some changes that could be implemented (discussion started here). Can we move the "templetization" of those options (if you still think they are needed) to a new issue and close this one?

[riqueps]

@jimmykarily thanks for help me to overcome this issues.
Yes, I think it's important get this options exposed via values and of course, we can close this issue!!

[jimmykarily]

Fixing the ingress pathType here: #1685
Also changed the order of rules.