Has anyone deployed this on Kubernetes running on AWS?

[raviada]

Description:
I am wondering if anyone successfully implemented this on Kubernetes running on AWS EC2? I got everything deployed, including nginx ingress. Everything seems to be working, it even created Classic Load Balancer on AWS with two listeners 80/443, and a security group that accepts all inbound traffic. I am not able to get store or publisher sites working. It is always going to default backend.

kubectl get pods --namespace wso2

apim-rdbms-deployment-fc688fcd8-8l2mp     1/1       Running   0          21h
efs-provisioner-578dc8989f-kt2f4          1/1       Running   0          15d
wso2apim-analytics-1-84d66b6d66-9mxjc     1/1       Running   0          21h
wso2apim-analytics-2-579645c7f4-lgz44     1/1       Running   0          21h
wso2apim-is-as-km-69f85fb856-46vxs        1/1       Running   0          21h
wso2apim-manager-worker-69fdc9994-9tk6c   1/1       Running   0          21h
wso2apim-manager-worker-69fdc9994-g7j7p   1/1       Running   0          21h
wso2apim-pubstore-tm-1-7cf75854bf-jdqmt   1/1       Running   0          21h
wso2apim-pubstore-tm-2-8568c75b45-fn9tn   1/1       Running   0          21h

kubectl get services --namespace wso2

apim-rdbms                ClusterIP   None             <none>        3306/TCP                                                         21h
wso2apim                  NodePort    10.96.252.21     <none>        9763:31232/TCP,9443:31389/TCP                                    17h
wso2apim-analytics        ClusterIP   10.96.105.108    <none>        9764/TCP,9444/TCP,8082/TCP,4041/TCP,11501/TCP                    21h
wso2apim-analytics-1      ClusterIP   None             <none>        7612/TCP,7712/TCP,11225/TCP,10006/TCP,11001/TCP                  21h
wso2apim-analytics-2      ClusterIP   None             <none>        7612/TCP,7712/TCP,11225/TCP,10006/TCP,11001/TCP                  21h
wso2apim-gw               ClusterIP   10.106.238.224   <none>        8280/TCP,8243/TCP                                                21h
wso2apim-gw-sv            ClusterIP   10.106.110.32    <none>        9763/TCP,9443/TCP                                                21h
wso2apim-km-clustering    ClusterIP   None             <none>        9673/TCP,9443/TCP                                                21h
wso2apim-km-internal      ClusterIP   10.108.235.110   <none>        9763/TCP,9443/TCP                                                21h
wso2apim-manager-worker   ClusterIP   None             <none>        9673/TCP,9443/TCP                                                21h
wso2apim-pubstore-tm-1    ClusterIP   None             <none>        5672/TCP,9711/TCP,9611/TCP,7711/TCP,7611/TCP,9763/TCP,9443/TCP   17h
wso2apim-pubstore-tm-2    ClusterIP   None             <none>        5672/TCP,9711/TCP,9611/TCP,7711/TCP,7611/TCP,9763/TCP,9443/TCP   17h

kubectl get pods --namespace ingress-nginx

default-http-backend-55c6c69b88-bp4wt       1/1       Running   0          20h
nginx-ingress-controller-78cc9d4ccb-xtjxt   1/1       Running   0          20h

kubectl get services --namespace ingress-nginx

default-http-backend   ClusterIP      10.96.63.34      <none>             80/TCP                       20h
ingress-nginx          LoadBalancer   10.108.178.119   a0c93fd6939d6...   80:31310/TCP,443:30640/TCP   20h

curl -v http://a0c93fd6939d611e885020a6cd68e025-1142036405.us-east-1.elb.amazonaws.com:443/store
*   Trying 34.202.146.173...
* TCP_NODELAY set
* Connected to a0c93fd6939d611e885020a6cd68e025-1142036405.us-east-1.elb.amazonaws.com (34.202.146.173) port 443 (#0)
> GET /store HTTP/1.1
> Host: a0c93fd6939d611e885020a6cd68e025-1142036405.us-east-1.elb.amazonaws.com:443
> User-Agent: curl/7.57.0
> Accept: */*
> 
* Recv failure: Connection reset by peer
* Closing connection 0
curl: (56) Recv failure: Connection reset by peer

[chirangaalwis]

@raviada FYI, currently there is an effort being carried out on improving and fine tuning the Kubernetes resources for WSO2 Enterprise Integrator which includes focus on deploying the Kubernetes resources in different environments (including AWS).

The same effort is expected to be repeated for Kubernetes resources for WSO2 API Manager, as well.

[raviada]

Thanks Chiranga for quick response. I used this repo to create kube resources on kube running on AWS, everything was working as expected. All containers started and running.

The only problem is not being able to access the store/publisher urls from AWS loadbalancer created by ingress controller. If I can figure this out, I might save all the work I did for the last few days..

[AmreeshTyagi]

@raviada Sorry for context switching, but can you please help me on this issue , because you already have running cluster with all required services. I have tried to deploy patter-2 & pattern-4 too.

Thanks in advance.

[raviada]

Hi Amreesh, If that is what you are facing, the issue you referred in your message, I did not come across it exactly like what you were facing, but I had several issues with EFS mounts on AWS.

If you are having trouble with EFS and PVs, I used https://github.com/kubernetes-incubator/external-storage/tree/master/aws/efs in my setup to solve problems around NFS mounts.

[pubudu538]

Hi Raviada,

Have you created any ingress resource for APIM as in https://github.com/wso2/kubernetes-apim/blob/master/pattern-1/artifacts/ingresses/wso2apim-ingress.yaml?

I can see that you have used wso2apim service as NodePort. In that case, you can access the server as below.

https://Any_K8s_Node_IP:31389/store.

Thank you!
Pubudu.

[raviada]

Thanks for the reply, but all the nodes have internal IPs and cannot be accessed. Don't we access them thru ELB?

kubectl get nodes
NAME                          STATUS    ROLES     AGE       VERSION
ip-10-0-17-194.ec2.internal   Ready     master    22d       v1.9.3
ip-10-0-25-208.ec2.internal   Ready     <none>    22d       v1.9.3
ip-10-0-26-185.ec2.internal   Ready     <none>    22d       v1.9.3

[imesh]

@raviada Shall we try to troubleshoot the problem you are having as follows:

• First, we may need to check the output of the WSO2 API Manager containers using "kubectl logs" command or via the K8S dashboard.
• Ideally there should not be any errors in these outputs. If we see errors, we may first need to fix those.
• Then, if we do not see any errors in WSO2 API Manager containers, we may need to check whether the API Manager K8S services have deployed properly. We can use "kubectl get services --namespace=wso2" command for this.
• Thereafter, we may need to check the status of the ingresses. All API Manager ingresses should be successfully deployed.
• Then, we may need to check the health-checks of the AWS load balancers created for the ingresses.

If you can try out above and let us know the results we should be able to identify the root cause. Thanks!

[raviada]

Imesh, I did all that, I think way passed that stage. I am having trouble accessing the store/publisher urls because wso2apim service from the got repo is of type NodePort, all the nodes have internal IPs. So I changed it to LoadBalancer type now, I see that there is LB created on AWS.

but still I can't access the webapps.

here is the output of api amanger log
kubectl logs -f wso2apim-manager-worker-69fdc9994-9tk6c -n wso2

[2018-04-06 18:39:37,855]  INFO - RegistryEventingServiceComponent Successfully Initialized Eventing on Registry
[2018-04-06 18:39:38,049]  INFO - JMSListener Started to listen on destination : throttleData of type topic for listener Siddhi-JMS-Consumer#throttleData
[2018-04-06 18:39:38,096]  INFO - JMXServerManager JMX Service URL  : service:jmx:rmi://localhost:11111/jndi/rmi://localhost:9999/jmxrmi
[2018-04-06 18:39:38,098]  INFO - StartupFinalizerServiceComponent Server           :  WSO2 API Manager-2.2.0
[2018-04-06 18:39:38,099]  INFO - StartupFinalizerServiceComponent WSO2 Carbon started in 211 sec
[2018-04-06 18:39:39,043]  INFO - CarbonUIServiceComponent Mgt Console URL  : https://192.168.117.189:9443/carbon/
[2018-04-06 18:39:39,044]  INFO - CarbonUIServiceComponent API Store Default Context : https://192.168.117.189:9443/store
[2018-04-06 18:39:39,044]  INFO - CarbonUIServiceComponent API Publisher Default Context : https://192.168.117.189:9443/publisher
[2018-04-06 18:54:34,292]  INFO - AndesRecoveryTask Running DB sync task.
[2018-04-06 19:09:34,292]  INFO - AndesRecoveryTask Running DB sync task.
.......
[2018-04-16 11:09:34,292]  INFO - AndesRecoveryTask Running DB sync task.
[2018-04-16 11:23:01,502]  INFO - SessionCleanUpService Session Data cleanup task is running successfully for removing expired Data
[2018-04-16 11:24:34,292]  INFO - AndesRecoveryTask Running DB sync task.

kubectl get services -n wso2

NAME                      CLUSTER-IP       EXTERNAL-IP        PORT(S)                                                          AGE
apim-rdbms                None             <none>             3306/TCP                                                         9d
wso2apim                  10.104.87.140    a1f39d6e13e99...   9763:31068/TCP,9443:30425/TCP                                    3d
wso2apim-analytics        10.96.105.108    <none>             9764/TCP,9444/TCP,8082/TCP,4041/TCP,11501/TCP                    9d
wso2apim-analytics-1      None             <none>             7612/TCP,7712/TCP,11225/TCP,10006/TCP,11001/TCP                  9d
wso2apim-analytics-2      None             <none>             7612/TCP,7712/TCP,11225/TCP,10006/TCP,11001/TCP                  9d
wso2apim-gw               10.106.238.224   <none>             8280/TCP,8243/TCP                                                9d
wso2apim-gw-sv            10.106.110.32    <none>             9763/TCP,9443/TCP                                                9d
wso2apim-km-clustering    None             <none>             9673/TCP,9443/TCP                                                9d
wso2apim-km-internal      10.108.235.110   <none>             9763/TCP,9443/TCP                                                9d
wso2apim-manager-worker   None             <none>             9673/TCP,9443/TCP                                                9d
wso2apim-pubstore-tm-1    None             <none>             5672/TCP,9711/TCP,9611/TCP,7711/TCP,7611/TCP,9763/TCP,9443/TCP   9d
wso2apim-pubstore-tm-2    None             <none>             5672/TCP,9711/TCP,9611/TCP,7711/TCP,7611/TCP,9763/TCP,9443/TCP   9d

kubectl get ingress -n wso2

wso2apim-ingress   wso2apim,wso2apim-gw   a1f39d6e13e99...   80, 443   7d

[pubudu538]

Hi Raviada,

You need to have an external IP if you use NodePort.

Based on your comments, your lb backend is working fine. Did you try the following?

1. Update the /etc/hosts entries on your machine with the following.
   <IP of the ELB> wso2apim wso2apim-gw

Thank you!
Pubudu.

[ghost]

Hi there,

I just deployed WSO2 API Manager using pattern-1 to a Kubernetes environment running in an AWS VPC. I use Amazon EFS via NFS for the persistent volume and disabled compression in catalina-server.xml files. I also added the Ingress address to my /etc/hosts file and can now access publisher and store using https://wso2apim/publisher resp. https://wso2apim/store

However - doing a sign-in / log-in to either publisher or store does not last a second request. I other words:

• going to https://wso2apim/store/, clickin on "Sign In", entering credentials and hitting "Sign In" brings me back to the store, but I am not signed in
• going to https://wso2apim/publisher/ redirects to the common login mask (https://wso2apim/publisher/site/pages/login.jag?requestedPage=/publisher/), but after entering credentials and hitting Sign In, I get returned to the same login mask again with no additional error.

I'll assume, that the session information get's lost somewhere in the AWS ELB/K8S Ingress/WSO2 Reverse Proxy communication, but don't know where. Any feedback on this is appreciated.

Kind regards

Christoph

[m0wlheld]

I can answer to my original question: The following addition is required to wso2apim-ingress.yaml (last three lines, the last one adds some security):

  annotations:
    kubernetes.io/ingress.class: "nginx"
    ingress.kubernetes.io/ssl-passthrough: "true"
    ingress.kubernetes.io/affinity: "cookie"
    ingress.kubernetes.io/session-cookie-name: "route"
    ingress.kubernetes.io/session-cookie-hash: "sha1"

Kind regards

Christoph

[chirangaalwis]

@raviada please refer to #432 (comment) regarding the test results of API Manager latest version 3.2.0 Helm charts in AWS EKS.

[chirangaalwis]

Closing this issue as per #79 (comment).

@raviada please feel free to re-open this issue or create new issues if you have any further thoughts, suggestions and concerns.