[Question] Does this plugin support AWS EKS?

[carlosjgp]

Due to the special configuration required to use EKS I'm not able to figure out how to configure this plugin properly

It might not be compatible at all...?

[sheroy]

Hi @carlosjgp, the plugin works just fine with EKS. We may need additional documentation for setting up with EKS. While we update our documentation, I'm putting a list of steps together that I'll share with you soon. But if you have specific questions, I'm happy to answer them as well.

[arvindsv]

@carlosjgp Though the plugin works, the mechanism to get the token isn't easy and the token can expire. So, the thought is that this needs a special authenticator to make it straightforward. Something which understands EKS more natively.

Do you have time to answer a couple of questions and maybe sometime over the next week or so, talk to us if necessary? The questions I have now are:

• Are you using the helm chart to deploy the GoCD server?
• Or, do you have a GoCD server somewhere (maybe EC2) and you're trying using EKS for the agents only?
• Would you need support for multiple clusters or do you plan to use this with only one cluster?

Another thought is: "If a GoCD server is already running inside EKS, to launch more agent pods, it shouldn't need the aws-iam-authenticator". It's something we're looking into. Maybe it is wrong.

Thank you!

[carlosjgp]

Hi @sheroy and @arvindsv ,

answering to your questions...

• Are you using the helm chart to deploy the GoCD server?
• Or, do you have a GoCD server somewhere (maybe EC2) and you're trying using EKS for the agents only?

• we are currently running GoCD server on an EC2 instance but we are on the process to migrate it to Kubernetes using the Helm chart.
• We only use one cluster to run our CD infrastructure

Another thought is: "If a GoCD server is already running inside EKS, to launch more agent pods, it shouldn't need the aws-iam-authenticator". It's something we're looking into. Maybe it is wrong.

In theory, if the server is inside the cluster is only requires to be deployed with a kubernetes service account and use a K8s Java client pointing to the internal K8s API endpoint... normally kubernetes.default(.svc.cluster.local)

I would happily help to implement this if you want

[sheroy]

Hi @carlosjgp,

Sorry about the delayed response. I tested the Helm chart with multiple EKS ingress options today and here are my findings.

1. The Helm chart creates a service account and pre-configures the Kubernetes elastic agent configuration with the service account token. This is very similar to what you described.

2. On EKS, the default ingress that is installed by the Helm chart will not work. There is no associated ingress controller on EKS that creates an ingress resources.

a. To get an ingress resource on EKS, I first disabled the ingress resource that is created with the Helm chart (helm install stable/gocd --set server.ingress.enabled=false).

b. I then created an AWS ALB ingress controller and an associated ingress resource. I found these instructions useful - https://aws.amazon.com/blogs/opensource/kubernetes-ingress-aws-alb-ingress-controller

c. I configured the gocd nodeport service as a backend in the ingress resource configuration.

This allowed me to connect to the GoCD instance on EKS.

3. Alternatively, you could set the service type for the GoCD service to LoadBalancer and that will allow you to connect to the service via an ELB. The issue here is that you'll have a specific ELB instance created for the GoCD service rather than connecting the service to a cluster wide ELB.

So, the general recommendation I have is, on EKS, don't install the ingress resources that ships with the GoCD helm chart. Instead, create an ingress resource based on the ingress controller you're using on your EKS cluster and connect it to the GoCD nodeport service.

Let me know if this works for you or if you'd like to discuss this further.

cc: @arvindsv

[varshavaradarajan]

@sheroy - can you please post the ingress resource that you created (along with the ALB) after disabling the ingress that comes with the helm chart? I wanted to know what the difference is and why the ingress.yaml that's packaged with the gocd helm chart doesn't work.

[sheroy]

Ingress definition:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: "gocdtest-ingress"
  namespace: "gocdtest"
  annotations:
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/scheme: internet-facing
  labels:
    app: gocd
spec:
  rules:
    - http:
        paths:
          - path: /*
            backend:
              serviceName: "gocdtest-server"
              servicePort: 8153

@varshavaradarajan the main difference I see is the ALB specific annotations.

[varshavaradarajan]

@sheroy - you can provide the ingress annotations in values.yaml for the gocd helm chart - https://github.com/helm/charts/blob/master/stable/gocd/templates/ingress.yaml#L14

[varshavaradarajan]

I think there is some setup involved wrt ingress controller depending on the platform on which the k8s cluster is hosted. For minikube, its minikube addons enable ingress, GCP provides the the GCE ingress controller by default. After setting up the cluster on EKS, perhaps enabling ALB or traefik and then installing the gocd helm chart is enough?

But one of the questions was will just having the helm chart installed on EKS enough to bring up elastic agents every time a gocd job is scheduled? Or will it require this k8s plugin to do something more to bring up elastic agents? That is, did you see jobs being successfully assigned elastic agents?

Another thing to take care of was the plugin bringing up an agent in the cluster and connecting to a remote gocd server, according to #81 (comment) .

[sheroy]

I think there is some setup involved wrt ingress controller depending on the platform on which the k8s cluster is hosted.

Yes, that's why I think the helm chart should be limited to setting up a nodeport service that can then be wired up to an ingress resource based on the target platform, rather than create a new ingress just for GoCD.

But one of the questions was will just having the helm chart installed on EKS enough to bring up elastic agents every time a gocd job is scheduled? Or will it require this k8s plugin to do something more to bring up elastic agents? That is, did you see jobs being successfully assigned elastic agents?

I'm not sure if I understand your question. If you mean does the elastic agent work on EKS as intended, yes it does.

Another thing to take care of was the plugin bringing up an agent in the cluster and connecting to a remote gocd server, according to #81 (comment)

I haven't tested this scenario specifically. I can test it tomorrow if I get some time. I thought @carlosjgp was migrating a GoCD server (and agents based on the k8s elastic agent) over to EKS. I might have misunderstood.

[sheroy]

@sheroy - you can provide the ingress annotations in values.yaml for the gocd helm chart - https://github.com/helm/charts/blob/master/stable/gocd/templates/ingress.yaml#L14

Hmm.. this should work, I can give it a shot.

[sheroy]

Updated steps based on @varshavaradarajan's comment.

1. Setup AWS ALB ingress controller per the instructions here - https://aws.amazon.com/blogs/opensource/kubernetes-ingress-aws-alb-ingress-controller/

2. Update the following annotations in values.yaml

     server:
       ingress:
         annotations:
           kubernetes.io/ingress.class: alb
           alb.ingress.kubernetes.io/scheme: internet-facing

Installing the helm chart now will bring up a GoCD server with the elastic agent pre-configured with a service account token, and an ingress resource with a GoCD service backend.

cc: @carlosjgp @arvindsv

[arvindsv]

@sheroy So, does that mean that all that needs to be done after this is to create the ALB ingress controller and everything works fine?

[sheroy]

@arvindsv yes, and the ALB ingress controller is created in step 1 above. So you need the ALB ingress controller installed, provide extra annotations while installing the Helm chart and you'll be good to go.

[carlosjgp]

Hi @sheroy,

thanks for all your hard work and investigations.
I have no problem using the default ingress options on the chart. Just as you described I have to install an Ingress controller first, in my case Nginx.

Now that you have mentioned the service account creation I have remembered how to get the token but the main challenge is to find the Certificate value to configure the plugin.

Can you point me to the documentation about to get his field for EKS? I've checked my '~/.kube/config' file but due to the especial configuration for EKS required to contact the AWS K8s cluster the certificate is not there anymore AWS uses my AWS CLI credentials to login

[varshavaradarajan]

@carlosjgp - once the helm chart is installed, you get a preconfigured k8s elastic agent plugin settings and profile, which will fill in values for security token and certificate. However, if you need to get the security token and cert for configuring the plugin settings again, you can use the below commands:

secret_name=$(kubectl get serviceaccount <service_account_name> --namespace=<namespace>  -o jsonpath="{.secrets[0].name}")
// for token
kubectl get secret $secret_name --namespace=<namespace> -o jsonpath="{.data['token']}" | base64 --decode 

// for cert

kubectl get secret $secret_name --namespace=<namespace>  -o jsonpath="{.data['ca\.crt']}" | base64 --decode

Make sure that the service account name that you're providing above has the right pod related permissions as specified in https://github.com/helm/charts/tree/master/stable/gocd#rbac-and-service-accounts. This is no different for EKS. Accessing kubectl or the kubernetes API will require AWS credentials from your commandline. But as @sheroy mentioned, once the helm chart was installed on EKS, it wasn't necessary to do any additional setup.

This issue can be closed, @sheroy

[sheroy]

@carlosjgp Sorry I missed your last comment. Did @varshavaradarajan's response give you what you need? I'll wait for a couple of days to hear back from you and close this issue.

\cc: @arvindsv