Skip to content

Files

Latest commit

 

History

History
547 lines (384 loc) · 20 KB

demo.adoc

File metadata and controls

547 lines (384 loc) · 20 KB

Kubernetes to the Cloud with JHipster Demo

Today, I’d like to show you how to use JHipster 7 to create a reactive Java microservices architecture and deploy it to Google Cloud with Kubernetes.

Prerequisites:

Table of Contents

Create a Kubernetes-Ready Microservices Architecture

In this demo, I’ll generate K8s deployment descriptors, use Spring Cloud Config with Git, encrypt your secrets, and make it all work on Google Cloud (GKE to be specific).

  1. Start by cloning the JHipster 7 { Vue, Spring Boot, WebFlux } reactive microservices project from GitHub:

    git clone https://github.com/oktadev/java-microservices-examples.git
cd java-microservices-examples/reactive-jhipster

  2. Install JHipster.

    npm i -g generator-jhipster@7

Generate Kubernetes Deployment Descriptors

  1. Open the reactive-jhipster project in a terminal and start the JHipster Kubernetes sub-generator.

    take k8s
jhipster k8s

  2. Answer the prompts:

    • Type of application: Microservice application

    • Root directory: ../

    • Which applications? <select all>

    • Set up monitoring? No

    • Which applications with clustered databases? select store

    • Admin password for JHipster Registry: <generate one>

    • Kubernetes namespace: demo

    • Docker repository name: <your docker hub username>

    • Command to push Docker image: docker push

    • Enable Istio? No

    • Kubernetes service type? LoadBalancer

    • Use dynamic storage provisioning? Yes

    • Use a specific storage class? <leave empty>

Note
 If you don’t want to publish your images on Docker Hub, leave the Docker repository name blank.

I already showed you how to get everything working with Docker Compose in the previous tutorial. So today, let’s run things locally with Minikube.

Install Minikube to Run Kubernetes Locally

  1. Run minikube start to begin.

    minikube --cpus 8 start
    Caution
    		 If this doesn’t work, use brew install minikube, or see Minikube’s installation instructions.

    This command will start Minikube with 16 GB of RAM and 8 CPUs. Unfortunately, the default, which is 16 GB RAM and two CPUs, did not work for me.

Create Docker Images with Jib

  1. In the {gateway, blog, store } directories, run the following Gradle command (where <image-name> is gateway, store, or blog).

    ./gradlew bootJar -Pprod jib -Djib.to.image=<docker-repo-name>/<image-name>

Register an OIDC App for Auth

  1. Use the Okta CLI and run okta apps create jhipster.

  2. Update k8s/registry-k8s/application-configmap.yml to contain your OIDC settings from the .okta.env file the Okta CLI just created. The Spring Cloud Config server reads from this file and shares the values with the gateway and microservices.

    data:
  application.yml: |-
    ...
    spring:
      security:
        oauth2:
          client:
            provider:
              oidc:
                issuer-uri: https://<your-okta-domain>/oauth2/default
            registration:
              oidc:
                client-id: <client-id>
                client-secret: <client-secret>

    Whhaaattt??? Plain-text secrets in YAML files?! WTF?? I’ll come back to this in a minute.

  3. To configure the JHipster Registry to use OIDC for authentication, modify k8s/registry-k8s/jhipster-registry.yml to enable the oauth2 profile.

    - name: SPRING_PROFILES_ACTIVE
  value: prod,k8s,oauth2

Start Your Spring Boot Microservices with K8s

  1. In the k8s directory, start your engines!

    ./kubectl-apply.sh -f

  2. You can see if everything starts up successfully using kubectl get pods -n demo. Or, even better, use K9s (k9s -n demo).

  3. Port-forward the registry and gateway to see them in a browser.

    kubectl port-forward svc/jhipster-registry -n demo 8761
kubectl port-forward svc/gateway -n demo 8080

  4. Sign in with your Okta credentials at http://localhost:8761 and http://localhost:8080.

[Optional] Test with Cypress

  1. You can also automate testing to ensure that everything works. Set your Okta credentials as environment variables and run end-to-end tests using Cypress (from the gateway directory).

    export CYPRESS_E2E_USERNAME=<your-okta-username>
export CYPRESS_E2E_PASSWORD=<your-okta-password>
npm run e2e

Encrypt Your Secrets with Spring Cloud Config

The JHipster Registry has an encryption mechanism you can use to encrypt your secrets. That way, it’s safe to store them in public repositories.

  1. Add an ENCRYPT_KEY to the environment variables in k8s/registry-k8s/jhipster-registry.yml.

    - name: ENCRYPT_KEY
  value: really-long-string-of-random-charters-that-you-can-keep-safe
    Tip

    You can use JShell to generate a UUID you can use for your encrypt key.

    jshell

UUID.randomUUID()

    You can quit by typing /exit.

  2. Restart your JHipster Registry containers from the k8s directory.

    ./kubectl-apply.sh -f

Encrypt Your OIDC Client Secret

  1. Sign in to http://localhost:8761 and go to Configuration > Encryption.

  2. Copy and paste your client secret from application-configmap.yml (or .okta.env) and click Encrypt.

  3. Then, copy the encrypted value back into application-configmap.yml. Make sure to wrap it in quotes!

  4. Apply these changes and restart all deployments.

    ./kubectl-apply.sh -f
kubectl rollout restart deploy -n demo

  5. Verify everything still works at http://localhost:8080.

Tip
 If you don’t want to restart the Spring Cloud Config server when you update its configuration, see Refresh the Configuration in Your Spring Cloud Config Server.

Change Spring Cloud Config to use Git

You might want to store your app’s configuration externally. That way, you don’t have to redeploy everything to change values. Good news! Spring Cloud Config makes it easy to switch to Git instead of the filesystem to store your configuration.

  1. In k8s/registry-k8s/jhipster-registry.yml, find the following variables:

    - name: SPRING_CLOUD_CONFIG_SERVER_COMPOSITE_0_TYPE
  value: native
- name: SPRING_CLOUD_CONFIG_SERVER_COMPOSITE_0_SEARCH_LOCATIONS
  value: file:./central-config

    Below these values, add a second lookup location.

    - name: SPRING_CLOUD_CONFIG_SERVER_COMPOSITE_1_TYPE
  value: git
- name: SPRING_CLOUD_CONFIG_SERVER_COMPOSITE_1_URI
  value: https://github.com/mraible/reactive-java-ms-config/
- name: SPRING_CLOUD_CONFIG_SERVER_COMPOSITE_1_SEARCH_PATHS
  value: config
- name: SPRING_CLOUD_CONFIG_SERVER_COMPOSITE_1_LABEL
  value: main

  2. Create a GitHub repo that matches the URI, path, and branch you entered.

    In my case, I created reactive-java-ms-config and added a config/application.yml file in the main branch. Then, I added my spring.security.* values to it and removed them from k8s/registry-k8s/application-configmap.yml.

See Spring Cloud Config’s Git Backend docs for more information.

Deploy Spring Boot Microservices to Google Cloud

Giddyup, let’s go to production! 🤠

  1. Stop Minikube:

    minikube stop

    You can also use kubectl commands to switch clusters.

    kubectl config get-contexts
kubectl config use-context XXX
    Tip
    		 The cool kids use kubectx and kubens to set the default context and namespace. You can learn how to install and use them via the kubectx GitHub project.

Create a Container Registry on Google Cloud

  1. Sign up for Google Cloud Platform (GCP), log in, and create a project.

  2. Open a console in your browser or download and install the gcloud CLI if you want to run things locally.

    glcoud auth login
gcloud config set project <project-id>

  3. Enable the Google Kubernetes Engine API and Container Registry:

    gcloud services enable container.googleapis.com containerregistry.googleapis.com

Create a Kubernetes Cluster

  1. Create a cluster for your apps.

    gcloud container clusters create CLUSTER_NAME \
--zone us-central1-a \
--machine-type n1-standard-4 \
--enable-autorepair \
--enable-autoupgrade

  2. Navigate to the gateway directory and run:

    ./gradlew bootJar -Pprod jib -Djib.to.image=gcr.io/<your-project-id>/gateway

  3. Repeat the process for blog and store. You can run these processes in parallel to speed things up.

  4. In your k8s/*/-deployment.yml files, add gcr.io/<your-project-id> as a prefix.

    containers:
  - name: gateway-app
    image: gcr.io/jhipster7/gateway

  5. In the k8s directory, apply all the deployment descriptors to run all your images.

    ./kubectl-apply.sh -f
    Tip

    If you get an error that localhost:8080 was refused, run the following command:

    gcloud container clusters get-credentials <cluster-name> --zone us-central1-a

Access Your Gateway on Google Cloud

  1. Once everything is up and running, get the external IP of your gateway.

    kubectl get svc gateway -n demo

  2. You’ll need to add the external IP address as a valid redirect to your Okta OIDC app. Run okta login, open the returned URL in your browser, and sign in to the Okta Admin Console. Go to the Applications section, find your application, and edit it.

  3. Add the standard JHipster redirect URIs using the IP address. For example, http://34.71.48.244:8080/login/oauth2/code/oidc for the login redirect URI, and http://34.71.48.244:8080 for the logout redirect URI.

  4. Use the following command to set your gateway’s IP address as a variable you can curl.

    EXTERNAL_IP=$(kubectl get svc gateway -ojsonpath="{.status.loadBalancer.ingress[0].ip}" -n demo)
curl $EXTERNAL_IP:8080

  5. Run open http://$EXTERNAL_IP:8080, and you should be able to sign in.

Now that you know things work, let’s integrate better security, starting with HTTPS.

Add HTTPS to Your Reactive Gateway

You should always use HTTPS. It’s one of the easiest ways to secure things, especially with the free certificates offered these days. Ray Tsang’s External Load Balancing docs was a big help in figuring out all these steps.

  1. Create a static IP to assign your TLS (the official name for HTTPS) certificate.

    gcloud compute addresses create gateway-ingress-ip --global

  2. Run the following command to make sure it worked.

    gcloud compute addresses describe gateway-ingress-ip --global --format='value(address)'

  3. Then, create a k8s/ingress.yml file:

    apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: gateway
  annotations:
    kubernetes.io/ingress.global-static-ip-name: "gateway-ingress-ip"
spec:
  rules:
  - http:
      paths:
      - path: /*
        pathType: ImplementationSpecific
        backend:
          service:
            name: gateway
            port:
              number: 8080

  4. Deploy it and make sure it worked.

    kubectl apply -f ingress.yml -n demo

# keep running this command displays an IP address
# (hint: up arrow recalls the last command)
kubectl get ingress gateway -n demo

  5. To use a TLS certificate, you must have a fully qualified domain name and configure it to point to the IP address. If you don’t have a real domain, you can use nip.io.

  6. Set the IP in a variable, as well as the domain.

    EXTERNAL_IP=$(kubectl get ingress gateway -ojsonpath="{.status.loadBalancer.ingress[0].ip}" -n demo)
DOMAIN="${EXTERNAL_IP}.nip.io"

# Prove it works
echo $DOMAIN
curl $DOMAIN

  7. To create a certificate, create a k8s/certificate.yml file.

    cat << EOF > certificate.yml
apiVersion: networking.gke.io/v1
kind: ManagedCertificate
metadata:
  name: gateway-certificate
spec:
  domains:
  # Replace the value with your domain name
  - ${DOMAIN}
EOF

  8. Add the certificate to ingress.yml:

    metadata:
  name: gateway
  annotations:
    kubernetes.io/ingress.global-static-ip-name: "gateway-ingress-ip"
    networking.gke.io/managed-certificates: "gateway-certificate"

  9. Deploy both files:

    kubectl apply -f certificate.yml -f ingress.yml -n demo

  10. Check your certificate’s status until it prints Status: ACTIVE:

    kubectl describe managedcertificate gateway-certificate -n demo | grep Status

Force HTTPS with Spring Security

Spring Security’s WebFlux support makes it easy to redirect to HTTPS. However, if you redirect all HTTPS requests, the Kubernetes health checks will fail because they receive a 302 instead of a 200.

  1. Crack open SecurityConfiguration.java in the gateway project and add the following code to the springSecurityFilterChain() method.

    src/main/java/…​/gateway/config/SecurityConfiguration.java
    http.redirectToHttps(redirect -> redirect
    .httpsRedirectWhen(e -> e.getRequest().getHeaders().containsKey("X-Forwarded-Proto"))
);

  2. Rebuild the Docker image for the gateway project.

    ./gradlew bootJar -Pprod jib -Djib.to.image=gcr.io/<your-project-id>/gateway

  3. Start a rolling restart of gateway instances:

    kubectl rollout restart deployment gateway -n demo

  4. Now you should get a 302 when you access your domain using HTTPie.

    http $DOMAIN

  5. Update your Okta OIDC app to have https://${DOMAIN}/login/oauth2/code/oidc as a valid redirect URI. Add https://${DOMAIN} to the sign-out redirect URIs too.

Encrypt Kubernetes Secrets

Congratulations! Now you have everything running on GKE, using HTTPS! However, you have a lot of plain-text secrets in your K8s YAML files.

"But, wait!" you might say. Doesn’t Kubernetes Secrets solve everything?

In my opinion, no. They’re just unencrypted base64-encoded strings stored in YAML files. You might want to check in the k8s directory you just created.

Having secrets in your source code is a bad idea!

The Current State of Secret Management in Kubernetes

I recently noticed a tweet from Daniel Jacob Bilar that links to a talk from FOSDEM 2021 on the current state of secret management within Kubernetes. It’s an excellent overview of the various options.

Store Secrets in Git with Sealed Secrets and Kubeseal

Bitnami has a Sealed Secrets Apache-licensed open source project. Its README explains how it works.

Problem: "I can manage all my K8s config in git, except Secrets."

Solution: Encrypt your Secret into a SealedSecret, which is safe to store - even to a public repository. The SealedSecret can be decrypted only by the controller running in the target cluster, and nobody else (not even the original author) is able to obtain the original Secret from the SealedSecret.

  1. First, you’ll need to install the Sealed Secrets CRD (Custom Resource Definition).

    kubectl apply -f https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.16.0/controller.yaml

  2. Retrieve the certificate keypair that this controller generates.

    kubectl get secret -n kube-system -l sealedsecrets.bitnami.com/sealed-secrets-key -o yaml

  3. Copy the raw value of tls.crt and decode it.

    echo -n <paste-value-here> | base64 --decode

  4. Put the raw value in a tls.crt file.

  5. Next, install Kubeseal. On macOS, you can use Homebrew. For other platforms, see the release notes.

    brew install kubeseal

The major item you need to encrypt in this example is the ENCRYPT_KEY you used to encrypt the OIDC client secret.

  1. Run the following command to do this, where the value comes from your k8s/registry-k8s/jhipster-registry.yml file.

    kubectl create secret generic encrypt-key \
  --from-literal=ENCRYPT_KEY='your-value-here' \
  --dry-run=client -o yaml > secrets.yml

  2. Next, use kubeseal to convert the secrets to encrypted secrets.

    kubeseal --cert tls.crt --format=yaml -n demo < secrets.yml > sealed-secrets.yml

  3. Remove the original secrets file and deploy your sealed secrets.

    rm secrets.yml
kubectl apply -n demo -f sealed-secrets.yml && kubectl get -n demo sealedsecret encrypt-key

Configure JHipster Registry to use the Sealed Secret

  1. In k8s/registry-k8s/jhipster-registry.yml, change the ENCRYPT_KEY to use your new secret.

    - name: ENCRYPT_KEY
  valueFrom:
    secretKeyRef:
      name: encrypt-key
      key: ENCRYPT_KEY
    Tip
    		 You should be able to encrypt other secrets, like your database passwords, using a similar technique.

  2. Redeploy JHipster Registry and restart all your deployments.

    ./kubectl-apply.sh -f
kubectl rollout restart deployment -n demo

  3. You can use port-forwarding to see the JHipster Registry locally.

    kubectl port-forward svc/jhipster-registry -n demo 8761

Use Spring Vault for External Secrets

Using an external key management solution like HashiCorp Vault is also recommended. The JHipster Registry will have Vault support in its next release.

In the meantime, I recommend reading Secure Secrets With Spring Cloud Config and Vault.

Scale Your Reactive Java Microservices

You can scale your instances using the kubectl scale command.

kubectl scale deployments/store --replicas=2 -n demo

Scaling will work just fine for the microservice apps because they’re set up as OAuth 2.0 resource servers and are therefore stateless.

However, the gateway uses Spring Security’s OIDC login feature and stores the access tokens in the session. So if you scale it, sessions won’t be shared. Single sign-on should still work; you’ll just have to do the OAuth dance to get tokens if you hit a different instance.

To synchronize sessions, you can use Spring Session and Redis with JHipster.

Caution

If you leave everything running on Google Cloud, you will be charged for usage. Therefore, I recommend removing your cluster or deleting your namespace (kubectl delete ns demo) to reduce your cost.

gcloud container clusters delete <cluster-name> --zone=us-central1-a

Monitor Your Kubernetes Cluster with K9s

Using kubectl to monitor your Kubernetes cluster can get tiresome. That’s where K9s can be helpful. It provides a terminal UI to interact with your Kubernetes clusters. K9s was created by my good friend Fernand Galiana. He’s also created a commercial version called K9sAlpha.

To install it on macOS, run brew install k9s. Then run k9s -n demo to start it. You can navigate to your pods, select them with Return, and navigate back up with Esc.

Learn More About Kubernetes, Spring Boot, and JHipster