Skip to content

chore(nginx-gw): update docs to make it simpler to deploy #858

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Mar 1, 2025
Merged

chore(nginx-gw): update docs to make it simpler to deploy #858

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
36 changes: 36 additions & 0 deletions bin/install-nginx-gateway.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
#!/bin/bash
# shellcheck disable=SC2124,SC2145,SC2294

GLOBAL_OVERRIDES_DIR="/etc/genestack/helm-configs/global_overrides"
SERVICE_CONFIG_DIR="/etc/genestack/helm-configs/nginx-gateway-fabric"
BASE_OVERRIDES="/opt/genestack/base-helm-configs/nginx-gateway-fabric/helm-overrides.yaml"
NGINX_VERSION="1.4.0"
HELM_CMD="helm upgrade --install nginx-gateway-fabric oci://ghcr.io/nginx/charts/nginx-gateway-fabric \
--create-namespace \
--namespace=nginx-gateway \
--post-renderer /etc/genestack/kustomize/kustomize.sh \
--post-renderer-args gateway/overlay \
--version ${NGINX_VERSION}"

HELM_CMD+=" -f ${BASE_OVERRIDES}"

for dir in "$GLOBAL_OVERRIDES_DIR" "$SERVICE_CONFIG_DIR"; do
if compgen -G "${dir}/*.yaml" > /dev/null; then
for yaml_file in "${dir}"/*.yaml; do
# Avoid re-adding the base override file if present in the service directory
if [ "${yaml_file}" != "${BASE_OVERRIDES}" ]; then
HELM_CMD+=" -f ${yaml_file}"
fi
done
fi
done

HELM_CMD+=" $@"

kubectl kustomize "https://github.com/nginxinc/nginx-gateway-fabric/config/crd/gateway-api/standard?ref=v${NGINX_VERSION}" | kubectl apply -f -

kubectl apply -f /opt/genestack/manifests/nginx-gateway/nginx-gateway-namespace.yaml

echo "Executing Helm command:"
echo "${HELM_CMD}"
eval "${HELM_CMD}"
72 changes: 72 additions & 0 deletions bin/setup-nginx-gateway.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,72 @@
#!/bin/bash
# shellcheck disable=SC2045,SC2124,SC2145,SC2164,SC2236,SC2294

if [ -z "${ACME_EMAIL}" ]; then
read -rp "Enter a valid email address for use with ACME, press enter to skip: " ACME_EMAIL
fi

if [ -z "${GATEWAY_DOMAIN}" ]; then
echo "The domain name for the gateway is required, if you do not have a domain name press enter to use the default"
read -rp "Enter the domain name for the gateway [cluster.local]: " GATEWAY_DOMAIN
export GATEWAY_DOMAIN=${GATEWAY_DOMAIN:-cluster.local}
fi

if [ -z "${GATEWAY_DOMAIN}" ]; then
echo "Gateway domain is required"
exit 1
fi

kubectl kustomize /etc/genestack/kustomize/gateway/nginx-gateway-fabric | kubectl apply -f -

echo "Waiting for the gateway to be programmed"
kubectl -n nginx-gateway wait --timeout=5m gateways.gateway.networking.k8s.io flex-gateway --for=condition=Programmed

if [ ! -z "${ACME_EMAIL}" ]; then
cat <<EOF | kubectl apply -f -
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: letsencrypt-prod
spec:
acme:
server: https://acme-v02.api.letsencrypt.org/directory
email: ${ACME_EMAIL}
privateKeySecretRef:
name: letsencrypt-prod
solvers:
- http01:
gatewayHTTPRoute:
parentRefs:
- group: gateway.networking.k8s.io
kind: Gateway
name: flex-gateway
namespace: nginx-gateway
EOF

kubectl patch --namespace nginx-gateway \
--type merge \
--patch-file /etc/genestack/gateway-api/gateway-letsencrypt.yaml \
gateway flex-gateway

kubectl rollout restart deployment cert-manager --namespace cert-manager
fi

sudo mkdir -p /etc/genestack/gateway-api/routes
for route in $(ls -1 /opt/genestack/etc/gateway-api/routes); do
sed "s/your.domain.tld/${GATEWAY_DOMAIN}/g" "/opt/genestack/etc/gateway-api/routes/${route}" > "/tmp/${route}"
sudo mv -v "/tmp/${route}" "/etc/genestack/gateway-api/routes/${route}"
done

kubectl apply -f /etc/genestack/gateway-api/routes

sudo mkdir -p /etc/genestack/gateway-api/listeners
for listener in $(ls -1 /opt/genestack/etc/gateway-api/listeners); do
sed "s/your.domain.tld/${GATEWAY_DOMAIN}/g" "/opt/genestack/etc/gateway-api/listeners/${listener}" > "/tmp/${listener}"
sudo mv -v "/tmp/${listener}" "/etc/genestack/gateway-api/listeners/${listener}"
done

kubectl patch -n nginx-gateway gateway flex-gateway \
--type='json' \
--patch="$(jq -s 'flatten | .' /etc/genestack/gateway-api/listeners/*)"

echo "Setup Complete"
212 changes: 21 additions & 191 deletions docs/infrastructure-nginx-gateway-api.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,220 +8,50 @@ hide:
The [NGINX Gateway Fabric](https://github.com/nginxinc/nginx-gateway-fabric) is an open-source project that provides an
implementation of the Gateway API using NGINX as the data plane.

## Install the Gateway API Resource from Kubernetes
## Installation

=== "Stable _(Recommended)_"
Run the helm command to install NGINX Gateway.

``` shell
kubectl kustomize "https://github.com/nginxinc/nginx-gateway-fabric/config/crd/gateway-api/standard?ref=v1.4.0" | kubectl apply -f -
```

=== "Experimental"

The experimental version of the Gateway API is available in the `v1.6.1` checkout. Use with caution.
??? example "Run the NGINX Gateway deployment Script `/opt/genestack/bin/install-nginx-gateway.sh`"

``` shell
kubectl kustomize "https://github.com/nginx/nginx-gateway-fabric/config/crd/gateway-api/experimental?ref=v1.6.1" | kubectl apply -f -
--8<-- "bin/install-nginx-gateway.sh"
```

## Install the NGINX Gateway Fabric controller

The NGINX Gateway Fabric controller is a Kubernetes controller that manages the Gateway API resources.

### Create the Namespace

``` shell
kubectl apply -f /opt/genestack/manifests/nginx-gateway/nginx-gateway-namespace.yaml
```

!!! tip

If attempting to perform an **upgrade** of an existing Gateway API deployment, note that the Helm install does not automatically upgrade the CRDs for
this resource. To upgrade them, refer to the process outlined by the
[Nginx upgrade documentation](https://docs.nginx.com/nginx-gateway-fabric/installation/installing-ngf/helm/#upgrade-nginx-gateway-fabric-crds). You
can safely ignore this note for new installations.

=== "Stable _(Recommended)_"

Edit the file `/etc/genestack/helm-configs/nginx-gateway-fabric/helm-overrides.yaml`.

!!! example "Create an empty override file"
The install script will deploy NGINX Gateway to the `nginx-gateway` namespace via Helm.

If no overrides are needed, create an empty file.
## Setup

``` shell
echo "---" | tee /etc/genestack/helm-configs/nginx-gateway-fabric/helm-overrides.yaml
```
??? example "Run the NGINX Gateway setup Script `/opt/genestack/bin/setup-nginx-gateway.sh`"

``` shell
pushd /opt/genestack/submodules/nginx-gateway-fabric/charts || exit 1
helm upgrade --install nginx-gateway-fabric ./nginx-gateway-fabric \
--namespace=nginx-gateway \
--create-namespace \
-f /opt/genestack/base-helm-configs/nginx-gateway-fabric/helm-overrides.yaml \
-f /etc/genestack/helm-configs/nginx-gateway-fabric/helm-overrides.yaml \
--post-renderer /etc/genestack/kustomize/kustomize.sh \
--post-renderer-args gateway/overlay
popd || exit 1
--8<-- "bin/setup-nginx-gateway.sh"
```

=== "Experimental"

The experimental version of the Gateway API is available in the `v1.6.1` checkout. Use with caution.

Update the submodule with the experimental version of the Gateway API.

Edit the file `/etc/genestack/helm-configs/nginx-gateway-fabric/helm-overrides.yaml`.

!!! example "Create the experimental override file"

``` yaml
---
nginxGateway:
replicaCount: 3
gwAPIExperimentalFeatures:
enable: true
service:
## The externalTrafficPolicy of the service. The value Local preserves the client source IP.
externalTrafficPolicy: Cluster
## The annotations of the NGINX Gateway Fabric service.
annotations:
"metallb.universe.tf/address-pool": "gateway-api-external"
"metallb.universe.tf/allow-shared-ip": "openstack-external-svc"
```

Run the helm command to install the experimental version of the Gateway API.

``` shell
helm upgrade --install nginx-gateway-fabric oci://ghcr.io/nginx/charts/nginx-gateway-fabric \
--create-namespace \
--namespace=nginx-gateway \
-f /etc/genestack/helm-configs/nginx-gateway-fabric/helm-overrides.yaml \
--post-renderer /etc/genestack/kustomize/kustomize.sh \
--post-renderer-args gateway/overlay \
--version 1.6.1
```

Once deployed ensure a system rollout has been completed for Cert Manager.

``` shell
kubectl rollout restart deployment cert-manager --namespace cert-manager
```

## Create the shared gateway resource

``` shell
kubectl kustomize /etc/genestack/kustomize/gateway/nginx-gateway-fabric | kubectl apply -f -
```

## Deploy with Let's Encrypt Certificates

By default, certificates are issued by an instance of the selfsigned-cluster-issuer. This section focuses on replacing that with a
Let's Encrypt issuer to ensure valid certificates are deployed in our cluster.

[![asciicast](https://asciinema.org/a/h7npXnDjkSpn3uQtuQwWG9zju.svg)](https://asciinema.org/a/h7npXnDjkSpn3uQtuQwWG9zju)

### Apply the Let's Encrypt Cluster Issuer

Before we can have Cert Manager start coordinating Let's Encrypt certificate
requests for us, we need to add an ACME issuer with a valid, monitored
email (for expiration reminders and other important ACME related information).

``` yaml
read -p "Enter a valid email address for use with ACME: " ACME_EMAIL; \
cat <<EOF | kubectl apply -f -
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: letsencrypt-prod
spec:
acme:
server: https://acme-v02.api.letsencrypt.org/directory
email: ${ACME_EMAIL}
privateKeySecretRef:
name: letsencrypt-prod
solvers:
- http01:
gatewayHTTPRoute:
parentRefs:
- group: gateway.networking.k8s.io
kind: Gateway
name: flex-gateway
namespace: nginx-gateway
EOF
```

!!! note
The setup script will ask the following questions:

It is also possible to use cert-manager to create [self-signed CA for Gateway API](https://docs.rackspacecloud.com/gateway-api-ca-issuer/)
* Enter a valid email address for use with ACME, press enter to skip"
* Enter the domain name for the gateway"

## Patch Gateway with valid listeners
These values will be used to generate a certificate for the gateway and set the routes used within the flex-gateway,
typically for OpenStack. This script can also be fully automated by providing the required values as arguments.

By default, a generic Gateway is created using a hostname of `*.cluster.local`. To add specific hostnames/listeners to the gateway, you can either
create a patch or update the gateway YAML to include your specific hostnames and then apply the patch/update. Each listener must have a
unique name.

??? abstract "An example patch file you can modify to include your own domain name can be found at `/opt/genestack/etc/gateway-api/listeners/gateway-api/http-wildcard-listener.json`"

``` json
--8<-- "etc/gateway-api/listeners/http-wildcard-listener.json"
```

!!! example "Example modifying the Gateway listener patches"
!!! example "Run the NGINX Gateway setup Script with arguments"

``` shell
mkdir -p /etc/genestack/gateway-api/listeners
for listener in $(ls -1 /opt/genestack/etc/gateway-api/listeners); do
sed 's/your.domain.tld/<YOUR_DOMAIN>/g' /opt/genestack/etc/gateway-api/listeners/$listener > /etc/genestack/gateway-api/listeners/$listener
done
ACME_EMAIL="username@your.domain.tld" GATEWAY_DOMAIN="your.domain.tld" /opt/genestack/bin/setup-nginx-gateway.sh
```

``` shell
kubectl patch -n nginx-gateway gateway flex-gateway \
--type='json' \
--patch="$(jq -s 'flatten | .' /etc/genestack/gateway-api/listeners/*)"
```

## Apply Related Gateway routes
## Validation

Another example with most of the OpenStack services is located at `/opt/genestack/etc/gateway-api/routes/http-wildcard-listener.yaml`. Similarly, you must modify
and apply them as shown below, or apply your own.

??? abstract "Example routes file"

``` yaml
--8<-- "etc/gateway-api/routes/http-wildcard-listener.yaml"
```

All routes can be found at `/etc/genestack/gateway-api/routes`.

!!! example "Example modifying all available Gateway routes with `your.domain.tld`"

``` shell
mkdir -p /etc/genestack/gateway-api/routes
for route in $(ls -1 /opt/genestack/etc/gateway-api/routes); do
sed 's/your.domain.tld/<YOUR_DOMAIN>/g' /opt/genestack/etc/gateway-api/routes/$route > /etc/genestack/gateway-api/routes/$route
done
```
At this point, flex-gateway has a listener pointed to the port 80 matching *.your.domain.tld hostname. The
HTTPRoute resource configures routes for this gateway. Here, we match all path and simply pass any request
from the matching hostname to kube-prometheus-stack-prometheus backend service.

``` shell
kubectl apply -f /etc/genestack/gateway-api/routes
kubectl -n openstack get httproute
```

## Patch Gateway with Let's Encrypt Cluster Issuer

??? abstract "Example patch to enable LetsEncrypt `/etc/genestack/gateway-api/gateway-letsencrypt.yaml`"

``` yaml
--8<-- "etc/gateway-api/gateway-letsencrypt.yaml"
```

``` shell
kubectl patch --namespace nginx-gateway \
--type merge \
--patch-file /etc/genestack/gateway-api/gateway-letsencrypt.yaml \
gateway flex-gateway
kubectl -n nginx-gateway get gateways.gateway.networking.k8s.io flex-gateway
```

At this point, flex-gateway has a listener pointed to the port 80 matching *.your.domain.tld hostname. The HTTPRoute resource configures routes
for this gateway. Here, we match all path and simply pass any request from the matching hostname to kube-prometheus-stack-prometheus backend service.
Loading