Setting up Nexus with Nginx Reverse proxy
++Nexus with Nginx Reverse proxy++setting up nexus with Nginx Reverse proxy++Use Docker with Reverses proxy++Nexus & Docker with Nginx proxy++
A must Read document:
- Install & Configure Nginx Reverse Proxy to access Nexus. You could use the Ansible playbook to accomplish it.
- The Playbook is available at https://github.com/q-uest/Ansible_playbooks.git. And, the Readme.md is availble @https://github.com/q- uest/Ansible_playbooks/tree/patch-1.
- The playbook generates the required certificate, and the NGINX configuration file to terminate the SSL session and forward traffic in plain HTTP to Nexus.
The config file is available at the below path and looks like below:
root@nginx-revproxy2:/etc/nginx/sites-enabled# cat nexus
server {
listen 443 ssl;
listen [::]:443 ssl;
ssl_certificate /root/app.crt;
ssl_certificate_key /root/app_key.pem;
ssl_protocols TLSv1.3;
ssl_prefer_server_ciphers on;
ssl_ciphers EECDH+AESGCM:EDH+AESGCM;
ssl_ecdh_curve secp384r1;
ssl_session_timeout 10m;
ssl_session_cache shared:SSL:10m;
ssl_session_tickets off;
ssl_stapling on;
ssl_stapling_verify on;
resolver 8.8.8.8 8.8.4.4 valid=300s;
resolver_timeout 5s;
# Disable strict transport security for now. You can uncomment the following
# line if you understand the implications.
#add_header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload";
add_header X-Frame-Options DENY;
add_header X-Content-Type-Options nosniff;
add_header X-XSS-Protection "1; mode=block";
server_name nginx-revprxy-4-nexus.com;
access_log /var/log/nginx/nexus.access.log;
error_log /var/log/nginx/nexus.error.log;
# For Nexus Web URL:
location /
{
proxy_pass http://10.128.0.18:8081;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_buffering off;
proxy_request_buffering off;
keepalive_timeout 5 5;
tcp_nodelay on;
proxy_connect_timeout 90;
proxy_send_timeout 120;
proxy_read_timeout 300;
client_max_body_size 10m;
}
# For Docker Connections (this is used by Docker client by default):
location /v2/
{
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto "https";
# "client_max_body_size" allows large uploads of files - refer to nginx documentation
client_max_body_size 2048m;
proxy_pass http://10.128.0.18:8082;
}
}
server {
listen 80;
listen [::]:80;
server_name 34.67.194.116;
return 302 https://$server_name$request_uri;
}
-
The Nginx reverse proxy is to handle SSL certificates and termination.
-
There are 2 server blocks in the above configuration for 2 different purposes.
The second server block makes NGINX listen to HTTP requests on port 80 and redirects it to HTTPs.
The first server block declares a server on port 443 to listen to HTTPs traffic, it points to the certificate and key that the TLS endpoint must use, and it configures two URL locations:
-
The root ‘/’ corresponds to the Nexus3 UI which listens on TCP port 8081.
-
The ‘/v2/’ corresponds to the docker registry path and forwards traffic that comes from the docker command line client (docker login, pull or push) to the registry on port 8082. This port needs to be configured in the Nexus repo.
Note that 10.128.0.18 is the [private] IP address of the Nexus host.
- Access the Nexus Web User Interface
To access the web page of Nexus you can now open your browser and enter one of the following URLs:
-
HTTP: [http://34.67.194.116] (replace with your IP address or DNS name) this will redirect you to [https://34.67.194.116] per the NGINX configuration done above.
-
HTTPs: [https://34.67.194.116]
- Configure the Docker Client Daemon to Trust the Certificate
Because we have configured HTTPs and we used a self-signed certificate, the docker client will not trust the certificate if we don’t configure it to do so.
For the docker engine to trust the self-signed certificate we used for NGINX, we must place a copy of this certificate in a special directory on the client host. On the Docker clients (including all Kubernetes hosts), create a directory by the IP address of the Reveres proxy under /etc/docker/certs.d/ and copy the .crt (of the application, no certificate Authority file is required to be copied over) file generated earlier on the reverse proxy host.
This is how Docker client looks on Jenkins host (@/etc/docker/certs.d/35.202.196.238) with the certificates:
root@jenkins:/etc/docker/certs.d/34.67.194.116# ls -ltr
total 4
-rwxr-xr-x 1 root root 1712 May 29 16:55 app.crt
The same needs to be done on all the Docker/Kubernetes hosts.
- Logging into Docker:
Provide the IP address of the Nginx host.
docker login -u admin -p password 34.67.194.116
The command adds /v2/ to the IP address automatically above and uses 8082 port as configured in Nginx above.
Try the below to see whether it works
docker pull 34.67.194.116/appointme-admin-api:129
Here, the IP address of the reverse proxy prefixed with the image works like an alias (with https) to the Nexus host. Behind the scenes, the configuration done at Nginx end does the redirection to the Nexus host.
-
DO NOT need add the Reverse proxy's IP address/domain to "insecure-registries" in "/etc/docker/daemon.json".
-
Just Create Kubernetes Secret
In order to pull images through Kubernetes, create a kubernetes secret from the Dockerr's config.json as below. Otherwise, you may end up getting errors like ""unauthorized: authentication required" :
kubectl create secret generic regcred --from-file=.dockerconfigjson=/root/.docker/config.json --type=kubernetes.io/dockerconfigjson
Note:
You would see an entry having Nginx host's IP address @/root/.docker/config.json.
Kubernetes version 1.19+ and COS/Ubuntu images based on containerd for GKE nodes. Before the 1.19 version Kubernetes used to use Docker for building images, but now it uses containerd. More details could be found in the official Google Cloud documentation.
The following commands to be executed on both master & slave nodes to copy/update the certificate:
sudo mkdir /usr/local/share/ca-certificates/extra
sudo cp app.crt /usr/local/share/ca-certificates/extra/.
sudo update-ca-certificates
ls -l /etc/ssl/certs/app*
nsenter --target 1 --mount systemctl restart containerd
Note:
All the steps including the ones pertaining to "docker" also should have been executed without any fail on ALL the nodes. It is also important to create and provide the K8s Secret object (created from the docker's config.json above) while creating deployments/pods. Otherwise, it will throw errors like the below:
Failed to pull image "34.125.112.64/appointme-admin-api:34": rpc error: code = Unknown desc = failed to pull and unpack image "34.125.112.64/appointme-admin-api:34": failed to resolve reference "34.125.112.64/appointme-admin-api:34": pulling from host 34.125.112.64 failed with status code [manifests 34]: 401 Unauthorized
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deployment
spec:
selector:
matchLabels:
app: nginx
replicas: 2 # tells deployment to run 2 pods matching the template
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: "34.125.87.205/appointme-admin-api:nexus-reg"
ports:
- containerPort: 80
imagePullSecrets:
- name: regcred