[NMS] Unable to communicate with magma controller: Not Found

[BioZrod]

NMS has been working without issues and suddenly today I got the following screen with error: "Unable to communicate with magma controller: Not Found"

I checked the pods and all have been running with kubectl -n orc8r get pods

NAME                                             READY   STATUS    RESTARTS   AGE
nms-magmalte-7b88f9dbc6-5l5m5                    1/1     Running   0          7d15h
nms-nginx-proxy-6686676d79-9dbzg                 1/1     Running   0          6d2h
orc8r-accessd-8677db8576-h2jqq                   1/1     Running   0          7d15h
orc8r-accessd-8677db8576-qms8g                   1/1     Running   0          7d15h
orc8r-alertmanager-84d79f774b-2qk4h              1/1     Running   0          365d
orc8r-alertmanager-configurer-6c55bf95cb-jp7cj   1/1     Running   0          7d15h
orc8r-analytics-5b5465885d-9z8d4                 1/1     Running   0          7d15h
orc8r-analytics-5b5465885d-q9xvl                 1/1     Running   0          7d15h
orc8r-bootstrapper-7b46dc8f7f-2rr6q              1/1     Running   0          7d15h
orc8r-bootstrapper-7b46dc8f7f-v95v5              1/1     Running   0          7d15h
orc8r-certifier-5f74677c48-4whh7                 1/1     Running   0          7d15h
orc8r-certifier-5f74677c48-dltw7                 1/1     Running   0          7d15h
orc8r-configurator-86c77bc786-b4t9f              1/1     Running   0          7d15h
orc8r-configurator-86c77bc786-qw8pt              1/1     Running   0          7d15h
orc8r-ctraced-66c45bb8c8-dqtlc                   1/1     Running   0          7d15h
orc8r-ctraced-66c45bb8c8-pscgj                   1/1     Running   0          7d15h
orc8r-device-565d6454f7-lf9gt                    1/1     Running   0          7d15h
orc8r-device-565d6454f7-vnzzn                    1/1     Running   0          7d15h
orc8r-directoryd-775cbc48fd-2pxzj                1/1     Running   0          7d15h
orc8r-directoryd-775cbc48fd-95bl6                1/1     Running   0          7d15h
orc8r-dispatcher-6d6f98c545-ttlj4                1/1     Running   0          7d15h
orc8r-dispatcher-6d6f98c545-vxrts                1/1     Running   0          7d15h
orc8r-eventd-68d7975f5b-bcpqg                    1/1     Running   0          7d15h
orc8r-eventd-68d7975f5b-fhk54                    1/1     Running   0          7d15h
orc8r-ha-875884b48-7cww4                         1/1     Running   0          7d15h
orc8r-ha-875884b48-bzjv9                         1/1     Running   0          7d15h
orc8r-lte-5dbb99b9f-9c5x6                        1/1     Running   0          7d15h
orc8r-lte-5dbb99b9f-m986g                        1/1     Running   0          7d15h
orc8r-metricsd-5474b55fff-jthh4                  1/1     Running   0          7d15h
orc8r-metricsd-5474b55fff-tlhnf                  1/1     Running   0          7d15h
orc8r-nginx-7fbfbbd8c4-7xvgc                     1/1     Running   0          7d15h
orc8r-nginx-7fbfbbd8c4-hln59                     1/1     Running   0          7d15h
orc8r-obsidian-779d49cbc7-9gkrg                  1/1     Running   0          7d15h
orc8r-obsidian-779d49cbc7-v5s8s                  1/1     Running   0          7d15h
orc8r-orchestrator-6b56f7d8b4-c96zn              1/1     Running   0          7d15h
orc8r-orchestrator-6b56f7d8b4-r7njh              1/1     Running   0          7d15h
orc8r-policydb-6d774b5b4b-7xvph                  1/1     Running   0          7d15h
orc8r-policydb-6d774b5b4b-snhc4                  1/1     Running   0          7d15h
orc8r-prometheus-7fc468d458-ftfmd                1/1     Running   6          7d15h
orc8r-prometheus-cache-5c8d9d7b8-njksm           1/1     Running   0          7d15h
orc8r-prometheus-configurer-546d98769f-nwz2b     1/1     Running   0          7d15h
orc8r-service-registry-68c9d79c86-vg75g          1/1     Running   0          7d15h
orc8r-service-registry-68c9d79c86-zrdmp          1/1     Running   0          7d15h
orc8r-smsd-77495d8f74-kqjtx                      1/1     Running   0          7d15h
orc8r-smsd-77495d8f74-mqs76                      1/1     Running   0          7d15h
orc8r-state-789885d989-5lxjf                     1/1     Running   0          7d15h
orc8r-state-789885d989-wkcdb                     1/1     Running   0          7d15h
orc8r-streamer-57d9bf5cc5-2vjsr                  1/1     Running   0          7d15h
orc8r-streamer-57d9bf5cc5-qz4xh                  1/1     Running   0          7d15h
orc8r-subscriberdb-7d9d79499d-gz4c8              1/1     Running   0          7d15h
orc8r-subscriberdb-7d9d79499d-lt8hk              1/1     Running   0          7d15h
orc8r-tenants-69bc77dc9-2kjrx                    1/1     Running   0          7d15h
orc8r-tenants-69bc77dc9-kj7zv                    1/1     Running   0          7d15h
orc8r-user-grafana-bc644b4fc-ptd6w               1/1     Running   2          294d

Last issue I had was getting forbidden when trying to into NMS and it got solved by restarting nms-nginx-proxy by running kubectl -n orc8r rollout restart deployment/nms-nginx-proxy #5577 I retried this command just in case but it did not work.

I was wondering if I should restart other or all the services or if the issue might be related to something else.

[karthiksubraveti]

As you mentioned everything looks alright. I suspect it is something to do with the admin cert on the NMS. Can you check if it expired and if you have to renew it.

[BioZrod]

you are completely right I did not notice it has been a year already since original certificates where made. Should I just repeat this steps https://docs.magmacore.org/docs/orc8r/deploy_install#assemble-certificates?

I didn't do the original installation so I haven't done this steps before.

[BioZrod]

Update: Indeed original certificates expired on Mar 23 I did run the steps on https://docs.magmacore.org/docs/orc8r/deploy_install#assemble-certificates and certificates have been uploaded to AWS Secrets I deleted old admin_operator.pfx from my chrome certs and added new one but I still get the same page is there something I might be missing?

I ran the following commands:

mkdir -p ~/secrets/certs
cd ~/secrets/certs

On certs folder:

${MAGMA_ROOT}/orc8r/cloud/deploy/scripts/self_sign_certs.sh yourdomain.com
${MAGMA_ROOT}/orc8r/cloud/deploy/scripts/create_application_certs.sh yourdomain.com
openssl pkcs12 -export -inkey admin_operator.key.pem -in admin_operator.pem -out admin_operator.pfx

On folder where main.tf is:

terraform taint module.orc8r-app.null_resource.orc8r_seed_secrets
terraform apply -target=module.orc8r-app.null_resource.orc8r_seed_secrets

Output:

biozrod@Rodrigos-MacBook-Pro eks % terraform apply -target=module.orc8r-app.null_resource.orc8r_seed_secrets
module.orc8r.aws_secretsmanager_secret.orc8r_secrets: Refreshing state... [id=REDACTED]
module.orc8r-app.null_resource.orc8r_seed_secrets: Refreshing state... [id=REDACTED]
An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
-/+ destroy and then create replacement
Terraform will perform the following actions:
 # module.orc8r-app.null_resource.orc8r_seed_secrets is tainted, so must be replaced
-/+ resource "null_resource" "orc8r_seed_secrets" {
   ~ id = "8280814536074226349" -> (known after apply)
  }
Plan: 1 to add, 0 to change, 1 to destroy.
Warning: Resource targeting is in effect
You are creating a plan with the -target option, which means that the result
of this plan may not represent all of the changes requested by the current
configuration.
		
The -target option is not for routine use, and is provided only for
exceptional situations such as recovering from errors or mistakes, or when
Terraform specifically suggests to use it as part of an error message.
Warning: Version constraints inside provider configuration blocks are deprecated
 on main.tf line 16, in provider "aws":
 16:  version = ">= 2.6.0"
Terraform 0.13 and earlier allowed provider version constraints inside the
provider configuration block, but that is now deprecated and will be removed
in a future version of Terraform. To silence this warning, move the provider
version constraint into the required_providers block.
(and 10 more similar warnings elsewhere)
Do you want to perform these actions?
 Terraform will perform the actions described above.
 Only 'yes' will be accepted to approve.
 Enter a value: yes
module.orc8r-app.null_resource.orc8r_seed_secrets: Destroying... [id=REDACTED]
module.orc8r-app.null_resource.orc8r_seed_secrets: Destruction complete after 0s
module.orc8r-app.null_resource.orc8r_seed_secrets: Creating...
module.orc8r-app.null_resource.orc8r_seed_secrets: Provisioning with 'local-exec'...
module.orc8r-app.null_resource.orc8r_seed_secrets (local-exec): Executing: ["/bin/sh" "-c" "   .terraform/modules/orc8r-app/orc8r/cloud/deploy/terraform/orc8r-helm-aws/scripts/create_orc8r_secrets.py \\\n    'orc8r-secrets' 'us-east-1' \\\n    \"/Users/biozrod/secrets/certs\"\n"]
module.orc8r-app.null_resource.orc8r_seed_secrets: Creation complete after 2s [id=REDACTED]
Warning: Applied changes may be incomplete
The plan was created with the -target option in effect, so some changes
requested in the configuration may have been ignored and the output values may
not be fully updated. Run the following command to verify that no other
changes are pending:
  terraform plan
	
Note that the -target option is not suitable for routine use, and is provided
only for exceptional situations such as recovering from errors or mistakes, or
when Terraform specifically suggests to use it as part of an error message.
Apply complete! Resources: 1 added, 0 changed, 1 destroyed.

I ran terraform apply just in case cause of the warning it applied successfully. And then an terraform plan and it says it does not detect any differences.

Finally I tried kubectl -n orc8r exec ${ORC_POD} -- envdir /var/opt/magma/envdir /var/opt/magma/bin/accessc add-existing -admin -cert /var/opt/magma/certs/admin_operator.pem admin_operator and got

2021/03/26 06:25:05 Error 'rpc error: code = InvalidArgument desc = Certificate Verification Failure: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "certifier.REDACTED") for Certificate SN E1E32DEC08645C30' Adding Certificate From /var/opt/magma/certs/admin_operator.pem
command terminated with exit code 1

[karthiksubraveti]

@BioZrod This is the process to replace existing admin certs. cc @sudhikan @wallyrb

Scenario - NMS is unable to communicate with the controller or API access is no longer functional; it used to be now it’s not.

Cause - The certificate used for the TLS handshake has expired.

Resolution - New admin_operator certificates need to be generated and uploaded to the NMS k8s pod for NMS access and to the browser for the API access.

How to generate new admin_operator certificates:

1. Log on to a host that has kubectl access to your orc8r cluster.
2. Access the shell on your controller pod.

export CNTLR_POD=$(kubectl -n orc8r get pod -l app.kubernetes.io/component=controller -o jsonpath='{.items[0].metadata.name}')
kubectl exec -it ${CNTLR_POD} bash

3. Run the following commands inside the k8s controller pod to generate a new admin_operator cert. The admin_operator.pfx file is only for API access for the user to get to the browser.

The following commands are to be run inside the pod

(pod)$ cd /var/opt/magma/bin
(pod)$ envdir /var/opt/magma/envdir ./accessc add-admin -cert admin_operator admin_operator
(pod)$ openssl pkcs12 -export -out admin_operator.pfx -inkey admin_operator.key.pem -in admin_operator.pem

Enter Export Password:
Verifying - Enter Export Password:

4. Copy the k8s certs from the controller k8s pod to the local directoy where all your secrets are held.

NOTE: Make sure the location where these certificates are being copied to is the same location which is referenced in the main.tf file for the seed_certs_dir variable.

cd ~/secrets/certs
for certfile in admin_operator.pem admin_operator.key.pem admin_operator.pfx
do
    kubectl cp orc8r/${CNTLR_POD}:/var/opt/magma/certs/${certfile} ./${certfile}
done

5. At this point the new certs have been generated, you can use the admin_operator.pfx to validate that you can reach the API endpoint. However, they still need to be applied to the k8s secrets manager so that they can be mounted in the nms pod. To do this, initialize your terraform first using terraform init -upgrade.

6. To replace certs, we have to first taint the secrets in terraform so that terraform knows to destroy those secrets first and then re-apply them. Taint them using terraform taint module.orc8r-app.null_resource.orc8r_seed_secrets

7. Once the secrets have been tainted, you can then go ahead and apply the new secrets by running terraform apply -target=module.orc8rapp.null_resource.orc8r_seed_secrets followed by terraform apply

NOTE: terraform apply command outputs the “plan” of what it intends to add,destroy,modify. Please scrutinize this output before typing “yes” on the confirm prompt. If there are any changes that are not consistent with your expectations, please cancel the run. You can specifically target the secrets portion by doing terraform apply -target=module.

8. Once the terraform apply succeeds, the NMS will not automatically get the new certificates until we destroy the existing pod and force the replication controller to instantiate another instance of the pod. To do this, run the following commands:

export NMS_POD=$(kubectl -n orc8r get pod -l  app.kubernetes.io/component=magmalte -o jsonpath='{.items[0].metadata.name}')


> kubectl -n orc8r delete pod $NMS_POD

9. Once the pod reinitializes, it should have the latest admin_operator certs and be able to re-establish communication with the controller.

[BioZrod]

This solution works perfectly just to add as we discussed:

For magma v1.4 which is the one I am using right now this should change from controller to orchestrator:
export CNTLR_POD=$(kubectl -n orc8r get pod -l app.kubernetes.io/component=orchestrator -o jsonpath='{.items[0].metadata.name}')

Also when running commands inside the pod the new admin operator certs are created in the same folder:

And certs folder which is used to get the certificates on local directory using:

cd ~/secrets/certs
for certfile in admin_operator.pem admin_operator.key.pem admin_operator.pfx
do
    kubectl cp orc8r/${CNTLR_POD}:/var/opt/magma/certs/${certfile} ./${certfile}
done

Only has the following certs:

so the code to get the certs will be:

cd ~/secrets/certs
for certfile in admin_operator.pem admin_operator.key.pem admin_operator.pfx
do
    kubectl cp orc8r/${CNTLR_POD}:/var/opt/magma/bin/${certfile} ./${certfile}
done

With those changes in consideration I followed the guide and successfully have NMS running again. Just unsure if there is the need to somehow put all the admin operator certs in pod certs folder as when I tried to move them there I got folder is read only