NMS Throws forbidden after upgrade

[BioZrod]

Yesterday I successfully went from version 1.2 of orc8r to 1.3 after successful terraform apply I did the data migration. Checked everything was running and then proceeded to upgrade to 1.4 following documentation.

I used v1.14 tag that was released yesterday to build and push to docker registry and helm repository. After a successful terraform apply I did test NMS it was working I checked grafana explorer to check data was still there and pushed some data to it and it was working fine.

After about 5 hours I tried login again to nms and got a blank page with forbidden printed on it. I did run kubectl -n orc8r get pods to check they were working and everything seems to be running properly.

nms-magmalte-7b88f9dbc6-5l5m5          1/1   Running  0     17h
nms-nginx-proxy-57b8585d6-v8mwh         1/1   Running  0     287d
orc8r-accessd-8677db8576-h2jqq          1/1   Running  0     17h
orc8r-accessd-8677db8576-qms8g          1/1   Running  0     17h
orc8r-alertmanager-84d79f774b-2qk4h       1/1   Running  0     359d
orc8r-alertmanager-configurer-6c55bf95cb-jp7cj  1/1   Running  0     17h
orc8r-analytics-5b5465885d-9z8d4         1/1   Running  0     17h
orc8r-analytics-5b5465885d-q9xvl         1/1   Running  0     17h
orc8r-bootstrapper-7b46dc8f7f-2rr6q       1/1   Running  0     17h
orc8r-bootstrapper-7b46dc8f7f-v95v5       1/1   Running  0     17h
orc8r-certifier-5f74677c48-4whh7         1/1   Running  0     17h
orc8r-certifier-5f74677c48-dltw7         1/1   Running  0     17h
orc8r-configurator-86c77bc786-b4t9f       1/1   Running  0     17h
orc8r-configurator-86c77bc786-qw8pt       1/1   Running  0     17h
orc8r-ctraced-66c45bb8c8-dqtlc          1/1   Running  0     17h
orc8r-ctraced-66c45bb8c8-pscgj          1/1   Running  0     17h
orc8r-device-565d6454f7-lf9gt          1/1   Running  0     17h
orc8r-device-565d6454f7-vnzzn          1/1   Running  0     17h
orc8r-directoryd-775cbc48fd-2pxzj        1/1   Running  0     17h
orc8r-directoryd-775cbc48fd-95bl6        1/1   Running  0     17h
orc8r-dispatcher-6d6f98c545-ttlj4        1/1   Running  0     17h
orc8r-dispatcher-6d6f98c545-vxrts        1/1   Running  0     17h
orc8r-eventd-68d7975f5b-bcpqg          1/1   Running  0     17h
orc8r-eventd-68d7975f5b-fhk54          1/1   Running  0     17h
orc8r-ha-875884b48-7cww4             1/1   Running  0     17h
orc8r-ha-875884b48-bzjv9             1/1   Running  0     17h
orc8r-lte-5dbb99b9f-9c5x6            1/1   Running  0     17h
orc8r-lte-5dbb99b9f-m986g            1/1   Running  0     17h
orc8r-metricsd-5474b55fff-jthh4         1/1   Running  0     17h
orc8r-metricsd-5474b55fff-tlhnf         1/1   Running  0     17h
orc8r-nginx-7fbfbbd8c4-7xvgc           1/1   Running  0     17h
orc8r-nginx-7fbfbbd8c4-hln59           1/1   Running  0     17h
orc8r-obsidian-779d49cbc7-9gkrg         1/1   Running  0     17h
orc8r-obsidian-779d49cbc7-v5s8s         1/1   Running  0     17h
orc8r-orchestrator-6b56f7d8b4-c96zn       1/1   Running  0     17h
orc8r-orchestrator-6b56f7d8b4-r7njh       1/1   Running  0     17h
orc8r-policydb-6d774b5b4b-7xvph         1/1   Running  0     17h
orc8r-policydb-6d774b5b4b-snhc4         1/1   Running  0     17h
orc8r-prometheus-7fc468d458-ftfmd        1/1   Running  6     17h
orc8r-prometheus-cache-5c8d9d7b8-njksm      1/1   Running  0     17h
orc8r-prometheus-configurer-546d98769f-nwz2b   1/1   Running  0     17h
orc8r-service-registry-68c9d79c86-vg75g     1/1   Running  0     17h
orc8r-service-registry-68c9d79c86-zrdmp     1/1   Running  0     17h
orc8r-smsd-77495d8f74-kqjtx           1/1   Running  0     17h
orc8r-smsd-77495d8f74-mqs76           1/1   Running  0     17h
orc8r-state-789885d989-5lxjf           1/1   Running  0     17h
orc8r-state-789885d989-wkcdb           1/1   Running  0     17h
orc8r-streamer-57d9bf5cc5-2vjsr         1/1   Running  0     17h
orc8r-streamer-57d9bf5cc5-qz4xh         1/1   Running  0     17h
orc8r-subscriberdb-7d9d79499d-gz4c8       1/1   Running  0     17h
orc8r-subscriberdb-7d9d79499d-lt8hk       1/1   Running  0     17h
orc8r-tenants-69bc77dc9-2kjrx          1/1   Running  0     17h
orc8r-tenants-69bc77dc9-kj7zv          1/1   Running  0     17h
orc8r-user-grafana-bc644b4fc-ptd6w        1/1   Running  1     287d

Then I ran kubectl -n orc8r logs -f nms-magmalte-7b88f9dbc6-515m5 and found this in the logs:

{"message":"::ffff:10.10.102.10 - - [17/Mar/2021:15:52:41 +0000] \"GET /nms/static/images/favicon.png HTTP/1.0\" 200 - \"https://test.nms.openschema.magma.etagecom.io/user/login?to=%2Fnms\" \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.82 Safari/537.36\"\n","level":"info","label":"express-middleware/appMiddleware.js","timestamp":"2021-03-17T15:52:41.058Z"}
ForbiddenError: invalid csrf token
    at csrf (/usr/src/node_modules/csurf/index.js:112:19)
    at Layer.handle [as handle_request] (/usr/src/node_modules/express/lib/router/layer.js:95:5)
    at trim_prefix (/usr/src/node_modules/express/lib/router/index.js:317:13)
    at /usr/src/node_modules/express/lib/router/index.js:284:7
    at Function.process_params (/usr/src/node_modules/express/lib/router/index.js:335:12)
    at next (/usr/src/node_modules/express/lib/router/index.js:275:10)
    at Function.handle (/usr/src/node_modules/express/lib/router/index.js:174:3)
    at router (/usr/src/node_modules/express/lib/router/index.js:47:12)
    at Layer.handle [as handle_request] (/usr/src/node_modules/express/lib/router/layer.js:95:5)
    at trim_prefix (/usr/src/node_modules/express/lib/router/index.js:317:13)
    at /usr/src/node_modules/express/lib/router/index.js:284:7
    at Function.process_params (/usr/src/node_modules/express/lib/router/index.js:335:12)
    at next (/usr/src/node_modules/express/lib/router/index.js:275:10)
    at setup (/usr/src/node_modules/@fbcnms/auth/access.js:45:5)
    at Layer.handle [as handle_request] (/usr/src/node_modules/express/lib/router/layer.js:95:5)
    at trim_prefix (/usr/src/node_modules/express/lib/router/index.js:317:13)
    at /usr/src/node_modules/express/lib/router/index.js:284:7
    at Function.process_params (/usr/src/node_modules/express/lib/router/index.js:335:12)
    at next (/usr/src/node_modules/express/lib/router/index.js:275:10)
    at /usr/src/node_modules/express/lib/router/index.js:635:15
    at next (/usr/src/node_modules/express/lib/router/index.js:260:14)
    at Function.handle (/usr/src/node_modules/express/lib/router/index.js:174:3)
    ```

[karthiksubraveti]

@BioZrod Can you restart nms-nginx-proxy ?
kubectl -n orc8r rollout restart deployment/nms-nginx-proxy
In 1.4, we updated the config files for nms-nginx proxy which should help fix this issue.

[BioZrod]

This fixed the issue thank you