HTTP404 while trying to check the request status.

[makautzn]

Hi,
when I POST a new delivery request, I do receive a HTTP202, which seems to be expected.

HTTP/2 202 
server: nginx/1.15.6
date: Wed, 13 Nov 2019 11:15:43 GMT
content-type: application/json;charset=UTF-8
location: http://deliveries/api/deliveries/d3e64649-fae2-4fc5-8498-09b34de70227
request-context: appId=cid-v1:b338a12e-bd41-4560-aa9c-1508ed91a24e
strict-transport-security: max-age=15724800; includeSubDomains

If I try to check the request status I keep getting HTTP404.

curl "https://$EXTERNAL_INGEST_FQDN/api/deliveries/d3e64649-fae2-4fc5-8498-09b34de70227" --header 'Accept: application/json' -k -i
HTTP/2 404 
server: nginx/1.15.6
date: Wed, 13 Nov 2019 11:21:16 GMT
content-length: 0
request-context: appId=cid-v1:b338a12e-bd41-4560-aa9c-1508ed91a24e
strict-transport-security: max-age=15724800; includeSubDomains

All services seem to be available.

helm ls
NAME                            REVISION        UPDATED                         STATUS  CHART                    APP VERSION     NAMESPACE          
delivery-v0.1.0-dev             1               Wed Nov 13 10:59:13 2019        DEPLOYEDdelivery-v0.1.0          v0.1.0          backend-dev        
dronescheduler-v0.1.0-dev       1               Wed Nov 13 11:27:00 2019        DEPLOYEDdronescheduler-v0.1.0    v0.1.0          backend-dev        
ingestion-v0.1.0-dev            1               Wed Nov 13 11:21:38 2019        DEPLOYEDingestion-v0.1.0         v0.1.0          backend-dev        
nginx-ingress                   1               Wed Nov 13 10:49:01 2019        DEPLOYEDnginx-ingress-1.0.1      0.21.0          ingress-controllers
package-v0.1.0-dev              1               Wed Nov 13 11:03:43 2019        DEPLOYEDpackage-v0.1.0           v0.1.0          backend-dev        
workflow-v0.1.0-dev             1               Wed Nov 13 11:13:15 2019        DEPLOYEDworkflow-v0.1.0          v0.1.0          backend-dev    

[ferantivero]

thanks for sharing this @makautzn, the http202 is just perfect!

Regarding the http404, could you please share the outcome from the following two executions:

1. kubectl get po -l app.kubernetes.io/part-of=dronedelivery -n backend-dev
2. kubectl get svc -l app.kubernetes.io/part-of=dronedelivery -n backend-dev

Also if happens that you find a pod with phase different than Running, I'd suggest to dig a bit deeper by executing:

3. kubectl logs <pod-name-in-not-running-phase> -n backend-dev

[makautzn]

Thanks @ferantivero for your quick response!

kubectl get pod -l app.kubernetes.io/part-of=dronedelivery -n backend-dev
NAME                                      READY   STATUS    RESTARTS   AGE
delivery-v010-dev-6864cffd8c-594cg        1/1     Running   2          161m
dronescheduler-v010-dev-fb8c75747-j9xgz   1/1     Running   0          133m
ingestion-v010-dev-967cf5f56-84djd        1/1     Running   0          139m
package-v010-dev-5696fc8575-2nl8m         1/1     Running   0          156m
workflow-v010-dev-7f5ff64bb4-xhhk6        1/1     Running   0          147m

kubectl get svc -l app.kubernetes.io/part-of=dronedelivery -n backend-dev
NAME                  TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE
delivery              ClusterIP   10.0.221.207   <none>        80/TCP    161m
delivery-v010         ClusterIP   10.0.229.20    <none>        80/TCP    162m
dronescheduler        ClusterIP   10.0.70.132    <none>        80/TCP    134m
dronescheduler-v010   ClusterIP   10.0.79.50     <none>        80/TCP    134m
ingestion             ClusterIP   10.0.83.41     <none>        80/TCP    139m
ingestion-v010        ClusterIP   10.0.191.189   <none>        80/TCP    139m
package               ClusterIP   10.0.81.94     <none>        80/TCP    157m
package-v010          ClusterIP   10.0.86.185    <none>        80/TCP    157m

"kubectl logs delivery-v010-dev-6864cffd8c-594cg -n backend-dev" output as attached
delivery-v010-dev-6864cffd8c-594cg.txt

[ferantivero]

thanks for sharing the output logs with us @makautzn, we've been able to reprod this; it seems the little bit oldie Nginx you got installed is truncating the GET url (it cuts the id, so it results in a 404).

Please let's follow the troubleshooting steps below to get it back on business. The plan is to install a newer Nginx version:

1. helm del nginx-ingress --purge
2. helm repo update
3. helm install stable/nginx-ingress --name nginx-ingress --namespace ingress-controllers --set rbac.create=true --version 1.24.7
4. until export INGRESS_LOAD_BALANCER_IP=$(kubectl get services/nginx-ingress-controller -n ingress-controllers -o jsonpath="{.status.loadBalancer.ingress[0].ip}" 2> /dev/null) && test -n "$INGRESS_LOAD_BALANCER_IP"; do echo "Waiting for load balancer deployment" && sleep 20; done
5. export INGRESS_LOAD_BALANCER_IP_ID=$(az network public-ip list --query "[?ipAddress!=null]|[?contains(ipAddress, '$INGRESS_LOAD_BALANCER_IP')].[id]" --output tsv)
6. export EXTERNAL_INGEST_DNS_NAME="${RESOURCE_GROUP}-ingest"
7. export EXTERNAL_INGEST_FQDN=$(az network public-ip update --ids $INGRESS_LOAD_BALANCER_IP_ID --dns-name $EXTERNAL_INGEST_DNS_NAME --query "dnsSettings.fqdn" --output tsv)

After this point, you should be able to execute the deliveries GET request without issues 😸

I'm closing this issue for now, but please don't hesite on re-open it or create a new one if you need more support.

[makautzn]

Thanks @ferantivero! Updating NGinx fixed it.