djapp "connection refused" error

[alexei-led]

Going over djapp walkthrough, done it 3 times.

Platform
EKS

To Reproduce
Steps to reproduce the behavior:

1. Deploy app and virtual nodes and services following the guide
2. Get into djapp pod antr try to access Jazz or Metal virtual services with curl jazz.prod.svc.cluster.local:9080;echo command
3. Got curl: (7) Failed to connect to jazz.prod.svc.cluster.local port 9080: Connection refused message

Expected behavior
Expect that example should work

[lavignes]

Hey @alexei-led we're starting to look into this. We'll reply back soon when we know what's up or can suggest something.

[kiranmeduri]

@alexei-led can you provide output of kubectl get all -nprod. I tried but unable to reproduce.

root@dj-85dc6bb78-8jtrt:/usr/src/app# curl http://metal.prod.svc.cluster.local:9080/;echo
["Megadeth (Los Angeles, California)","Judas Priest (West Bromwich, England)"]
root@dj-85dc6bb78-8jtrt:/usr/src/app# curl http://jazz.prod.svc.cluster.local:9080/;echo
["Astrud Gilberto","Miles Davis"]

[alexei-led]

@karanvasnani hi
sorry it was my error: I've created app-mesh injector and controller manually, follwoing instructions on their github pages. I also did a typo mistake in mesh name (used _ instead of -). So the example did not work as expected.

Closing this issue.

p.s.: is there a way to see/debug such mistake? I was looking at logs, but could no see anything. IMHO troubleshooting documentation could be very helpful here.

[sweluhu]

Same issue here. Curling Virtual Service Jazz and Metal do not redirect to weight distributed v1 and v2. Curling each version works. I went over Injector and CRDs twice with no success.

Please advise

✔ ~/workspace/aws-app-mesh-examples/examples/apps/djapp [master ↓·3|✚ 2…2]
15:37 $ kubectl get pods -nprod
NAME                        READY     STATUS    RESTARTS   AGE
dj-7b54cb8fc9-jbj4g         1/1       Running   0          17s
jazz-v1-d7f8b5c9-597fp      1/1       Running   0          15s
jazz-v2-59b7fccd8-nbclq     1/1       Running   0          37m
metal-v1-784c497fb9-rzs6w   1/1       Running   0          16s
metal-v2-646b7b6dc-w9fd9    1/1       Running   0          36m
✔ ~/workspace/aws-app-mesh-examples/examples/apps/djapp [master ↓·3|✚ 2…2]
15:37 $ kubectl get pods -nprod -lapp=dj
NAME                  READY     STATUS    RESTARTS   AGE
dj-7b54cb8fc9-jbj4g   1/1       Running   0          53s
✔ ~/workspace/aws-app-mesh-examples/examples/apps/djapp [master ↓·3|✚ 2…2]
15:38 $ kubectl exec -nprod -it dj-7b54cb8fc9-jbj4g -c dj bash

root@dj-7b54cb8fc9-jbj4g:/usr/src/app# for i in jazz jazz-v1 jazz-v2 metal metal-v1 metal-v2; do curl $i.prod.svc.cluster.local:9080;echo; done
curl: (7) Failed to connect to jazz.prod.svc.cluster.local port 9080: Connection refused

["Astrud Gilberto","Miles Davis"]
["Astrud Gilberto (Bahia, Brazil)","Miles Davis (Alton, Illinois)"]
curl: (7) Failed to connect to metal.prod.svc.cluster.local port 9080: Connection refused

["Megadeth","Judas Priest"]
["Megadeth (Los Angeles, California)","Judas Priest (West Bromwich, England)"]
root@dj-7b54cb8fc9-jbj4g:/usr/src/app#

[sweluhu]

Same issue here. Curling Virtual Service Jazz and Metal do not redirect to weight distributed v1 and v2. Curling each version works. I went over Injector and CRDs twice with no success.

Please advise

✔ ~/workspace/aws-app-mesh-examples/examples/apps/djapp [master ↓·3|✚ 2…2]
15:37 $ kubectl get pods -nprod
NAME                        READY     STATUS    RESTARTS   AGE
dj-7b54cb8fc9-jbj4g         1/1       Running   0          17s
jazz-v1-d7f8b5c9-597fp      1/1       Running   0          15s
jazz-v2-59b7fccd8-nbclq     1/1       Running   0          37m
metal-v1-784c497fb9-rzs6w   1/1       Running   0          16s
metal-v2-646b7b6dc-w9fd9    1/1       Running   0          36m
✔ ~/workspace/aws-app-mesh-examples/examples/apps/djapp [master ↓·3|✚ 2…2]
15:37 $ kubectl get pods -nprod -lapp=dj
NAME                  READY     STATUS    RESTARTS   AGE
dj-7b54cb8fc9-jbj4g   1/1       Running   0          53s
✔ ~/workspace/aws-app-mesh-examples/examples/apps/djapp [master ↓·3|✚ 2…2]
15:38 $ kubectl exec -nprod -it dj-7b54cb8fc9-jbj4g -c dj bash

root@dj-7b54cb8fc9-jbj4g:/usr/src/app# for i in jazz jazz-v1 jazz-v2 metal metal-v1 metal-v2; do curl $i.prod.svc.cluster.local:9080;echo; done
curl: (7) Failed to connect to jazz.prod.svc.cluster.local port 9080: Connection refused

["Astrud Gilberto","Miles Davis"]
["Astrud Gilberto (Bahia, Brazil)","Miles Davis (Alton, Illinois)"]
curl: (7) Failed to connect to metal.prod.svc.cluster.local port 9080: Connection refused

["Megadeth","Judas Priest"]
["Megadeth (Los Angeles, California)","Judas Priest (West Bromwich, England)"]
root@dj-7b54cb8fc9-jbj4g:/usr/src/app#

Here is the kubectl get all:

✔ ~/workspace/aws-app-mesh-examples/examples/apps/djapp [master ↓·3|✚ 2…2]
15:55 $ kubectl get all -nprod
NAME                            READY     STATUS    RESTARTS   AGE
pod/dj-7b54cb8fc9-jbj4g         1/1       Running   0          17m
pod/jazz-v1-d7f8b5c9-597fp      1/1       Running   0          17m
pod/jazz-v2-59b7fccd8-nbclq     1/1       Running   0          54m
pod/metal-v1-784c497fb9-rzs6w   1/1       Running   0          17m
pod/metal-v2-646b7b6dc-w9fd9    1/1       Running   0          53m

NAME               TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
service/dj         ClusterIP   10.100.120.85    <none>        9080/TCP   2h
service/jazz       ClusterIP   10.100.115.246   <none>        9080/TCP   1h
service/jazz-v1    ClusterIP   10.100.225.131   <none>        9080/TCP   2h
service/jazz-v2    ClusterIP   10.100.167.144   <none>        9080/TCP   54m
service/metal      ClusterIP   10.100.71.1      <none>        9080/TCP   1h
service/metal-v1   ClusterIP   10.100.207.235   <none>        9080/TCP   2h
service/metal-v2   ClusterIP   10.100.172.172   <none>        9080/TCP   53m

NAME                       DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/dj         1         1         1            1           2h
deployment.apps/jazz-v1    1         1         1            1           2h
deployment.apps/jazz-v2    1         1         1            1           54m
deployment.apps/metal-v1   1         1         1            1           2h
deployment.apps/metal-v2   1         1         1            1           53m

NAME                                  DESIRED   CURRENT   READY     AGE
replicaset.apps/dj-67768854d5         0         0         0         1h
replicaset.apps/dj-7858b85876         0         0         0         1h
replicaset.apps/dj-787b6fcf45         0         0         0         1h
replicaset.apps/dj-7b54cb8fc9         1         1         1         17m
replicaset.apps/dj-7fbb4d499c         0         0         0         2h
replicaset.apps/jazz-v1-6d5dd497f     0         0         0         2h
replicaset.apps/jazz-v1-7dbb8c764f    0         0         0         1h
replicaset.apps/jazz-v1-8597c5c566    0         0         0         1h
replicaset.apps/jazz-v1-d7f8b5c9      1         1         1         17m
replicaset.apps/jazz-v2-59b7fccd8     1         1         1         54m
replicaset.apps/metal-v1-56c9768d76   0         0         0         1h
replicaset.apps/metal-v1-746d77c67b   0         0         0         2h
replicaset.apps/metal-v1-784c497fb9   1         1         1         17m
replicaset.apps/metal-v1-7d5747f5c9   0         0         0         1h
replicaset.apps/metal-v2-646b7b6dc    1         1         1         53m

NAME                                   AGE
virtualnode.appmesh.k8s.aws/dj         1h
virtualnode.appmesh.k8s.aws/jazz       1h
virtualnode.appmesh.k8s.aws/jazz-v1    1h
virtualnode.appmesh.k8s.aws/jazz-v2    54m
virtualnode.appmesh.k8s.aws/metal      1h
virtualnode.appmesh.k8s.aws/metal-v1   1h
virtualnode.appmesh.k8s.aws/metal-v2   53m

NAME                                                          AGE
virtualservice.appmesh.k8s.aws/jazz.prod.svc.cluster.local    1h
virtualservice.appmesh.k8s.aws/metal.prod.svc.cluster.local   1h

NAME                          AGE
mesh.appmesh.k8s.aws/dj-app   1h