Difficulty running e2e tests

[jfmyers9]

I keep running into an error running the end to end tests on the develop branch with the following failure:

==> deploy initial
Running 'kapp deploy -f - -a test-template --diff-changes --tty -n kapp-test --into-ns kapp-test --yes'...
Running 'kapp delete -a test-template -n kapp-test --yes'...
--- FAIL: TestTemplate (610.06s)
    kapp.go:83: Failed to successfully execute 'kapp deploy -f - -a test-template --diff-changes --tty -n kapp-test --into-ns kapp-test --yes': Execution error: stdout: '--- create configmap/config-ver-1 (v1) namespace: kapp-test
              0 + apiVersion: v1
              1 + data:
              2 +   key1: val1
              3 + kind: ConfigMap
              4 + metadata:
              5 +   annotations:
              6 +     kapp.k14s.io/versioned: ""
              7 +   labels:
              8 +     kapp.k14s.io/app: "1574793934192273000"
              9 +     kapp.k14s.io/association: v1.62442fcb57d209abe192460ff4403c7c
             10 +   name: config-ver-1
             11 +   namespace: kapp-test
             12 +
        --- create secret/secret-ver-1 (v1) namespace: kapp-test
              0 + apiVersion: v1
              1 + data:
              2 +   key1: val1
              3 + kind: Secret
              4 + metadata:
              5 +   annotations:
              6 +     kapp.k14s.io/versioned: ""
              7 +   labels:
              8 +     kapp.k14s.io/app: "1574793934192273000"
              9 +     kapp.k14s.io/association: v1.8275953a67aea7cbb147343af1608f5b
             10 +   name: secret-ver-1
             11 +   namespace: kapp-test
             12 +
        --- create deployment/dep (apps/v1) namespace: kapp-test
              0 + apiVersion: apps/v1
              1 + kind: Deployment
              2 + metadata:
              3 +   labels:
              4 +     kapp.k14s.io/app: "1574793934192273000"
              5 +     kapp.k14s.io/association: v1.6f33d83389b69bd6ab35a82aa23e12fc
              6 +   name: dep
              7 +   namespace: kapp-test
              8 + spec:
              9 +   replicas: 1
             10 +   selector:
             11 +     matchLabels:
             12 +       app: dep
             13 +       kapp.k14s.io/app: "1574793934192273000"
             14 +   template:
             15 +     metadata:
             16 +       labels:
             17 +         app: dep
             18 +         kapp.k14s.io/app: "1574793934192273000"
             19 +         kapp.k14s.io/association: v1.6f33d83389b69bd6ab35a82aa23e12fc
             20 +     spec:
             21 +       containers:
             22 +       - args:
             23 +         - -listen=:80
             24 +         - -text=hello
             25 +         envFrom:
             26 +         - configMapRef:
             27 +             name: config-ver-1
             28 +         image: hashicorp/http-echo
             29 +         name: echo
             30 +         ports:
             31 +         - containerPort: 80
             32 +       volumes:
             33 +       - name: vol1
             34 +         secret:
             35 +           secretName: secret
             36 +

        Changes

        Namespace  Name          Kind        Conds.  Age  Op      Wait to    Rs  Ri
        kapp-test  config-ver-1  ConfigMap   -       -    create  reconcile  -   -
        ^          dep           Deployment  -       -    create  reconcile  -   -
        ^          secret-ver-1  Secret      -       -    create  reconcile  -   -

        Op:      3 create, 0 delete, 0 update, 0 noop
        Wait to: 3 reconcile, 0 delete, 0 noop

        10:45:35AM: ---- applying 3 changes [0/3 done] ----
        10:45:35AM: create configmap/config-ver-1 (v1) namespace: kapp-test
        10:45:35AM: create secret/secret-ver-1 (v1) namespace: kapp-test
        10:45:35AM: create deployment/dep (apps/v1) namespace: kapp-test
        10:45:36AM: ---- waiting on 3 changes [0/3 done] ----
        10:45:37AM: ok: reconcile configmap/config-ver-1 (v1) namespace: kapp-test
        10:45:37AM: ok: reconcile secret/secret-ver-1 (v1) namespace: kapp-test
        10:45:38AM: ongoing: reconcile deployment/dep (apps/v1) namespace: kapp-test
        10:45:38AM:  ^ Waiting for 1 unavailable replicas
        10:45:38AM:  L ok: waiting on replicaset/dep-5658f778d6 (apps/v1) namespace: kapp-test
        10:45:38AM:  L ongoing: waiting on pod/dep-5658f778d6-mcksg (v1) namespace: kapp-test
        10:45:38AM:     ^ Pending: ContainerCreating
        10:45:38AM: ---- waiting on 1 changes [2/3 done] ----
        10:46:38AM: ongoing: reconcile deployment/dep (apps/v1) namespace: kapp-test
        10:46:38AM:  ^ Waiting for 1 unavailable replicas
        10:46:38AM:  L ok: waiting on replicaset/dep-5658f778d6 (apps/v1) namespace: kapp-test
        10:46:38AM:  L ongoing: waiting on pod/dep-5658f778d6-mcksg (v1) namespace: kapp-test
        10:46:38AM:     ^ Pending: ContainerCreating
        10:46:39AM: ---- waiting on 1 changes [2/3 done] ----
        10:47:39AM: ongoing: reconcile deployment/dep (apps/v1) namespace: kapp-test
        10:47:39AM:  ^ Waiting for 1 unavailable replicas
        10:47:39AM:  L ok: waiting on replicaset/dep-5658f778d6 (apps/v1) namespace: kapp-test
        10:47:39AM:  L ongoing: waiting on pod/dep-5658f778d6-mcksg (v1) namespace: kapp-test
        10:47:39AM:     ^ Pending: ContainerCreating
        10:47:40AM: ---- waiting on 1 changes [2/3 done] ----
        10:48:39AM: ongoing: reconcile deployment/dep (apps/v1) namespace: kapp-test
        10:48:39AM:  ^ Waiting for 1 unavailable replicas
        10:48:39AM:  L ok: waiting on replicaset/dep-5658f778d6 (apps/v1) namespace: kapp-test
        10:48:39AM:  L ongoing: waiting on pod/dep-5658f778d6-mcksg (v1) namespace: kapp-test
        10:48:39AM:     ^ Pending: ContainerCreating
        10:48:40AM: ---- waiting on 1 changes [2/3 done] ----
        10:49:40AM: ongoing: reconcile deployment/dep (apps/v1) namespace: kapp-test
        10:49:40AM:  ^ Waiting for 1 unavailable replicas
        10:49:40AM:  L ok: waiting on replicaset/dep-5658f778d6 (apps/v1) namespace: kapp-test
        10:49:40AM:  L ongoing: waiting on pod/dep-5658f778d6-mcksg (v1) namespace: kapp-test
        10:49:40AM:     ^ Pending: ContainerCreating
        10:49:41AM: ---- waiting on 1 changes [2/3 done] ----
        10:50:40AM: ongoing: reconcile deployment/dep (apps/v1) namespace: kapp-test
        10:50:40AM:  ^ Waiting for 1 unavailable replicas
        10:50:40AM:  L ok: waiting on replicaset/dep-5658f778d6 (apps/v1) namespace: kapp-test
        10:50:40AM:  L ongoing: waiting on pod/dep-5658f778d6-mcksg (v1) namespace: kapp-test
        10:50:40AM:     ^ Pending: ContainerCreating
        10:50:41AM: ---- waiting on 1 changes [2/3 done] ----
        10:51:41AM: ongoing: reconcile deployment/dep (apps/v1) namespace: kapp-test
        10:51:41AM:  ^ Waiting for 1 unavailable replicas
        10:51:41AM:  L ok: waiting on replicaset/dep-5658f778d6 (apps/v1) namespace: kapp-test
        10:51:41AM:  L ongoing: waiting on pod/dep-5658f778d6-mcksg (v1) namespace: kapp-test
        10:51:41AM:     ^ Pending: ContainerCreating
        10:51:42AM: ---- waiting on 1 changes [2/3 done] ----
        10:52:42AM: ongoing: reconcile deployment/dep (apps/v1) namespace: kapp-test
        10:52:42AM:  ^ Waiting for 1 unavailable replicas
        10:52:42AM:  L ok: waiting on replicaset/dep-5658f778d6 (apps/v1) namespace: kapp-test
        10:52:42AM:  L ongoing: waiting on pod/dep-5658f778d6-mcksg (v1) namespace: kapp-test
        10:52:42AM:     ^ Pending: ContainerCreating
        10:52:43AM: ---- waiting on 1 changes [2/3 done] ----
        10:53:43AM: ongoing: reconcile deployment/dep (apps/v1) namespace: kapp-test
        10:53:43AM:  ^ Waiting for 1 unavailable replicas
        10:53:43AM:  L ok: waiting on replicaset/dep-5658f778d6 (apps/v1) namespace: kapp-test
        10:53:43AM:  L ongoing: waiting on pod/dep-5658f778d6-mcksg (v1) namespace: kapp-test
        10:53:43AM:     ^ Pending: ContainerCreating
        10:53:44AM: ---- waiting on 1 changes [2/3 done] ----
        10:54:44AM: ongoing: reconcile deployment/dep (apps/v1) namespace: kapp-test
        10:54:44AM:  ^ Waiting for 1 unavailable replicas
        10:54:44AM:  L ok: waiting on replicaset/dep-5658f778d6 (apps/v1) namespace: kapp-test
        10:54:44AM:  L ongoing: waiting on pod/dep-5658f778d6-mcksg (v1) namespace: kapp-test
        10:54:44AM:     ^ Pending: ContainerCreating
        10:54:45AM: ---- waiting on 1 changes [2/3 done] ----
        10:55:36AM: fail: reconcile deployment/dep (apps/v1) namespace: kapp-test

        ' stderr: 'Error: waiting on reconcile deployment/dep (apps/v1) namespace: kapp-test: finished unsuccessfully (Deployment is not progressing: ProgressDeadlineExceeded (message: ReplicaSet "dep-5658f778d6" has timed out progressing.))
        ' error: 'exit status 1'

When I describe the pod that is associated with this test I see the following:

Events:
  Type     Reason       Age                From                                                     Message
  ----     ------       ----               ----                                                     -------
  Normal   Scheduled    96s                default-scheduler                                        Successfully assigned kapp-test/dep-dc9bf645-bzrsp to gke-jims-playground-default-pool-477b1489-shzn
  Warning  FailedMount  32s (x8 over 96s)  kubelet, gke-jims-playground-default-pool-477b1489-shzn  MountVolume.SetUp failed for volume "vol1" : secrets "secret" not found

Is there some test setup that is necessary before running the end to end tests?

[cppforlife]

@jfmyers9 looks like it's related to thelatest change on develop (4db982f). im guessing e2e is shelling out and is seeing older binary and not one built from source code. (currently e2e tests just use kapp from the PATH).

[cppforlife]

(i have /usr/local/bin/kapp symlinked to ./kapp produced by ./hack/builld.sh; prolly not a good idea long term.)

[jfmyers9]

Makes sense. Seems like we could probably have the test-all.sh put kapp on the path before running the e2e suite.

[cppforlife]

... or just have e2e use ./kapp instead of kapp.

[cppforlife]

being addressed in the above PR. closing.