Add testing for minimal profile with knative

[howardjohn]

This is a step towards merging installer/operator into istio repo. In
order to do this, we parity on test coverage. The installer has
substantial testing of minimal profile, so its important we port this
over (including Knative minimal profile testing!).

This does make a change to the old helm templates to allow Ingress
controller using Pilot (not sure how this was not possible before...).
This is because we don't have all tests running using the new installer
yet, and ideally we want the same set of tests to run on old and new, so
I think it makes sense to make this change since its minimal and
undocumented.

I've also added some additional debug logging here, since its pretty useful to figure out why the test is sitting there doing nothing for 10 minutes.

[sdake]

In theory, generated code or manifests should be generated by make gen.

[sdake]

@howardjohn this file (to me) appears to be misnamed. Would you consider knative-crds.yaml?

[sdake]

knative-serving.yaml would be a better name here.

A make target to generate this testdata would be helpful as well. It isn't clear how this testdata was generated. If knative revs, how do we keep track of the new application-specific manifest?

e.g. make gen-test-app-manifests which is tied into make gen. Perhaps there could be a better target name.

[howardjohn]

I renamed and added a comment where it comes from.

I don't think we want/need/can this to be a part of gen, it is manually updated when required

[nmittler]

lgtm overall ... will approve after you address comments from @sdake

[nak3]

Just FYI, this annotation is dropped in knative serving as we control it by namespace's.

https://github.com/knative/serving/pull/6413/files#diff-6daded100a43545c2331ebe741ea6a92L31

[nak3]

This test looks good. But is it possible to expand the timeout (30sec as per logs)? I am wondering that deploying knative's system containers + application within 30sec is not enough.

2019-12-20T17:53:18.230177Z	info	CI	Applying YAML: kubectl apply --kubeconfig=/home/.kube/config -n knative-1-8102 -f /logs/artifacts/minimal-test-52926969dbae4b1b86/_suite_context/env-kube-768482011/istio-kube-accessor-987922885/accessor_206457236
2019-12-20T17:53:48.570496Z	info	CI	Deleting YAML file: /logs/artifacts/minimal-test-52926969dbae4b1b86/_suite_context/env-kube-768482011/istio-kube-accessor-987922885/accessor_860902883

[howardjohn]

This test looks good. But is it possible to expand the timeout (30sec as per logs)? I am wondering that deploying knative's system containers + application within 30sec is not enough.

Running this locally I saw when it failed it was permanent, ill try this with 3min timeout though

[howardjohn]

@nak3 with long timeout this still fails

[nak3]

@howardjohn Thank you. Timeout is no problem, then I am suspecting the test image (istio-testing/app:istio-testing).

Knative resolves image tags to a digest when creating a revision. You can refer to the docs or this slide for more detail. In fact, the image istio-testing/app:istio-testing cannot be resolved on my local env by this error:

$ kubectl create -f accessor_738217051
$ kubectl get event
  ...
1s          Warning   InternalError         revision/knative-service-mpqg2                           failed to resolve image to digest: failed to fetch image information: MANIFEST_UNKNOWN: Failed to fetch "istio-testing" from request "/v2/istio-testing/app/manifests/istio-testing".

For the solution, I could find gcr.io/istio-testing/app:latest in public gcr.io, so it worked fine by replacing the image as:

cat <<EOF | kubectl apply -f -
apiVersion: serving.knative.dev/v1alpha1
kind: Service
metadata:
  name: knative-service
spec:
  runLatest:
    configuration:
      revisionTemplate:
        spec:
          container:
            image: gcr.io/istio-testing/app:latest
EOF

Is it possible to test with image: gcr.io/istio-testing/app:latest? Otherwise, I think we need to check more pod's logs, event logs or yamls. (I could find some info in this directory but maybe that's all?)

[howardjohn]

maybe that is the issue. what we do is using kind, we directly load the image onto the node - it's not in any registry somewhere, which possibly breaks knative?

…

On Wed, Jan 15, 2020, 3:29 AM Kenjiro Nakayama ***@***.***> wrote: @howardjohn <https://github.com/howardjohn> Thank you. Timeout is no problem, then I am suspecting the test image (istio-testing/app:istio-testing) <https://storage.googleapis.com/istio-prow/pr-logs/pull/istio_istio/19675/integ-pilot-k8s-tests_istio/5673/artifacts/minimal-test-16f2ff5f428e4a1581/_suite_context/env-kube-829275960/istio-kube-accessor-303342890/accessor_738217051> . Knative resolves image tags to a digest when creating a revision. You can refer to the docs <https://knative.dev/docs/serving/tag-resolution/> or this slide <https://docs.google.com/presentation/d/1gjcVniYD95H1DmGM_n7dYJ69vD9d6KgJiA-D9dydWGU/edit#slide=id.p> for more detail. In fact, the image istio-testing/app:istio-testing cannot be resolved on my local env by this error: $ kubectl create -f accessor_738217051 $ kubectl get event ... 1s Warning InternalError revision/knative-service-mpqg2 failed to resolve image to digest: failed to fetch image information: MANIFEST_UNKNOWN: Failed to fetch "istio-testing" from request "/v2/istio-testing/app/manifests/istio-testing". I could find gcr.io/istio-testing/app:latest in public gcr.io, so it worked fine by replacing the image as: cat <<EOF | kubectl apply -f - apiVersion: serving.knative.dev/v1alpha1 kind: Service metadata: name: knative-service spec: runLatest: configuration: revisionTemplate: spec: container: image: gcr.io/istio-testing/app:latest EOF Is it possible to test with image: gcr.io/istio-testing/app:latest? Otherwise, I think we need to check more pod's logs, event logs or yamls. (I could find some info in this directory <https://gcsweb.istio.io/gcs/istio-prow/pr-logs/pull/istio_istio/19675/integ-pilot-k8s-tests_istio/5673/artifacts/minimal-test-16f2ff5f428e4a1581/_suite_context/env-kube-829275960/istio-kube-accessor-303342890/> but maybe that's all?) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#19675?email_source=notifications&email_token=AAEYGXNJOESLPFAFUFLY55TQ53XPTA5CNFSM4J4QFU6KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEI774VI#issuecomment-574619221>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAEYGXI3AJTHP7NOVWYQ2FTQ53XPTANCNFSM4J4QFU6A> .

[nak3]

Ah! it is a local image, then yes, it fails to deploy knative apps by default. You can refer to this thread especially this comment knative/serving#6101 (comment) for the instruction. You need additional configuration.

[howardjohn]

@nak3 I think i configured things correctly here but failing the same. Thanks for the reference though

[istio-testing]

@howardjohn: PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[istio-testing]

@howardjohn: The following tests failed, say /retest to rerun all failed tests:

Test name	Commit	Details	Rerun command
gencheck_istio	8c25f4c	link	/test gencheck_istio
integ-galley-k8s-tests_istio	8c25f4c	link	/test integ-galley-k8s-tests_istio
integ-pilot-k8s-tests_istio	8c25f4c	link	/test integ-pilot-k8s-tests_istio
integ-conformance-local-tests_istio	8c25f4c	link	/test integ-conformance-local-tests_istio
integ-conformance-k8s-tests_istio	8c25f4c	link	/test integ-conformance-k8s-tests_istio
integ-distroless-k8s-tests_istio	8c25f4c	link	/test integ-distroless-k8s-tests_istio
integ-local-tests_istio	8c25f4c	link	/test integ-local-tests_istio

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[istio-policy-bot]

🚧 This issue or pull request has been closed due to not having had activity from an Istio team member since 2020-01-16. If you feel this issue or pull request deserves attention, please reopen the issue. Please see this wiki page for more information. Thank you for your contributions.

Created by the issue and PR lifecycle manager.