Probe Envoy pods to determine when a ClusterIngress is actually deployed #4734

JRBANCEL · 2019-07-12T17:50:49Z

Proposed Changes

See the design doc for details.

TODO

Working implementation
Fix existing tests
Add tests

File	Old Coverage	New Coverage	Delta
pkg/apis/networking/v1alpha1/ingress_lifecycle.go	91.7%	73.3%	-18.3
pkg/network/prober/prober.go	97.7%	91.3%	-6.4
pkg/reconciler/ingress/ingress.go	Do not exist	0.0%
pkg/reconciler/ingress/resources/virtual_service.go	97.6%	95.9%	-1.7
pkg/reconciler/ingress/status.go	Do not exist	0.0%

vagababov

Mostly superficial items.

pkg/apis/networking/v1alpha1/ingress_lifecycle.go

pkg/reconciler/ingress/resources/virtual_service.go

pkg/reconciler/ingress/status.go

markusthoemmes · 2019-07-15T18:22:49Z

pkg/reconciler/ingress/resources/virtual_service.go

@@ -140,6 +176,37 @@ func makeVirtualServiceSpec(ia v1alpha1.IngressAccessor, gateways map[v1alpha1.I
 	return &spec
 }

+func makeProbeRoute(host string) v1alpha3.HTTPRoute {


Why do we need to add a dummy route? Couldn't we also use our probe? It'll either return activator or queue-proxy but we'd know it works without side-effects I think? Is this to keep the layering strict?

Why do we need to add a dummy route?

It is an invalid Spec without it.

Couldn't we also use our probe? It'll either return activator or queue-proxy but we'd know it works without side-effects I think? Is this to keep the layering strict?

We could. It is simply because it is simpler, has lower latency and more reliable.

Is it guaranteed that the fake route is propageted on the same paths as the actual route? As in: Can we guarantee the actual route is ready and propagated to envoys if we probe the fake route?

We can double check the Istio implementation.
I am assuming that yes, since both routes are bound to the same Istio Gateway, they will be both applied to the same Envoy pods.

An interesting related question is how does Istio apply the config changes?
If it is atomic, good. If not, it is possible that the probe route is applied before the actual route.

pkg/reconciler/ingress/resources/virtual_service.go

knative-prow-robot

@JRBANCEL: 1 warning.

In response to this:

/lint
/test pull-knative-serving-go-coverage

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

pkg/reconciler/ingress/status.go

knative-prow-robot

@JRBANCEL: 1 warning.

In response to this:

/lint
/test pull-knative-serving-go-coverage

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

pkg/reconciler/ingress/status.go

knative-prow-robot

@JRBANCEL: 0 warnings.

In response to this:

/lint
/test pull-knative-serving-go-coverage

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

JRBANCEL · 2019-07-17T02:41:14Z

/test pull-knative-serving-integration-tests

markusthoemmes

Should we remove the retries on 404s to assert that this really works as intended?

See

serving/test/v1alpha1/route.go

Lines 77 to 79 in bcd9277

    
           if resp.StatusCode == http.StatusNotFound { 
        
           	return false, nil 
        
           }

and the respective file in v1beta1.

if you choose to do that: Please leave the helper there even though it doesn't do anything currently. We can use it as a convenient hook until we have baked the networking flakiness some more and are sure we can take the extra wheels off.

JRBANCEL · 2019-07-17T17:05:09Z

Should we remove the retries on 404s to assert that this really works as intended?

Yes, it is planned. See last item in the TODO.
I'll most likely do it in another PR because it is a bit more complex than just removing the retries for 404.

mattmoor · 2019-07-28T16:33:58Z

Needs a rebase.

mattmoor

Seems like this still has a bunch of debugging stuff in it?

pkg/apis/serving/v1alpha1/service_lifecycle_test.go

pkg/apis/serving/v1beta1/service_defaults_test.go

mattmoor · 2019-07-28T16:39:04Z

pkg/reconciler/clusteringress/clusteringress.go

+	resyncIngressOnVirtualServiceReady := func(vs *v1alpha3.VirtualService) {
+		// Reconcile when a VirtualService becomes ready
+		impl.EnqueueLabelOfClusterScopedResource(networking.ClusterIngressLabelKey)(vs)
+	}


Could this just be:

resyncIngressOnVirtualServiceReady := impl.EnqueueLabelOfClusterScopedResource(networking.ClusterIngressLabelKey)

Unfortunately, EnqueueLabelOfClusterScopedResource is func(obj interface{}), resyncIngressOnVirtualServiceReady expects a func(*v1alpha3.VirtualService)

JRBANCEL · 2019-07-29T20:34:56Z

@tcnghia, @mattmoor it is ready to be approved/lgtm 🚀

/test pull-knative-serving-istio-1.1-mesh
/test pull-knative-serving-istio-1.1-no-mesh
/test pull-knative-serving-istio-1.2-mesh
/test pull-knative-serving-istio-1.2-no-mesh

tcnghia

/approve

There is one left over from merge, but otherwise /lgtm

pkg/reconciler/ingress/resources/virtual_service.go

JRBANCEL · 2019-07-29T20:58:03Z

/test pull-knative-serving-istio-1.1-mesh
/test pull-knative-serving-istio-1.1-no-mesh
/test pull-knative-serving-istio-1.2-mesh
/test pull-knative-serving-istio-1.2-no-mesh

mattmoor · 2019-07-29T21:37:26Z

/lgtm
/approve

Let's get it baking.

knative-prow-robot · 2019-07-29T21:37:35Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: JRBANCEL, mattmoor, tcnghia

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [mattmoor]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

JRBANCEL · 2019-07-29T21:49:07Z

/hold cancel 🚀

tcnghia · 2019-07-29T21:49:43Z

/hold cancel

vagababov · 2019-07-29T22:15:57Z

🚀

knative-prow-robot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Jul 12, 2019

googlebot added the cla: yes Indicates the PR's author has signed the CLA. label Jul 12, 2019

knative-prow-robot reviewed Jul 12, 2019

View reviewed changes

knative-prow-robot added area/API API objects and controllers area/networking labels Jul 12, 2019

knative-prow-robot requested review from lichuqiang, mdemirhan, tcnghia and vagababov July 12, 2019 17:51

vagababov reviewed Jul 13, 2019

View reviewed changes

markusthoemmes reviewed Jul 15, 2019

View reviewed changes

JRBANCEL force-pushed the cluster-ingress-status branch from e3906a2 to 11432e2 Compare July 15, 2019 21:34

tcnghia reviewed Jul 15, 2019

View reviewed changes

pkg/reconciler/ingress/resources/virtual_service.go Outdated Show resolved Hide resolved

knative-prow-robot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Jul 16, 2019

JRBANCEL force-pushed the cluster-ingress-status branch from 82db439 to 9b4c24d Compare July 16, 2019 23:07

knative-prow-robot reviewed Jul 16, 2019

View reviewed changes

pkg/reconciler/ingress/status.go Show resolved Hide resolved

knative-prow-robot reviewed Jul 16, 2019

View reviewed changes

pkg/reconciler/ingress/status.go Outdated Show resolved Hide resolved

knative-prow-robot reviewed Jul 16, 2019

View reviewed changes

JRBANCEL changed the title ~~[WIP] Probe Envoy pods to determine when a ClusterIngress is actually deployed~~ Probe Envoy pods to determine when a ClusterIngress is actually deployed Jul 16, 2019

knative-prow-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jul 16, 2019

markusthoemmes reviewed Jul 17, 2019

View reviewed changes

knative-prow-robot added area/test-and-release It flags unit/e2e/conformance/perf test issues for product features do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. labels Jul 17, 2019

JRBANCEL force-pushed the cluster-ingress-status branch from 03135d0 to 754a463 Compare July 17, 2019 20:09

knative-prow-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jul 17, 2019

JRBANCEL force-pushed the cluster-ingress-status branch 2 times, most recently from 5467969 to be182f5 Compare July 25, 2019 22:43

knative-prow-robot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Jul 25, 2019

mattmoor reviewed Jul 28, 2019

View reviewed changes

JRBANCEL force-pushed the cluster-ingress-status branch from be182f5 to 85386cc Compare July 29, 2019 17:35

knative-prow-robot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Jul 29, 2019

Implement (Cluster)Ingress Status.

e8b03d7

JRBANCEL force-pushed the cluster-ingress-status branch from 85386cc to e8b03d7 Compare July 29, 2019 20:33

tcnghia approved these changes Jul 29, 2019

View reviewed changes

pkg/reconciler/ingress/resources/virtual_service.go Outdated Show resolved Hide resolved

Remove dead code.

3912db4

knative-prow-robot assigned mattmoor Jul 29, 2019

knative-prow-robot added the lgtm Indicates that a PR is ready to be merged. label Jul 29, 2019

knative-prow-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jul 29, 2019

knative-prow-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jul 29, 2019

knative-prow-robot merged commit cbe9951 into knative:master Jul 29, 2019

This was referenced Jul 29, 2019

Brief 404 errors after creating a route. #3312

Closed

Remove 404 retry logic in E2E tests #4976

Merged

Enable autoscale for Istio gateways in E2E tests #5064

Merged

This was referenced Aug 14, 2019

Ignore Gateways that cannot be probed (yet). #5149

Merged

Extend VirtualService/Gateway probing to HTTPS #5156

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Probe Envoy pods to determine when a ClusterIngress is actually deployed #4734

Probe Envoy pods to determine when a ClusterIngress is actually deployed #4734

JRBANCEL commented Jul 12, 2019 •

edited

Loading

knative-prow-robot left a comment

knative-metrics-robot commented Jul 12, 2019

vagababov left a comment

markusthoemmes Jul 15, 2019

JRBANCEL Jul 15, 2019 •

edited

Loading

markusthoemmes Jul 15, 2019

JRBANCEL Jul 15, 2019 •

edited

Loading

knative-prow-robot left a comment

knative-prow-robot left a comment

knative-prow-robot left a comment

JRBANCEL commented Jul 17, 2019

markusthoemmes left a comment •

edited

Loading

JRBANCEL commented Jul 17, 2019 •

edited

Loading

mattmoor commented Jul 28, 2019

mattmoor left a comment

mattmoor Jul 28, 2019

JRBANCEL Jul 29, 2019

JRBANCEL commented Jul 29, 2019

tcnghia left a comment

JRBANCEL commented Jul 29, 2019

mattmoor commented Jul 29, 2019

knative-prow-robot commented Jul 29, 2019

JRBANCEL commented Jul 29, 2019

tcnghia commented Jul 29, 2019

vagababov commented Jul 29, 2019

	if resp.StatusCode == http.StatusNotFound {
	return false, nil
	}

Probe Envoy pods to determine when a ClusterIngress is actually deployed #4734

Probe Envoy pods to determine when a ClusterIngress is actually deployed #4734

Conversation

JRBANCEL commented Jul 12, 2019 • edited Loading

Proposed Changes

TODO

Next

knative-prow-robot left a comment

Choose a reason for hiding this comment

Proposed Changes

TODO

knative-metrics-robot commented Jul 12, 2019

vagababov left a comment

Choose a reason for hiding this comment

markusthoemmes Jul 15, 2019

Choose a reason for hiding this comment

JRBANCEL Jul 15, 2019 • edited Loading

Choose a reason for hiding this comment

markusthoemmes Jul 15, 2019

Choose a reason for hiding this comment

JRBANCEL Jul 15, 2019 • edited Loading

Choose a reason for hiding this comment

knative-prow-robot left a comment

Choose a reason for hiding this comment

knative-prow-robot left a comment

Choose a reason for hiding this comment

knative-prow-robot left a comment

Choose a reason for hiding this comment

JRBANCEL commented Jul 17, 2019

markusthoemmes left a comment • edited Loading

Choose a reason for hiding this comment

JRBANCEL commented Jul 17, 2019 • edited Loading

mattmoor commented Jul 28, 2019

mattmoor left a comment

Choose a reason for hiding this comment

mattmoor Jul 28, 2019

Choose a reason for hiding this comment

JRBANCEL Jul 29, 2019

Choose a reason for hiding this comment

JRBANCEL commented Jul 29, 2019

tcnghia left a comment

Choose a reason for hiding this comment

JRBANCEL commented Jul 29, 2019

mattmoor commented Jul 29, 2019

knative-prow-robot commented Jul 29, 2019

JRBANCEL commented Jul 29, 2019

tcnghia commented Jul 29, 2019

vagababov commented Jul 29, 2019

JRBANCEL commented Jul 12, 2019 •

edited

Loading

JRBANCEL Jul 15, 2019 •

edited

Loading

JRBANCEL Jul 15, 2019 •

edited

Loading

markusthoemmes left a comment •

edited

Loading

JRBANCEL commented Jul 17, 2019 •

edited

Loading