New probe handling in Queue-Proxy & Activator #5159
Conversation
knative-prow-robot
added
area/networking
size/L
There was a problem hiding this comment.
@JRBANCEL: 0 warnings.
In response to this:
Part 1 & 2 & 3 of #5156.
Proposed Changes
I am not 100% sure why, but today Activator and Queue-Proxy care if the a probe request was sent to them specifically. For #5156, we need to be able to probe Envoy though the real data path (eventually reaching either Activator or Queue-Proxy) without distinction. This change introduces a new probe handler at the head of the handlers pipeline doing just that as well as a new header that will be populated by Envoy and used to find out which version of the VirtualService/Gateway Envoy is using.
Next
Maybe take a holistic look at probing in all components and consolidate/refactor.
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.
|
/assign @vagababov |
| @@ -242,6 +242,7 @@ func main() { | |||
| ah = &activatorhandler.HealthHandler{HealthCheck: statSink.Status, NextHandler: ah} | |||
| // NOTE: MetricHandler is being used as the outermost handler for the purpose of measuring the request latency. | |||
| ah = activatorhandler.NewMetricHandler(revisionInformer.Lister(), reporter, logger, ah) | |||
| ah = network.NewProbeHandler(ah) | |||
There was a problem hiding this comment.
Is this one going to replace activatorhandler.ProbeHandler in future?
And please make the metric handler as the outermost one.
There was a problem hiding this comment.
No, both probing are different.
See my other comment about the metrics handler.
| @@ -388,6 +388,7 @@ func main() { | |||
| composedHandler = pushRequestMetricHandler(composedHandler, requestCountM, responseTimeInMsecM, env) | |||
| } | |||
| composedHandler = tracing.HTTPSpanMiddleware(composedHandler) | |||
| composedHandler = network.NewProbeHandler(composedHandler) | |||
There was a problem hiding this comment.
Is this one going to replace the probe requests check part in func handler in future?
And same for the handlers order. I don't know why currently metric handler is not the outermost one but it should be so that the latency is the time added by all handlers.
There was a problem hiding this comment.
Queue-Proxy probing is different than this probing, so I don't think they'll ever be merged.
Regarding the ordering, you can argue that tracing also needs to be the outer most.
But all these handlers (metrics, tracing, probing) are not all doing sub-millisecond work, plus the metrics handler already filters out probing requests, so it doesn't matter here.
|
As I described earlier: originally the k8s service that was used was the same and this was checking for network re-programming. Currently we do actually use |
|
Also what is the problem with systems verifying that the probe was targeted to them specifically? |
Because here we want to probe through Envoy (to know which version Envoy is using), and there is no way to know where Envoy will sent the request, Queue-Proxy or Activator, and it doesn't matter. |
|
Can we change the existing probe handlers to be more relaxed about headers (perhaps accept |
While As I said in the Next section. I will revisit and eventually merge what makes sense, but right now, it is not obvious. Small handlers is cleaner than more generic complex handlers. |
JRBANCEL
force-pushed
the
probing-handling
branch
from
796d692
to
dc6ffc0
Compare
|
All (the pull request submitter and all commit authors) CLAs are signed, but one or more commits were authored or co-authored by someone other than the pull request submitter. We need to confirm that all authors are ok with their commits being contributed to this project. Please have them confirm that by leaving a comment that contains only Note to project maintainer: There may be cases where the author cannot leave a comment, or the comment is not properly detected as consent. In those cases, you can manually confirm consent of the commit author(s), and set the ℹ️ Googlers: Go here for more info. |
googlebot
added
cla: no
JRBANCEL
force-pushed
the
probing-handling
branch
from
8c7e9bf
to
baa0f32
Compare
|
CLAs look good, thanks! ℹ️ Googlers: Go here for more info. |
googlebot
added
cla: yes
|
The following tests are currently flaky. Running them again to verify...
Automatically retrying... |
knative-prow-robot
added
do-not-merge/hold
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: JRBANCEL, vagababov The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
knative-prow-robot
added
the
approved
JRBANCEL
force-pushed
the
probing-handling
branch
from
baa0f32
to
9d515a4
Compare
|
/hold cancel |
knative-prow-robot
removed
the
do-not-merge/hold
|
/lgtm |
Part 1 & 2 & 3 of #5156.
Proposed Changes
I am not 100% sure why, but today Activator and Queue-Proxy care if the a probe request was sent to them specifically. For #5156, we need to be able to probe Envoy though the real data path (eventually reaching either Activator or Queue-Proxy) without distinction. This change introduces a new probe handler at the head of the handlers pipeline doing just that as well as a new header that will be populated by Envoy and used to find out which version of the VirtualService/Gateway Envoy is using.
Next
Maybe take a holistic look at probing in all components and consolidate/refactor.