Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Validate Kiali against Telemetry v2 in Istio 1.5 #1894

Closed
israel-hdez opened this issue Nov 6, 2019 · 10 comments
Closed

Validate Kiali against Telemetry v2 in Istio 1.5 #1894

israel-hdez opened this issue Nov 6, 2019 · 10 comments

Comments

@israel-hdez
Copy link
Member

@israel-hdez israel-hdez commented Nov 6, 2019

Telemetry V2 is aka Mixer-less telemetry

Istio's P&T WG is asking to actively test Kiali on Telemetry v2 (and they may "broadcast" a similar message to other add-ons).

It's expected that Telemetry v2 will reach enough maturity to be enabled by default in a following release (after Istio v1.4).

Telemetry v2 metrics for HTTP are almost at par with Telemetry v1. So, we must make sure that all Kiali features dealing with HTTP metrics are working successfully.

Telemetry v2 metrics for TCP aren't at par yet, but they said it's very close. Kiali should be validated and fixed, if needed.

Once Kiali is validated and fixed, since Telemetry v2 is close to become the default, testing Kaili in both Telemetry V1 and V2 should become the usual validation to ensure compatibility with the following release.

Related to previous #1694.

@jshaughn jshaughn added this to Backlog in Sprint 31 via automation Nov 6, 2019
@jshaughn jshaughn added backlog and removed discussion needed labels Nov 6, 2019
@jshaughn

This comment has been minimized.

Copy link
Contributor

@jshaughn jshaughn commented Nov 13, 2019

I've tried mixer-less (v2) telemetry with istio 1.4-beta4. The issue we reported against 1.3 is fixed and so v2 actually seems usable for HTTP traffic (TCP is still not available). I do see two issues:

  • In the prelim docs (https://preliminary.istio.io/docs/ops/telemetry/in-proxy-service-telemetry) the filter file locations are wrong. You want:
    • install/kubernetes/operator/charts/istio-control/istio-discovery/metadata-exchange-v2.yaml
    • install/kubernetes/operator/charts/istio-control/istio-discovery/stats-filter-v2.yaml
  • Traffic to PassthroughCluster is being reported with destination 'unknown instead of 'PassthroughCluster'. I haven't tested BlackHoleCluster, perhaps it has the same issue.
@jshaughn jshaughn self-assigned this Nov 13, 2019
@jshaughn jshaughn moved this from Backlog to In Progress in Sprint 31 Nov 13, 2019
@lucasponce lucasponce removed this from In Progress in Sprint 31 Nov 22, 2019
@lucasponce lucasponce added this to Backlog in Sprint 32 via automation Nov 22, 2019
@jshaughn

This comment has been minimized.

Copy link
Contributor

@jshaughn jshaughn commented Dec 2, 2019

The Istio 1.4 offers TCP metrics as an experimental feature, HTTP has been upgraded to alpha. We need to test the experimental TCP impl.

@lucasponce lucasponce removed this from Backlog in Sprint 32 Dec 13, 2019
@lucasponce lucasponce added this to Backlog in Sprint 33 via automation Dec 13, 2019
@lucasponce lucasponce removed this from Backlog in Sprint 33 Jan 10, 2020
@lucasponce lucasponce added this to Backlog in Sprint 34 via automation Jan 10, 2020
@jshaughn jshaughn moved this from Backlog to In Progress in Sprint 34 Jan 20, 2020
@jshaughn

This comment has been minimized.

Copy link
Contributor

@jshaughn jshaughn commented Jan 21, 2020

Although the 1.4 release doc states,

We’ve also added new experimental features that don’t require Mixer: TCP metrics and Stackdriver metrics.

In Slack I was told that mixer-less TCP support is actually coming in Istio 1.5. I did validate that HTTP support (now Alpha) seems to still work fine in 1.4. I think the intent is to make V2 Telemetry the default in 1.5, but considering the TCP support did not make 1.4, I'm not sure tyhe new V2 default will actually happen.

1.4 config change for V2:
https://istio.io/docs/ops/configuration/telemetry/in-proxy-service-telemetry/

@jshaughn jshaughn changed the title Validate Kiali against Telemetry v2 Validate Kiali against Telemetry v2 in Istio 1.5 Jan 21, 2020
@jshaughn jshaughn moved this from In Progress to Backlog in Sprint 34 Jan 21, 2020
@jshaughn

This comment has been minimized.

Copy link
Contributor

@jshaughn jshaughn commented Jan 24, 2020

It looks like this GRPC status PR will be merged in 1.5. We will need to make updates to make use of the change as the grpc status will be reported in a new field, not in the current response_code field.

@lucasponce lucasponce added this to Backlog in Sprint 35 via automation Feb 3, 2020
@lucasponce lucasponce removed this from Backlog in Sprint 34 Feb 3, 2020
@jshaughn

This comment has been minimized.

Copy link
Contributor

@jshaughn jshaughn commented Feb 13, 2020

Notes from P&T:

In Istio 1.5, the default HTTP telemetry is generated by the in-proxy Stats filter (i.e. Telem V2). This represents a 50% reduction in total CPU usage. Telemetry V2 in Istio 1.5 has the following major feature gaps and differences vis-a-vis Mixer Telemetry.

  • Out of mesh telemetry
    • When the source or the destination of the traffic is not side-car injected, some telemetry information is missing.
  • Egress Telemetry is not supported.
  • Histogram bucketization in V2 telemetry is significantly different than Mixer.
    • WE Hanve already added backward compatible support.
  • TCP telemetry is only supported with mTLS.
  • Needs a larger Prometheus instance than Mixer V1 since all proxies must be scraped.
@lucasponce

This comment has been minimized.

Copy link
Contributor

@lucasponce lucasponce commented Feb 13, 2020

So, perhaps we need to put a note for Kiali 1.13 (the target for Istio 1.5) that would require Mixer Telemetry.

Isn't it ?

@jotak

This comment has been minimized.

Copy link
Contributor

@jotak jotak commented Feb 13, 2020

* TCP telemetry is only supported with mTLS.

this one in particular sounds really weird, do you know if there's an issue to track that?

@jshaughn

This comment has been minimized.

Copy link
Contributor

@jshaughn jshaughn commented Feb 17, 2020

@lucasponce No, we certainly don't require Mixer telemetry, we support Telemetry V2. But of course any feature-gaps/telemetry-reporting-changes will be reflected in Kiali because the underlying reporting will have changed. I think it would be helpful if we actually had release notes so that we could just pass on that fact to our users, and maybe reduce the number of Kiali issues reported.

@jotak That TCP note is significant, I guess, although I'm not sure if TCP is used in most cases and therefore impact will be small. Maybe I'll ask that on Slack...

@jshaughn jshaughn moved this from Backlog to In Progress in Sprint 35 Feb 17, 2020
@jshaughn jshaughn removed this from In Progress in Sprint 35 Feb 21, 2020
@jshaughn jshaughn added this to Backlog in Sprint 36 via automation Feb 21, 2020
@jshaughn jshaughn moved this from Backlog to In Progress in Sprint 36 Feb 21, 2020
@jshaughn

This comment has been minimized.

Copy link
Contributor

@jshaughn jshaughn commented Feb 21, 2020

Filed this bug when testing 1.5: istio/istio#21324

It does not affect Kiali because we don't yet use the canonical_service info. Added #2276 for adding that support.

@lucasponce lucasponce mentioned this issue Feb 21, 2020
0 of 4 tasks complete
@jshaughn

This comment has been minimized.

Copy link
Contributor

@jshaughn jshaughn commented Mar 6, 2020

Closing, 1.5 is released and I think we will have finished addressing the upstream changes with the Kiali 1.15 release. Although, Kiali 1.14 is packaged with 1.5 and is functional.

@jshaughn jshaughn closed this Mar 6, 2020
Sprint 36 automation moved this from In Progress to Done Mar 6, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Sprint 36
  
Done
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
4 participants
You can’t perform that action at this time.