Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pod info is missing in data generated by mixer when pod name contains '.' #20028

Closed
zhranklin opened this issue Jan 9, 2020 · 2 comments
Closed

Comments

@zhranklin
Copy link
Member

@zhranklin zhranklin commented Jan 9, 2020

Bug description
When two pods with '.' in their names are injected sidecars and then make http request from one to another, the generated mixer data is missing some fields which depends on pod info, which includes:

  • source/destination.labels
  • source/destination.name
  • source/destination.owner
  • source/destination.workload
  • destination.namespace (parsed as default but it's wrong)

When making a request from a.deploy.powerful-mix to b.deploy.powerful-mix, we can't get information of this request from prometheus or kiali, and the access log on istio-telemetry is:

2020-01-09T11:53:40.645249Z     info    accesslog.instance.istio-system {"apiClaims": "", "apiKey": "", "clientTraceId": "", "connection_security_policy": "unknown", "destinationApp": "", "destinationIp": "10.244.2.200", "destinationName": "unknown", "destinationNamespace": "default", "destinationOwner": "unknown", "destinationPrincipal": "", "destinationServiceHost": "b.powerful-mix.svc.cluster.local", "destinationWorkload": "unknown", "grpcMessage": "", "grpcStatus": "", "httpAuthority": "b", "latency": "7.100119ms", "method": "POST", "permissiveResponseCode": "none", "permissiveResponsePolicyID": "none", "protocol": "http", "receivedBytes": 712, "referer": "", "reporter": "source", "requestId": "938fb7c0-5c3f-4958-8371-0030cc00d7da", "requestSize": 233, "requestedServerName": "", "responseCode": 200, "responseFlags": "-", "responseSize": 10, "responseTimestamp": "2020-01-09T11:53:40.652286Z", "sentBytes": 146, "sourceApp": "", "sourceIp": "0.0.0.0", "sourceName": "unknown", "sourceNamespace": "deploy-686ff98446-5xdvj", "sourceOwner": "unknown", "sourcePrincipal": "", "sourceWorkload": "unknown", "url": "/execute", "userAgent": "Java/1.8.0_222", "xForwardedFor": "0.0.0.0"}

The json is formatted as:

{
  "apiClaims": "",
  "apiKey": "",
  "clientTraceId": "",
  "connection_security_policy": "unknown",
  "destinationApp": "",
  "destinationIp": "10.244.2.200",
  "destinationName": "unknown",
  "destinationNamespace": "default",
  "destinationOwner": "unknown",
  "destinationPrincipal": "",
  "destinationServiceHost": "b.powerful-mix.svc.cluster.local",
  "destinationWorkload": "unknown",
  "grpcMessage": "",
  "grpcStatus": "",
  "httpAuthority": "b",
  "latency": "7.100119ms",
  "method": "POST",
  "permissiveResponseCode": "none",
  "permissiveResponsePolicyID": "none",
  "protocol": "http",
  "receivedBytes": 712,
  "referer": "",
  "reporter": "source",
  "requestId": "938fb7c0-5c3f-4958-8371-0030cc00d7da",
  "requestSize": 233,
  "requestedServerName": "",
  "responseCode": 200,
  "responseFlags": "-",
  "responseSize": 10,
  "responseTimestamp": "2020-01-09T11:53:40.652286Z",
  "sentBytes": 146,
  "sourceApp": "",
  "sourceIp": "0.0.0.0",
  "sourceName": "unknown",
  "sourceNamespace": "deploy-686ff98446-5xdvj",
  "sourceOwner": "unknown",
  "sourcePrincipal": "",
  "sourceWorkload": "unknown",
  "url": "/execute",
  "userAgent": "Java/1.8.0_222",
  "xForwardedFor": "0.0.0.0"
}

As we know, the default logentry config shows mappings of the fields:

      sourceApp: source.labels["{{ .Values.global.appLabelName }}"] | ""
      sourceName: source.name | ""
      sourceWorkload: source.workload.name | ""
      sourceOwner: source.owner | ""
      destinationApp: destination.labels["{{ .Values.global.appLabelName }}"] | ""
      destinationWorkload: destination.workload.name | ""
      destinationName: destination.name | ""
      destinationNamespace: destination.namespace | ""
      destinationOwner: destination.owner | ""

So we can infer the above attributes are missing or incorrectly parsed.

Expected behavior
The mixer access log should show the right information of source workload and destination workload.

And we should be able to see this request on kiali and prometheus.

Steps to reproduce the bug

  • create 2 deployments named a.deploy and b.deploy
  • make a request from a.deploy-xxx-xxx to b.deploy-xxx-xxx
  • see access log on mixer and the result in kiali and prometheus

Version (include the output of istioctl version --remote and kubectl version and helm version if you used Helm)

1.3.x

How was Istio installed?

installed with config for mixer:

mixer:
  telemetry:
    enabled: true
  adapters:
    stdio:
      enabled: true
      outputAsJson: false

Environment where bug was observed (cloud vendor, OS, etc)
Linux

@zhranklin

This comment has been minimized.

Copy link
Member Author

@zhranklin zhranklin commented Jan 9, 2020

I have fixed this bug in our production env, and try to make a PR: #20026

@jwendell

This comment has been minimized.

Copy link
Member

@jwendell jwendell commented Feb 10, 2020

Fixed by #20026

@jwendell jwendell closed this Feb 10, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
3 participants
You can’t perform that action at this time.