Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Istio not logging blocked traffic #17759

Closed
bturbes opened this issue Oct 10, 2019 · 5 comments · Fixed by #18541
Closed

Istio not logging blocked traffic #17759

bturbes opened this issue Oct 10, 2019 · 5 comments · Fixed by #18541

Comments

@bturbes
Copy link

@bturbes bturbes commented Oct 10, 2019

Bug description

The release notes for istio 1.3 say, “Added telemetry reporting for traffic destined to the Passthrough and BlackHole clusters”. Unless I am misunderstanding the purpose of the BlackHole cluster, when we have an outboundTrafficPolicy of REGISTRY_ONLY, blocked outgoing traffic should go to the BlackHole cluster and be logged. This older issue describes the same problem, but does not appear to have been fixed. #14110

Affected product area (please put an X in all that apply)

[ ] Configuration Infrastructure
[ ] Docs
[ ] Installation
[ ] Networking
[ ] Performance and Scalability
[x] Policies and Telemetry
[ ] Security
[ ] Test and Release
[ ] User Experience
[ ] Developer Infrastructure

Expected behavior
istio-proxy sidecar logs blocked external traffic

Steps to reproduce the bug
In a cluster with an outboundTrafficPolicy of REGISTRY_ONLY, make a curl request to anything (without a service entry allowing it). Request is blocked but not logged.

Version (include the output of istioctl version --remote and kubectl version)

$ istioctl version --remote
client version: unknown
citadel version: 1.3.1
citadel version: 1.3.1
egressgateway version: 1.3.1
egressgateway version: 1.3.1
galley version: 1.3.1
galley version: 1.3.1
ilbgateway version:
ilbgateway version:
ingressgateway version: 1.3.1
ingressgateway version: 1.3.1
pilot version: 1.3.1
pilot version: 1.3.1
policy version: 1.3.1
policy version: 1.3.1
sidecar-injector version: 1.3.1
sidecar-injector version: 1.3.1
telemetry version: 1.3.1
telemetry version: 1.3.1
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.6", GitCommit:"96fac5cd13a5dc064f7d9f4f23030a6aeface6cc", GitTreeState:"clean", BuildDate:"2019-08-19T11:13:49Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"13+", GitVersion:"v1.13.7-gke.19", GitCommit:"bebe882824db5431820e3d59851c8fb52cb41675", GitTreeState:"clean", BuildDate:"2019-07-26T00:09:47Z", GoVersion:"go1.11.5b4", Compiler:"gc", Platform:"linux/amd64"}

How was Istio installed?
Helm chart

Environment where bug was observed (cloud vendor, OS, etc)
GKE cluster

@howardjohn

This comment has been minimized.

Copy link
Member

@howardjohn howardjohn commented Oct 10, 2019

I think its in metrics only not logs. Is that right @nrjpoddar ?

@nrjpoddar

This comment has been minimized.

Copy link
Member

@nrjpoddar nrjpoddar commented Oct 10, 2019

Yes, we only added telemetry support. Just for clarity, when you mean logging, are you talking about Envoy or Mixer. Mixer should be both access logging and generating metrics for BlackHole.

It's a reasonable feature request to support Envoy access logging in all cases, if that's what you want.

@bturbes

This comment has been minimized.

Copy link
Author

@bturbes bturbes commented Oct 11, 2019

It sounds like I just misunderstood the release notes then.

Here's a hopefully more clear example of the behavior we're looking for. Let's say I have the following ServiceEntry.

apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: google-test
  namespace: default
spec:
  exportTo:
  - .
  hosts:
  - www.google.com
  location: MESH_EXTERNAL
  ports:
  - name: https
    number: 443
    protocol: TLS
  resolution: DNS

Then we can run a any pod in that namespace with curl on it.

kubectl -n prometheus run --generator=run-pod/v1 tmp-shell --rm -i --tty --image nicolaka/netshoot bash

From here if we run curl https://www.google.com/ we'll get a successful response and see the request logged in both istio-proxy and in mixer. If we run curl https://facebook.com/ the request will fail (as expected), but will not be logged in either istio-proxy or mixer.

@esauser

This comment has been minimized.

Copy link

@esauser esauser commented Oct 18, 2019

We are also looking to gain visibility on blocked traffic. Is this possible currently? If not, can it be added?

@mdhume

This comment has been minimized.

Copy link
Contributor

@mdhume mdhume commented Oct 23, 2019

@nrjpoddar We are running into a similar issue. The issue comes when you have a listener such as 0.0.0.0_443 with multiple filter chains with a filter chain match on server_names - https://gist.github.com/mdhume/bf262a5ec426d37e6aa94ddfeb328760 . If a request comes on a domain that is not in the filter chain match e.g. www.cnn.com then even though the correct listener gets matched, it does not match any filter chain and hence the request gets dropped. Such requests are not getting reported any where - not in access logs nor in metrics.

@nrjpoddar nrjpoddar self-assigned this Oct 23, 2019
nrjpoddar added a commit to nrjpoddar/istio that referenced this issue Nov 2, 2019
istio-testing added a commit that referenced this issue Nov 4, 2019
* Add fallthrough listener filer chain for BlackHoleCluster

* Fix telemetry reported

Fixes: #17271 #17759

* Tests for IsAllowAnyOutbound

* Use t.Run

* Added mixer plugin tests

* Added listener tests
istio-testing added a commit to istio-testing/istio that referenced this issue Nov 4, 2019
nrjpoddar added a commit to nrjpoddar/istio that referenced this issue Nov 4, 2019
* Add fallthrough listener filer chain for BlackHoleCluster

* Fix telemetry reported

Fixes: istio#17271 istio#17759

* Tests for IsAllowAnyOutbound

* Use t.Run

* Added mixer plugin tests

* Added listener tests
istio-testing added a commit that referenced this issue Nov 5, 2019
…er (#18619)

* Add fallthrough listener filer chain for BlackHoleCluster

* Fix telemetry reported

Fixes: #17271 #17759

* Tests for IsAllowAnyOutbound

* Use t.Run

* Added mixer plugin tests

* Added listener tests
nrjpoddar added a commit to nrjpoddar/istio that referenced this issue Nov 6, 2019
* Add fallthrough listener filer chain for BlackHoleCluster

* Fix telemetry reported

Fixes: istio#17271 istio#17759

* Tests for IsAllowAnyOutbound

* Use t.Run

* Added mixer plugin tests

* Added listener tests
istio-testing added a commit that referenced this issue Nov 6, 2019
…18620)

* Add fallthrough listener filer chain for BlackHoleCluster (#18541)

* Add fallthrough listener filer chain for BlackHoleCluster

* Fix telemetry reported

Fixes: #17271 #17759

* Tests for IsAllowAnyOutbound

* Use t.Run

* Added mixer plugin tests

* Added listener tests

* Fix unit tests

* Fix unit tests

* Fix lint
sdake added a commit to sdake/istio that referenced this issue Dec 1, 2019
* Add fallthrough listener filer chain for BlackHoleCluster

* Fix telemetry reported

Fixes: istio#17271 istio#17759

* Tests for IsAllowAnyOutbound

* Use t.Run

* Added mixer plugin tests

* Added listener tests
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
6 participants
You can’t perform that action at this time.