Stackdriver logging sink failed to create on large clusters

[shyamjvs]

The "Cluster level logging implemented by Stackdriver should ingest system logs from all nodes" e2e test is failing on our 2k-node gce clusters - https://k8s-testgrid.appspot.com/google-gce-scale#gce-large-correctness with the following error:

failed to create Stackdriver Logging sink: googleapi: Error 400: Filter cannot be longer than 20000 characters., badRequest

Seems to be because of creating a filter of O(#nodes) size:

Using the following filter for log entries: resource.type="gce_instance" AND (resource.labels.instance_id=9022783563020170818 OR resource.labels.instance_id=623491093136995906 OR resource.labels.instance_id=4780682787538400833 OR resource.labels.instance_id=5936394307195696705 OR resource.labels.instance_id=6219227236601230913 OR resource.labels.instance_id=5239226076018570818 OR resource.labels.instance_id=4293549143014307394 OR resource.labels.instance_id=6628560181373236803 OR resource.labels.instance_id=1898100776565012035 OR .....

cc @crassirostris @kubernetes/sig-instrumentation-bugs @kubernetes/sig-scalability-misc

[shyamjvs]

This is part of the correctness suite and large cluster tests are release-blocking.

[piosz]

@x13n could you please take a look since @crassirostris is OOO?

cc @fgrzadkowski

[x13n]

I can find some time to check this today.

@shyamjvs Is it possible to reproduce locally?

[shyamjvs]

Yes, it should be reproducible on a large cluster. The issue is with filter string growing linearly with #nodes and then hitting the limit of 20000 chars of stackdriver api.
From the above, each node seems to add about 52B to the filter, so any cluster of >=385 nodes should see the problem.

[shyamjvs]

Alternatively if you don't want to create a large cluster, just artificially bloating it here https://github.com/kubernetes/kubernetes/blob/master/test/e2e/instrumentation/logging/stackdrvier/utils.go#L216 should also reproduce it.

[x13n]

I don't think we should filter by each node id individually, will try to see what will break if I just remove this.

[x13n]

After a couple of hours, the test is still running. I will get back to this on Monday.

[k8s-github-robot]

[MILESTONENOTIFIER] Milestone Labels Complete

@crassirostris @shyamjvs

Issue label settings:

• sig/instrumentation sig/scalability: Issue will be escalated to these SIGs if needed.
• priority/important-soon: Escalate to the issue owners and SIG owner; move out of milestone after several unsuccessful escalation attempts.
• kind/bug: Fixes a bug discovered during the current release.

Additional instructions available here

[crassirostris]

Hah! We need to set up a filter to separate one test running in the same project from another since there's no resource for the K8s node

I'll look into it

[crassirostris]

/cc @igorpeshansky @summit

Would really appreciate your help here

[igorpeshansky]

@crassirostris Would filtering by cluster name work?

[crassirostris]

@igorpeshansky problem is, docker and kubelet logs are written against the gce vm resource. There are two options: remove that filter and let all the logs flow, or make a prefix filter on the vm name

[igorpeshansky]

@crassirostris Ah, then can GCE labels or tags be used?

[crassirostris]

@igorpeshansky There's compute.googleapis.com/resource_name label, but the more I think about it, the less I like the idea. I'll just remove the filter, that shouldn't be a problem in this case, since vm ids are used for checking the logs presence anyway. Thanks for chiming in!

[shyamjvs]

Ref #51718