Pilot needs better validation of XDS responses

[howardjohn]

Context: #14414

Pilot does some validation of XDS responses:

istio/pilot/pkg/proxy/envoy/v2/lds.go

Lines 59 to 67 in 1d13d1f

	if err = l.Validate(); err != nil {
	retErr := fmt.Errorf("LDS: Generated invalid listener for node %v: %v", con.modelNode, err)
	adsLog.Errorf("LDS: Generated invalid listener for node:%s: %v, %v", con.modelNode.ID, err, l)
	pushes.With(prometheus.Labels{"type": "lds_builderr"}).Add(1)
	// Generating invalid listeners is a bug.
	// Panic instead of trying to recover from that, since we can't
	// assume anything about the state.
	panic(retErr.Error())
	}

There are two problems:

• Most of the config is already serialized to struct/any. This will NOT be validated. (Actually, maybe if we do it to any it does, but I doubt it. Struct doesn't for sure). So we have a false sense of confidence that the responses are valid
• When we do detect something is invalid, we shouldn't panic. Instead we should drop the single item that failed in the response. Another option is dropping the whole response. This depends if we would rather be atomic in our failures or just minimize the impact of the failures

[howardjohn]

More info, ref https://github.com/istio/istio/pull/14854/files. We do Validate() in duplicated locations sometimes

[howardjohn]

We no longer panic