Potential NPE in multi destination translation loop

[kdorosh]

Describe the bug
Sample NPE

{"level":"info","ts":1628022930.8266501,"logger":"gloo-ee.v1.event_loop.setup.v1.event_loop.envoyTranslatorSyncer.translator","caller":"translator/translator.go:167","msg":"computing envoy resources for listener: listener-::-4443","version":"1.6.31"}
{"level":"info","ts":1628022930.8303745,"logger":"gloo-ee.v1.event_loop.setup.v1.event_loop.envoyTranslatorSyncer","caller":"runtime/panic.go:975","msg":"end sync 12766681374198960840","version":"1.6.31"}
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x288125d]
goroutine 1598 [running]:
github.com/solo-io/gloo/projects/gloo/pkg/plugins/consul.setWeightedClusters(0x43aa060, 0xc000a2cd80, 0xc0046a2000, 0xc003fd22c0, 0xc0063c62c0, 0x0, 0x0)
	/go/pkg/mod/github.com/solo-io/gloo@v1.6.28/projects/gloo/pkg/plugins/consul/subsets.go:81 +0xed
github.com/solo-io/gloo/projects/gloo/pkg/plugins/consul.(*plugin).ProcessRouteAction(0xc006715640, 0x43aa060, 0xc000a2cd80, 0xc0046a2000, 0xc00342a960, 0xc0063be2d0, 0xc003f0cf80, 0xc003f1b180, 0xc003fd2280, 0xc0063c62c0, ...)
	/go/pkg/mod/github.com/solo-io/gloo@v1.6.28/projects/gloo/pkg/plugins/consul/subsets.go:37 +0x124
github.com/solo-io/gloo/projects/gloo/pkg/translator.(*translatorInstance).setAction(0xc006f4ada8, 0x43aa060, 0xc000a2cd80, 0xc0046a2000, 0xc00342a960, 0xc0063be2d0, 0xc003f0cf80, 0xc006ce8120, 0xc003f1b180, 0xc0011fdce0)
	/go/pkg/mod/github.com/solo-io/gloo@v1.6.28/projects/gloo/pkg/translator/route_config.go:286 +0xdc2
github.com/solo-io/gloo/projects/gloo/pkg/translator.(*translatorInstance).envoyRoutes(0xc006f4ada8, 0x43aa060, 0xc000a2cd80, 0xc0046a2000, 0xc00342a960, 0xc0063be2d0, 0xc003f0cf80, 0xc006ce8120, 0xc003f1b180, 0x0, ...)
	/go/pkg/mod/github.com/solo-io/gloo@v1.6.28/projects/gloo/pkg/translator/route_config.go:160 +0x15e
github.com/solo-io/gloo/projects/gloo/pkg/translator.(*translatorInstance).computeVirtualHost(0xc006f4ada8, 0x43aa060, 0xc000a2cd80, 0xc0046a2000, 0xc00342a960, 0xc0063be2d0, 0xc003f0cf80, 0x1, 0xc006ce8180, 0x0)
	/go/pkg/mod/github.com/solo-io/gloo@v1.6.28/projects/gloo/pkg/translator/route_config.go:114 +0x2cb
github.com/solo-io/gloo/projects/gloo/pkg/translator.(*translatorInstance).computeVirtualHosts(0xc006f4ada8, 0x43aa060, 0xc000a2cd80, 0xc0046a2000, 0xc00342a960, 0xc0063be2d0, 0xc0072c4600, 0x0, 0x0, 0x0)
	/go/pkg/mod/github.com/solo-io/gloo@v1.6.28/projects/gloo/pkg/translator/route_config.go:91 +0x2fa
github.com/solo-io/gloo/projects/gloo/pkg/translator.(*translatorInstance).computeRouteConfig(0xc006f4ada8, 0x43aa060, 0xc000a2cd80, 0xc0046a2000, 0xc00342a960, 0xc0063be2d0, 0xc004c1f760, 0x17, 0xc003d21f40, 0x0)
	/go/pkg/mod/github.com/solo-io/gloo@v1.6.28/projects/gloo/pkg/translator/route_config.go:54 +0x18d
github.com/solo-io/gloo/projects/gloo/pkg/translator.(*translatorInstance).computeListenerResources(0xc006f4ada8, 0x43aa060, 0xc000a2cd50, 0xc0046a2000, 0xc00342a960, 0xc0063be2d0, 0xc003d21f40, 0x0)
	/go/pkg/mod/github.com/solo-io/gloo@v1.6.28/projects/gloo/pkg/translator/translator.go:230 +0x1ed
github.com/solo-io/gloo/projects/gloo/pkg/translator.(*translatorInstance).Translate(0xc006f4ada8, 0x43aa060, 0xc006d12060, 0xc0046a2000, 0xc00342a960, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/pkg/mod/github.com/solo-io/gloo@v1.6.28/projects/gloo/pkg/translator/translator.go:169 +0x133e
github.com/solo-io/gloo/projects/gloo/pkg/translator.(*translatorFactory).Translate(0xc00674bb30, 0x43aa060, 0xc008217d10, 0xc0046a2000, 0xc00342a960, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/pkg/mod/github.com/solo-io/gloo@v1.6.28/projects/gloo/pkg/translator/translator.go:77 +0x1f2
github.com/solo-io/gloo/projects/gloo/pkg/syncer.(*translatorSyncer).syncEnvoy(0xc003a88380, 0x43aa060, 0xc005477ad0, 0xc0046a2000, 0xc005477920, 0x0, 0x0)
	/go/pkg/mod/github.com/solo-io/gloo@v1.6.28/projects/gloo/pkg/syncer/envoy_translator_syncer.go:125 +0xd69
github.com/solo-io/gloo/projects/gloo/pkg/syncer.(*translatorSyncer).Sync(0xc003a88380, 0x43a9fa0, 0xc00411b340, 0xc0046a2000, 0x0, 0x0)
	/go/pkg/mod/github.com/solo-io/gloo@v1.6.28/projects/gloo/pkg/syncer/translator_syncer.go:80 +0xb9
github.com/solo-io/gloo/projects/gloo/pkg/api/v1.ApiSyncers.Sync(0xc003a8a140, 0x2, 0x2, 0x43a9fa0, 0xc00411b340, 0xc0046a2000, 0x0, 0x0)
	/go/pkg/mod/github.com/solo-io/gloo@v1.6.28/projects/gloo/pkg/api/v1/api_event_loop.sk.go:28 +0x104
github.com/solo-io/gloo/projects/gloo/pkg/api/v1.(*apiEventLoop).Run.func1(0xc002633bc0, 0x43aa060, 0xc007ed1b90, 0x0, 0xdf8475800, 0x0, 0x0, 0xc007ed1b60, 0xc001c615c0, 0xc00178bd28)
	/go/pkg/mod/github.com/solo-io/gloo@v1.6.28/projects/gloo/pkg/api/v1/api_event_loop.sk.go:84 +0x35f
created by github.com/solo-io/gloo/projects/gloo/pkg/api/v1.(*apiEventLoop).Run
	/go/pkg/mod/github.com/solo-io/gloo@v1.6.28/projects/gloo/pkg/api/v1/api_event_loop.sk.go:66 +0x3ef

To Reproduce
Steps to reproduce the behavior:

1. kube apply '...'
2. curl '....'
3. See error

Expected behavior
A clear and concise description of what you expected to happen.

Additional context
Add any other context about the problem here, e.g.

• Gloo Edge version
• Kubernetes version

[kdorosh]

reopening until backported to all LTS branches

[kdorosh]

The issue occurs because we can return an error here

gloo/projects/gloo/pkg/translator/route_config.go

Line 441 in b9f007e

return err

before we get a chance to update and set the routeAction here

gloo/projects/gloo/pkg/translator/route_config.go

Line 447 in b9f007e

out.ClusterSpecifier = clusterSpecifier

as was intended by the function. We should write unit tests and audit this codepath to ensure there are no other ways this can happen before closing.

The immediate fix has been backported to all LTS branches