fix(gateway): handle implicit kuma.io/service in pod annotation

[jakubdyszkiewicz]

Checklist prior to review

Fix #10072

As @Icarus9913 noticed in his PR #10073 we end up with kuma.io/tags: "null" which we then unmarshal to map and try to set value to nil map.

Why do we end up with kuma.io/tags: "null"?

It's because for gateway instance we now have "implicit" kuma.io/service tag. It should not be set by user. To handle this, we created a method called resolveGatewayInstanceServiceTag in GatewayInstanceReconciler which resolves this implicit tag. This however was only called in selector func of SelectGateway. If there was at least one gateway, we would resolve this, if not then we wouldn't. If gateway was missing, we would then go to createOrUpdateDeployment and write tags to an annotation. This is how ended up with "null".
It also has a bug that if the function was called multiple times (multiple gateways) we would emit a warning.

I'm not a fan of modifying Kube object and not storing the state (because of bugs like this), so I changed the logic a bit to separate validation if a user set a tag and to constructing a map of tags.

Additionally, I added a check to not crash a cp if for some reason we ended up in this situation again.

I tested it manually. It's hard to write E2E for this, because even if we crash a cp it would recover. I think we should consider adding restart policy never to our CP in E2E test, but it's out of scope.

• Link to relevant issue as well as docs and UI issues --
• This will not break child repos: it doesn't hardcode values (.e.g "kumahq" as a image registry) and it will work on Windows, system specific functions like syscall.Mkfifo have equivalent implementation on the other OS --
• Tests (Unit test, E2E tests, manual test on universal and k8s) --
  • Don't forget ci/ labels to run additional/fewer tests
• Do you need to update UPGRADE.md? --
• Does it need to be backported according to the backporting policy? (this GH action will add "backport" label based on these file globs, if you want to prevent it from adding the "backport" label use no-backport-autolabel label) --