21.11.X: k8s: fix bug reconciling clusters with fixed nodeport

[alenkacz]

Cover letter

The nodeport service when nodePort field provided behaves a bit differently and repeated create on the same resource does not return createifnotexists error but rather an allocation error. so because of that, after the external service gets created after first run of the reconciliation loop, it keeps failing with the nodeport allocated error. The fix is to switch to get-or-create method that creates only if GET call did not succeed.

➜  temp kubectl apply -f nodeport.yaml
service/my-service created
➜  temp kubectl apply -f nodeport.yaml
service/my-service unchanged
➜  temp kubectl create -f nodeport.yaml
The Service "my-service" is invalid: spec.ports[0].nodePort: Invalid value: 30007: provided port is already allocated

Fixes #3263

Release notes

Improvements

k8s: fixed bug with nodeport handling for clusters that have external connectivity with fixed port

[RafalKorepta]

LGTM