Repeated createOrReplace with a service fails

[shawkins]

If you do:

    service = client.services().inNamespace(session.getNamespace()).createOrReplace(service);
    service = client.services().inNamespace(session.getNamespace()).createOrReplace(service);

The second call will fail with:

io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: POST at: https://192.168.39.135:8443/api/v1/namespaces/itest-08651507/services. Message: Service "createorreplaceit-svc" is invalid: spec.clusterIPs: Invalid value: []string{"10.101.116.8"}: failed to allocated ip:10.101.116.8 with error:provided IP is already allocated. Received status: Status(apiVersion=v1, code=422, details=StatusDetails(causes=[StatusCause(field=spec.clusterIPs, message=Invalid value: []string{"10.101.116.8"}: failed to allocated ip:10.101.116.8 with error:provided IP is already allocated, reason=FieldValueInvalid, additionalProperties={})], group=null, kind=Service, name=createorreplaceit-svc, retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status, message=Service "createorreplaceit-svc" is invalid: spec.clusterIPs: Invalid value: []string{"10.101.116.8"}: failed to allocated ip:10.101.116.8 with error:provided IP is already allocated, metadata=ListMeta(_continue=null, remainingItemCount=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=Invalid, status=Failure, additionalProperties={}).

[shawkins]

Understanding a little more about what is happening, the server is trying to allocate the ip rather than first seeing if the there's a create conflict. The only way to smooth this over would be override create(service) and look for a 422 and determine if it should be a conflict instead.

@rohanKanojia is there anything that should be done here?

[rohanKanojia]

bdw, we will see the same behavior with kubectl too if we provide service from server:

kubernetes-resource-yamls : $ kubectl get svc my-service -o yaml | kubectl create -f -
The Service "my-service" is invalid: spec.clusterIPs: Invalid value: []string{"10.96.131.221"}: failed to allocated ip:10.96.131.221 with error:provided IP is already allocated

However, if I only provide original object in create I see that we get 409:

kubernetes-resource-yamls : $ kubectl create -f test-service.yml -v=8
I0519 17:28:14.359670   68609 loader.go:372] Config loaded from file:  /home/rohaan/.kube/config
I0519 17:28:14.360305   68609 cert_rotation.go:137] Starting client certificate rotation controller
I0519 17:28:14.360812   68609 round_trippers.go:432] GET https://192.168.39.108:8443/openapi/v2?timeout=32s
I0519 17:28:14.360820   68609 round_trippers.go:438] Request Headers:
I0519 17:28:14.360827   68609 round_trippers.go:442]     Accept: application/com.github.proto-openapi.spec.v2@v1.0+protobuf
I0519 17:28:14.360833   68609 round_trippers.go:442]     User-Agent: kubectl/v1.21.0 (linux/amd64) kubernetes/cb303e6
I0519 17:28:14.377503   68609 round_trippers.go:457] Response Status: 200 OK in 16 milliseconds
I0519 17:28:14.377520   68609 round_trippers.go:460] Response Headers:
I0519 17:28:14.377527   68609 round_trippers.go:463]     X-From-Cache: 1
I0519 17:28:14.377532   68609 round_trippers.go:463]     X-Kubernetes-Pf-Flowschema-Uid: 62b47cc7-1c57-4bee-870e-68c38b4a691a
I0519 17:28:14.377539   68609 round_trippers.go:463]     X-Kubernetes-Pf-Prioritylevel-Uid: 085ea002-2ac6-4844-97f0-99303ea73fc1
I0519 17:28:14.377544   68609 round_trippers.go:463]     Cache-Control: no-cache, private
I0519 17:28:14.377550   68609 round_trippers.go:463]     Date: Wed, 19 May 2021 11:58:14 GMT
I0519 17:28:14.377555   68609 round_trippers.go:463]     Etag: "06497197421B2A91C8BA55925490C851112A82F37E31759F31681A11FBCBB55F8DE6CA97B3D2486206F9EA939BFBEC3CB00DF9AE8764F5EBBFEAABFA38880E17"
I0519 17:28:14.377561   68609 round_trippers.go:463]     Last-Modified: Wed, 19 May 2021 11:02:11 GMT
I0519 17:28:14.377565   68609 round_trippers.go:463]     Vary: Accept-Encoding
I0519 17:28:14.377569   68609 round_trippers.go:463]     Vary: Accept
I0519 17:28:14.377573   68609 round_trippers.go:463]     X-Varied-Accept: application/com.github.proto-openapi.spec.v2@v1.0+protobuf
I0519 17:28:14.377578   68609 round_trippers.go:463]     Accept-Ranges: bytes
I0519 17:28:14.377583   68609 round_trippers.go:463]     Content-Type: application/octet-stream
I0519 17:28:14.461112   68609 request.go:1121] Response Body:
00000000  0a 03 32 2e 30 12 15 0a  0a 4b 75 62 65 72 6e 65  |..2.0....Kuberne|
00000010  74 65 73 12 07 76 31 2e  32 30 2e 32 42 d8 a3 9a  |tes..v1.20.2B...|
00000020  01 12 e8 27 0a 20 2f 61  70 69 73 2f 61 70 70 73  |...'. /apis/apps|
00000030  2f 76 31 2f 77 61 74 63  68 2f 73 74 61 74 65 66  |/v1/watch/statef|
00000040  75 6c 73 65 74 73 12 c3  27 12 b7 04 0a 07 61 70  |ulsets..'.....ap|
00000050  70 73 5f 76 31 1a 77 77  61 74 63 68 20 69 6e 64  |ps_v1.wwatch ind|
00000060  69 76 69 64 75 61 6c 20  63 68 61 6e 67 65 73 20  |ividual changes |
00000070  74 6f 20 61 20 6c 69 73  74 20 6f 66 20 53 74 61  |to a list of Sta|
00000080  74 65 66 75 6c 53 65 74  2e 20 64 65 70 72 65 63  |tefulSet. deprec|
00000090  61 74 65 64 3a 20 75 73  65 20 74 68 65 20 27 77  |ated: use the 'w|
000000a0  61 74 63 68 27 20 70 61  72 61 6d 65 74 65 72 20  |atch' parameter |
000000b0  77 69 74 68 20 61 20 6c  69 73 74 20 6f 70 65 72  |with a list oper|
000000c0  61 74 69 6f 6e 20 69 6e  73 74 65 61 64 2e 2a 2a  |ation instead.* [truncated 16252269 chars]
I0519 17:28:14.513435   68609 request.go:1123] Request Body: {"apiVersion":"v1","kind":"Service","metadata":{"name":"my-service","namespace":"default"},"spec":{"ports":[{"port":80,"protocol":"TCP","targetPort":9390}],"selector":{"app":"MyApp"}}}
I0519 17:28:14.513477   68609 round_trippers.go:432] POST https://192.168.39.108:8443/api/v1/namespaces/default/services?fieldManager=kubectl-create
I0519 17:28:14.513485   68609 round_trippers.go:438] Request Headers:
I0519 17:28:14.513491   68609 round_trippers.go:442]     Accept: application/json
I0519 17:28:14.513496   68609 round_trippers.go:442]     Content-Type: application/json
I0519 17:28:14.513502   68609 round_trippers.go:442]     User-Agent: kubectl/v1.21.0 (linux/amd64) kubernetes/cb303e6
I0519 17:28:14.545803   68609 round_trippers.go:457] Response Status: 409 Conflict in 32 milliseconds
I0519 17:28:14.545838   68609 round_trippers.go:460] Response Headers:
I0519 17:28:14.545849   68609 round_trippers.go:463]     Content-Length: 206
I0519 17:28:14.545860   68609 round_trippers.go:463]     Date: Wed, 19 May 2021 11:58:14 GMT
I0519 17:28:14.545880   68609 round_trippers.go:463]     Cache-Control: no-cache, private
I0519 17:28:14.545890   68609 round_trippers.go:463]     Content-Type: application/json
I0519 17:28:14.545899   68609 round_trippers.go:463]     X-Kubernetes-Pf-Flowschema-Uid: 62b47cc7-1c57-4bee-870e-68c38b4a691a
I0519 17:28:14.545909   68609 round_trippers.go:463]     X-Kubernetes-Pf-Prioritylevel-Uid: 085ea002-2ac6-4844-97f0-99303ea73fc1
I0519 17:28:14.545947   68609 request.go:1123] Response Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"services \"my-service\" already exists","reason":"AlreadyExists","details":{"name":"my-service","kind":"services"},"code":409}
I0519 17:28:14.546177   68609 helpers.go:216] server response object: [{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {},
  "status": "Failure",
  "message": "error when creating \"test-service.yml\": services \"my-service\" already exists",
  "reason": "AlreadyExists",
  "details": {
    "name": "my-service",
    "kind": "services"
  },
  "code": 409
}]

Note that this is working when using kubectl apply.

[manusa]

Yes, IIRC the issue had to do with the nodePort or IP already allocated. Since this value is provided, it can't be reused (what Steven says here)

[shawkins]

I went ahead and closed the pr - while this does seem to be odd behavior by the api server, it's probably not worth trying to workaround in the way that I was thinking. Our operator is always using the existing object as the base for calls to createOrReplace, so I'll just update the calling logic to use a null clusterIp.