Creating and deleting ServiceInstances has a built-in 5m31s timeout that fails on slow services

[mtthwcmpbll]

We're currently seeing service instances provisioned through the AWS service broker can take a long time to create and delete (in the order of 10-20 minutes). The CloudFoundryOperations in both 2.4.0.RELEASE and 2.5.0.BUILD-SNAPSHOT ultimately use code in the form of exponentialBackOff(Duration.ofSeconds(1), Duration.ofSeconds(15), Duration.ofMinutes(5)) (either in LastOperationUtils.waitForCompletion or JobUtils.waitForCompletion) to retry the operation.

Both of the following code snippets can fail with slow services, but these actually success in the sense that the service instance is created or deleted successfully (just long after the client has returned a failure response).

cloudFoundryOperations.services().createInstance(
  CreateServiceInstanceRequest.builder()
    .serviceName("mlab")
    .planName("sandbox")
    .serviceInstanceName("test-service-instance")
      .build())
    .block();

cloudFoundryOperations.services().deleteInstance(
  DeleteServiceInstanceRequest.builder()
    .name("pgsql-test-instance")
    .build())
  .block();

[mtthwcmpbll]

Actually it's a built-in 5m16s timeout, either it changed on me recently or I can't count :)

[nebhale]

@mtthwcmpbll It's not actually quite 5m16, but is based on request delays and exponential back off, but the point that it's non-configurable stands. We picked something arbitrary because the CLI doesn't actually have a time out. This seems to indicate that they just make the create call and depend on the user to poll for completion. We'll probably add a timeout even though it doesn't track with the Cli to make life simpler for our users.

[twoseat]

@mtthwcmpbll Thanks for your patience on this. The fix will be release very shortly, and as you'll see you've inspired a wide-ranging improvement on timeouts! If you see any problems please let us know.

[mtthwcmpbll]

Thanks folks! We'll look at these to fix some of the issues we've seen.