Why BaseOperation#createOrReplace's waitUntilCondition use 1s?

[pan3793]

BaseOperation#createOrReplace

  @Override
  public final T createOrReplace() {
    if (item == null) {
      throw new IllegalArgumentException("Nothing to create.");
    }
    R resource = resource(item);

    CreateOrReplaceHelper<T> createOrReplaceHelper = new CreateOrReplaceHelper<>(
        resource::create,
        resource::replace,
        m -> resource.waitUntilCondition(Objects::nonNull, 1, TimeUnit.SECONDS),
        m -> resource.fromServer().get());

    return createOrReplaceHelper.createOrReplace(item);
  }

Why it is hardcoded as 1s? it's likely to timeout when API server has even a little pressure.

[shawkins]

I can't speak fully for the motivation of 1 second, but this is the wait after a create fails with a 500+ code. Those errors are also subject to the retry settings in your config, which makes the 1 second retry redundant. For 6.5+ that defaults to 10 retries and a starting exponential timeout of 100 ms. Those retries will happen before this wait kicks in.

Also createOrReplace functionality has now been deprecated.

[pan3793]

For older versions the default requestRetryBackoffLimit is zero, meaning no retries.

Just notice that, thanks, let me try

[pan3793]

For older versions the default requestRetryBackoffLimit is zero, meaning no retries.

I checked the Spark code, it overwrites the default requestRetryBackoffLimit to 3 instead of 0.

https://github.com/apache/spark/blob/5103e00c4ce5fcc4264ca9c4df12295d42557af6/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/SparkKubernetesClientFactory.scala#L83-L86

Considering the requestTimeout is set to 30s, still not reasonable then ...

[shawkins]

Considering the requestTimeout is set to 30s, still not reasonable then ...

The requestTimeout is separately enforced, in 5.11 that will only be below the backoff values - I believe if it fails there would not be a retry. What you are doing with requestRetryBackoffLimit and requestRetryBackoffInterval is handling the 500 that would otherwise cause createOrReplace to call the wait function which is failing for you. For createOrReplace not to fail, you would need values in there that are sufficient to keep retrying for until the operation succeed - either the item is created, or a conflict code is returned.

[pan3793]

The requestTimeout is separately enforced, in 5.11 that will only be below the backoff values

That surprises me ... , I can't understand why backoff values affect the request timeout. To be clear, I'm using 5.12.2, does it have the same behavior? What about 6.5+ (I'm investigating upgrading to a new version)

[shawkins]

I can't understand why backoff values affect the request timeout.

They don't. They are enforced separately.