SharedIndexInformer for Custom resource stuck in "Start listing and watching" stage in reflector runnable

[sharath-sri-chellappa]

Client Version
14.0.0

Kubernetes Version
1.22.4

Java Version
Java 11

I am trying to create a SharedIndexInformer for a Kubevirt Virtual Machine Instance (custom resource - http://kubevirt.io/api-reference/main/index.html) and carry out a set of actions on add for the Custom Resource. I have used the Official Kubernetes Java client for the creation of the SharedIndexInformer (not fabric8)

I have gone ahead and created the libraries/models for the custom resource java client from the instructions given here - https://github.com/kubernetes-client/java/blob/master/docs/generate-model-from-third-party-resources.md

What I seem to be stuck at is when I am creating the sharedIndexInformer for the VMI Resource, the call to io.kubernetes.client.informer.cache.ReflectorRunnable.run() function seems to be stuck at Start listing and watching (code line - https://github.com/kubernetes-client/java/blob/master/util/src/main/java/io/kubernetes/client/informer/cache/ReflectorRunnable.java#L87)

Output Snippet

Kubeconfig Path - C:\GitFolder/config
16:31:55.867 [informer-controller-V1Pod] INFO io.kubernetes.client.informer.cache.Controller - informer#Controller: ready to run resync & reflector runnable
16:31:55.867 [informer-controller-V1alpha3VirtualMachineInstance] INFO io.kubernetes.client.informer.cache.Controller - informer#Controller: ready to run resync & reflector runnable
16:31:55.870 [informer-controller-V1Pod] INFO io.kubernetes.client.informer.cache.Controller - informer#Controller: resync skipped due to 0 full resync period
16:31:55.873 [controller-reflector-io.kubernetes.client.openapi.models.V1Pod-1] INFO io.kubernetes.client.informer.cache.ReflectorRunnable - class io.kubernetes.client.openapi.models.V1Pod#Start listing and watching...
16:31:55.873 [controller-reflector-io.kubernetes.client.kubevirt.models.V1alpha3VirtualMachineInstance-1] INFO io.kubernetes.client.informer.cache.ReflectorRunnable - class io.kubernetes.client.kubevirt.models.V1alpha3VirtualMachineInstance#Start listing and watching...
16:31:58.524 [controller-reflector-io.kubernetes.client.openapi.models.V1Pod-1] DEBUG io.kubernetes.client.informer.cache.ReflectorRunnable - class io.kubernetes.client.openapi.models.V1Pod#Extract resourceVersion 110011207 list meta
16:31:58.524 [controller-reflector-io.kubernetes.client.openapi.models.V1Pod-1] DEBUG io.kubernetes.client.informer.cache.ReflectorRunnable - class io.kubernetes.client.openapi.models.V1Pod#Start watching with 110011207...
16:31:58.524 [controller-reflector-io.kubernetes.client.openapi.models.V1Pod-1] DEBUG io.kubernetes.client.informer.cache.ReflectorRunnable - class io.kubernetes.client.openapi.models.V1Pod#Start watch with resource version 110011207
16:31:59.461 [controller-reflector-io.kubernetes.client.kubevirt.models.V1alpha3VirtualMachineInstance-1] INFO io.kubernetes.client.informer.cache.ReflectorRunnable - class io.kubernetes.client.kubevirt.models.V1alpha3VirtualMachineInstance#Start listing and watching...
16:32:00.726 [controller-reflector-io.kubernetes.client.kubevirt.models.V1alpha3VirtualMachineInstance-1] INFO io.kubernetes.client.informer.cache.ReflectorRunnable - class io.kubernetes.client.kubevirt.models.V1alpha3VirtualMachineInstance#Start listing and watching...
16:32:01.994 [controller-reflector-io.kubernetes.client.kubevirt.models.V1alpha3VirtualMachineInstance-1] INFO io.kubernetes.client.informer.cache.ReflectorRunnable - class io.kubernetes.client.kubevirt.models.V1alpha3VirtualMachineInstance#Start listing and watching...
16:32:03.267 [controller-reflector-io.kubernetes.client.kubevirt.models.V1alpha3VirtualMachineInstance-1] INFO io.kubernetes.client.informer.cache.ReflectorRunnable - class io.kubernetes.client.kubevirt.models.V1alpha3VirtualMachineInstance#Start listing and watching...
16:32:04.533 [controller-reflector-io.kubernetes.client.kubevirt.models.V1alpha3VirtualMachineInstance-1] INFO io.kubernetes.client.informer.cache.ReflectorRunnable - class io.kubernetes.client.kubevirt.models.V1alpha3VirtualMachineInstance#Start listing and watching...
16:32:05.810 [controller-reflector-io.kubernetes.client.kubevirt.models.V1alpha3VirtualMachineInstance-1] INFO io.kubernetes.client.informer.cache.ReflectorRunnable - class io.kubernetes.client.kubevirt.models.V1alpha3VirtualMachineInstance#Start listing and watching...

If you look at this output, an informer which I created from V1Pod resource seems to work correctly however that is not the case with V1alpha3VirtualMachineInstances.

Code Snippet for my Informer -

private static Call listNamespacedVmiCall(ApiClient client, String namespace, String pretty, Boolean allowWatchBookmarks, String _continue, String fieldSelector, String labelSelector, Integer limit, String resourceVersion, String resourceVersionMatch, Integer timeoutSeconds, Boolean watch, ApiCallback _callback) throws ApiException {
	Object localVarPostBody = null;
	String localVarPath = "/api/v1/namespaces/{namespace}/vmi".replaceAll("\\{namespace\\}", client.escapeString(namespace.toString()));
	List<Pair> localVarQueryParams = new ArrayList();
	List<Pair> localVarCollectionQueryParams = new ArrayList();
	if (pretty != null) {
		localVarQueryParams.addAll(client.parameterToPair("pretty", pretty));
	}

	if (allowWatchBookmarks != null) {
		localVarQueryParams.addAll(client.parameterToPair("allowWatchBookmarks", allowWatchBookmarks));
	}

	if (_continue != null) {
		localVarQueryParams.addAll(client.parameterToPair("continue", _continue));
	}

	if (fieldSelector != null) {
		localVarQueryParams.addAll(client.parameterToPair("fieldSelector", fieldSelector));
	}

	if (labelSelector != null) {
		localVarQueryParams.addAll(client.parameterToPair("labelSelector", labelSelector));
	}

	if (limit != null) {
		localVarQueryParams.addAll(client.parameterToPair("limit", limit));
	}

	if (resourceVersion != null) {
		localVarQueryParams.addAll(client.parameterToPair("resourceVersion", resourceVersion));
	}

	if (resourceVersionMatch != null) {
		localVarQueryParams.addAll(client.parameterToPair("resourceVersionMatch", resourceVersionMatch));
	}

	if (timeoutSeconds != null) {
		localVarQueryParams.addAll(client.parameterToPair("timeoutSeconds", timeoutSeconds));
	}

	if (watch != null) {
		localVarQueryParams.addAll(client.parameterToPair("watch", watch));
	}

	Map<String, String> localVarHeaderParams = new HashMap<String, String>();
	Map<String, String> localVarCookieParams = new HashMap<String, String>();
	Map<String, Object> localVarFormParams = new HashMap<String, Object>();
	String[] localVarAccepts = new String[]{"application/json", "application/yaml", "application/vnd.kubernetes.protobuf", "application/json;stream=watch", "application/vnd.kubernetes.protobuf;stream=watch"};
	String localVarAccept = client.selectHeaderAccept(localVarAccepts);
	if (localVarAccept != null) {
		localVarHeaderParams.put("Accept", localVarAccept);
	}

	String[] localVarContentTypes = new String[0];
	String localVarContentType = client.selectHeaderContentType(localVarContentTypes);
	localVarHeaderParams.put("Content-Type", localVarContentType);
	String[] localVarAuthNames = new String[]{"BearerToken"};

	return client.buildCall(
			localVarPath,
			"GET",
			localVarQueryParams,
			localVarCollectionQueryParams,
			localVarPostBody,
			localVarHeaderParams,
			localVarCookieParams,
			localVarFormParams,
			localVarAuthNames,
			_callback);
}
public static SharedIndexInformer<V1alpha3VirtualMachineInstance> createVmiInformer(String namespace, SharedInformerFactory factory, CoreV1Api coreV1Api) {
	ApiClient apiClient = coreV1Api.getApiClient();
        SharedIndexInformer<V1alpha3VirtualMachineInstance> vmiInformer =
                factory.sharedIndexInformerFor(
                (CallGeneratorParams params) -> {
                    return listNamespacedVmiCall(
                            apiClient,
                            namespace,
                            null,
                            null,
                            null,
                            null,
                            null,
                            null,
                            params.resourceVersion,
                            null,
                            params.timeoutSeconds,
                            params.watch,
                            null);
                },
                V1alpha3VirtualMachineInstance.class,
                V1alpha3VirtualMachineInstanceList.class);

	vmiInformer.addEventHandler(
			new ResourceEventHandler<V1alpha3VirtualMachineInstance>() {
				@Override
				public void onAdd(V1alpha3VirtualMachineInstance vmi) {
					System.out.printf("%s VMI added! Timestamp %s\n", vmi.getMetadata().getName(),
							vmi.getMetadata().getCreationTimestamp());
				}

				@Override
				public void onUpdate(V1alpha3VirtualMachineInstance oldVmi, V1alpha3VirtualMachineInstance newVmi) {
					System.out.printf(
							"%s => %s VMI updated!\n",
							oldVmi.getMetadata().getName(), newVmi.getMetadata().getName());
					System.out.printf("Data Updated - \nDifferences between the 2 strings - %s\n",
							StringUtils.difference(oldVmi.getMetadata().toString(), newVmi.getMetadata().toString()));
				}

				@Override
				public void onDelete(V1alpha3VirtualMachineInstance vmi, boolean deletedFinalStateUnknown) {
					System.out.printf("%s VMI deleted!\n", vmi.getMetadata().getName());
				}
			});

	return vmiInformer;
}

Here listNamespacedVmiCall is the function that lists and watches for the virtual machine instance on a particular namespace. This is a static function which has been written by me.

The Virtual Machine Class has been autogenerated using the instructions from the link given above (https://github.com/kubernetes-client/java/blob/master/docs/generate-model-from-third-party-resources.md) and looks extremely similar to the V1Pod Class -

public class V1alpha3VirtualMachineInstance implements io.kubernetes.client.common.KubernetesObject {
  public static final String SERIALIZED_NAME_API_VERSION = "apiVersion";
  @SerializedName(SERIALIZED_NAME_API_VERSION)
  private String apiVersion;

  public static final String SERIALIZED_NAME_KIND = "kind";
  @SerializedName(SERIALIZED_NAME_KIND)
  private String kind;

  public static final String SERIALIZED_NAME_METADATA = "metadata";
  @SerializedName(SERIALIZED_NAME_METADATA)
  private V1ObjectMeta metadata = null;

  public static final String SERIALIZED_NAME_SPEC = "spec";
  @SerializedName(SERIALIZED_NAME_SPEC)
  private V1alpha3VirtualMachineInstanceSpec spec;

  public static final String SERIALIZED_NAME_STATUS = "status";
  @SerializedName(SERIALIZED_NAME_STATUS)
  private V1alpha3VirtualMachineInstanceStatus status;


  public V1alpha3VirtualMachineInstance apiVersion(String apiVersion) {
    
    this.apiVersion = apiVersion;
    return this;
  }

   /**
   * APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources
   * @return apiVersion
  **/
  @javax.annotation.Nullable
  @ApiModelProperty(value = "APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources")

  public String getApiVersion() {
    return apiVersion;
  }


  public void setApiVersion(String apiVersion) {
    this.apiVersion = apiVersion;
  }


  public V1alpha3VirtualMachineInstance kind(String kind) {
    
    this.kind = kind;
    return this;
  }

   /**
   * Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds
   * @return kind
  **/
  @javax.annotation.Nullable
  @ApiModelProperty(value = "Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds")

  public String getKind() {
    return kind;
  }


  public void setKind(String kind) {
    this.kind = kind;
  }


  public V1alpha3VirtualMachineInstance metadata(V1ObjectMeta metadata) {
    
    this.metadata = metadata;
    return this;
  }

   /**
   * Get metadata
   * @return metadata
  **/
  @javax.annotation.Nullable
  @ApiModelProperty(value = "")

  public V1ObjectMeta getMetadata() {
    return metadata;
  }


  public void setMetadata(V1ObjectMeta metadata) {
    this.metadata = metadata;
  }


  public V1alpha3VirtualMachineInstance spec(V1alpha3VirtualMachineInstanceSpec spec) {
    
    this.spec = spec;
    return this;
  }

   /**
   * Get spec
   * @return spec
  **/
  @ApiModelProperty(required = true, value = "")

  public V1alpha3VirtualMachineInstanceSpec getSpec() {
    return spec;
  }


  public void setSpec(V1alpha3VirtualMachineInstanceSpec spec) {
    this.spec = spec;
  }


  public V1alpha3VirtualMachineInstance status(V1alpha3VirtualMachineInstanceStatus status) {
    
    this.status = status;
    return this;
  }

   /**
   * Get status
   * @return status
  **/
  @javax.annotation.Nullable
  @ApiModelProperty(value = "")

  public V1alpha3VirtualMachineInstanceStatus getStatus() {
    return status;
  }


  public void setStatus(V1alpha3VirtualMachineInstanceStatus status) {
    this.status = status;
  }


  @Override
  public boolean equals(Object o) {
    if (this == o) {
      return true;
    }
    if (o == null || getClass() != o.getClass()) {
      return false;
    }
    V1alpha3VirtualMachineInstance v1alpha3VirtualMachineInstance = (V1alpha3VirtualMachineInstance) o;
    return Objects.equals(this.apiVersion, v1alpha3VirtualMachineInstance.apiVersion) &&
        Objects.equals(this.kind, v1alpha3VirtualMachineInstance.kind) &&
        Objects.equals(this.metadata, v1alpha3VirtualMachineInstance.metadata) &&
        Objects.equals(this.spec, v1alpha3VirtualMachineInstance.spec) &&
        Objects.equals(this.status, v1alpha3VirtualMachineInstance.status);
  }

  @Override
  public int hashCode() {
    return Objects.hash(apiVersion, kind, metadata, spec, status);
  }


  @Override
  public String toString() {
    StringBuilder sb = new StringBuilder();
    sb.append("class V1alpha3VirtualMachineInstance {\n");
    sb.append("    apiVersion: ").append(toIndentedString(apiVersion)).append("\n");
    sb.append("    kind: ").append(toIndentedString(kind)).append("\n");
    sb.append("    metadata: ").append(toIndentedString(metadata)).append("\n");
    sb.append("    spec: ").append(toIndentedString(spec)).append("\n");
    sb.append("    status: ").append(toIndentedString(status)).append("\n");
    sb.append("}");
    return sb.toString();
  }

  /**
   * Convert the given object to string with each line indented by 4 spaces
   * (except the first line).
   */
  private String toIndentedString(Object o) {
    if (o == null) {
      return "null";
    }
    return o.toString().replace("\n", "\n    ");
  }

}

The same can be said for V1alpha3VirtualMachineInstanceList.java (which is on the lines of V1PodList.java)

public class V1alpha3VirtualMachineInstanceList implements io.kubernetes.client.common.KubernetesListObject {
  public static final String SERIALIZED_NAME_API_VERSION = "apiVersion";
  @SerializedName(SERIALIZED_NAME_API_VERSION)
  private String apiVersion;

  public static final String SERIALIZED_NAME_ITEMS = "items";
  @SerializedName(SERIALIZED_NAME_ITEMS)
  private List<V1alpha3VirtualMachineInstance> items = new ArrayList<>();

  public static final String SERIALIZED_NAME_KIND = "kind";
  @SerializedName(SERIALIZED_NAME_KIND)
  private String kind;

  public static final String SERIALIZED_NAME_METADATA = "metadata";
  @SerializedName(SERIALIZED_NAME_METADATA)
  private V1ListMeta metadata = null;


  public V1alpha3VirtualMachineInstanceList apiVersion(String apiVersion) {
    
    this.apiVersion = apiVersion;
    return this;
  }

   /**
   * APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources
   * @return apiVersion
  **/
  @javax.annotation.Nullable
  @ApiModelProperty(value = "APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources")

  public String getApiVersion() {
    return apiVersion;
  }


  public void setApiVersion(String apiVersion) {
    this.apiVersion = apiVersion;
  }


  public V1alpha3VirtualMachineInstanceList items(List<V1alpha3VirtualMachineInstance> items) {
    
    this.items = items;
    return this;
  }

  public V1alpha3VirtualMachineInstanceList addItemsItem(V1alpha3VirtualMachineInstance itemsItem) {
    this.items.add(itemsItem);
    return this;
  }

   /**
   * List of virtualmachineinstances. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md
   * @return items
  **/
  @ApiModelProperty(required = true, value = "List of virtualmachineinstances. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md")

  public List<V1alpha3VirtualMachineInstance> getItems() {
    return items;
  }


  public void setItems(List<V1alpha3VirtualMachineInstance> items) {
    this.items = items;
  }


  public V1alpha3VirtualMachineInstanceList kind(String kind) {
    
    this.kind = kind;
    return this;
  }

   /**
   * Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds
   * @return kind
  **/
  @javax.annotation.Nullable
  @ApiModelProperty(value = "Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds")

  public String getKind() {
    return kind;
  }


  public void setKind(String kind) {
    this.kind = kind;
  }


  public V1alpha3VirtualMachineInstanceList metadata(V1ListMeta metadata) {
    
    this.metadata = metadata;
    return this;
  }

   /**
   * Get metadata
   * @return metadata
  **/
  @javax.annotation.Nullable
  @ApiModelProperty(value = "")

  public V1ListMeta getMetadata() {
    return metadata;
  }


  public void setMetadata(V1ListMeta metadata) {
    this.metadata = metadata;
  }


  @Override
  public boolean equals(Object o) {
    if (this == o) {
      return true;
    }
    if (o == null || getClass() != o.getClass()) {
      return false;
    }
    V1alpha3VirtualMachineInstanceList v1alpha3VirtualMachineInstanceList = (V1alpha3VirtualMachineInstanceList) o;
    return Objects.equals(this.apiVersion, v1alpha3VirtualMachineInstanceList.apiVersion) &&
        Objects.equals(this.items, v1alpha3VirtualMachineInstanceList.items) &&
        Objects.equals(this.kind, v1alpha3VirtualMachineInstanceList.kind) &&
        Objects.equals(this.metadata, v1alpha3VirtualMachineInstanceList.metadata);
  }

  @Override
  public int hashCode() {
    return Objects.hash(apiVersion, items, kind, metadata);
  }


  @Override
  public String toString() {
    StringBuilder sb = new StringBuilder();
    sb.append("class V1alpha3VirtualMachineInstanceList {\n");
    sb.append("    apiVersion: ").append(toIndentedString(apiVersion)).append("\n");
    sb.append("    items: ").append(toIndentedString(items)).append("\n");
    sb.append("    kind: ").append(toIndentedString(kind)).append("\n");
    sb.append("    metadata: ").append(toIndentedString(metadata)).append("\n");
    sb.append("}");
    return sb.toString();
  }

  /**
   * Convert the given object to string with each line indented by 4 spaces
   * (except the first line).
   */
  private String toIndentedString(Object o) {
    if (o == null) {
      return "null";
    }
    return o.toString().replace("\n", "\n    ");
  }

}

I am really not sure where I am going wrong here since both classes implement io.kubernetes.client.common.KubernetesListObject and can work exactly like the V1Pod and V1PodList classes. Can someone tell me what I am doing wrong here ?

The way I am creating my informer is as follows -

        String kubeConfigPath = System.getenv("HOME") + "/config";
        System.out.println("Kubeconfig Path - " + kubeConfigPath);
        // loading the out-of-cluster config, a kubeconfig from file-system
        ApiClient client =
                ClientBuilder.kubeconfig(KubeConfig.loadKubeConfig(new FileReader(kubeConfigPath))).build();

        // set the global default api-client to the in-cluster one from above
        Configuration.setDefaultApiClient(client);
        CoreV1Api coreV1Api = new CoreV1Api();
        ApiClient apiClient = coreV1Api.getApiClient();
        OkHttpClient httpClient =
                apiClient.getHttpClient().newBuilder().readTimeout(0, TimeUnit.SECONDS).build();
        apiClient.setHttpClient(httpClient);

        SharedInformerFactory factory = new SharedInformerFactory();
        SharedIndexInformer<V1alpha3VirtualMachineInstance> vmiInformer = createVmiInformer(<namespace>, factory, coreV1Api);
        factory.startAllRegisteredInformers();

Any ideas would help me proceed here would be welcome.

[yue9944882]

@sharath-sri-chellappa thanks for the detailed report, overall i think both your model generation and the usage of informer is correct. but through your logs i find that the informer is trying re-listwatching every 1s so it's possibily because ReflectorRunnable::run keeps aborting which triggered the background controller to revive the reflector. but i dont see any error logs in your description, every exception that causes the abortion of the reflector should be logged unless you overrided the exception handler. are you sure if there's no error logs at all?

[sharath-sri-chellappa]

@yue9944882 Yes. I am sure that there are no error logs. Thats what seems to be confusing me. In fact, I tried changing my run() code in reflector runnable to try and see which portion does it actually error out in. But this isnt even triggered looks like from the output. Can you please help me understand where I am going wrong here ? Like if you notice, I have tried adding more logging statements but none are actually being shown on output

  public void run() {
    log.info("{}#Start listing and watching - 0...", apiTypeClass);
    log.info("{}#Coming here - 1...", apiTypeClass);
    try {
      ApiListType list =
          listerWatcher.list(
              new CallGeneratorParams(Boolean.FALSE, getRelistResourceVersion(), null));

      V1ListMeta listMeta = list.getMetadata();
      String resourceVersion = listMeta.getResourceVersion();
      List<? extends KubernetesObject> items = list.getItems();

      if (log.isDebugEnabled()) {
        log.info("{}#Extract resourceVersion {} list meta", apiTypeClass, resourceVersion);
      }
      this.syncWith(items, resourceVersion);
      this.lastSyncResourceVersion = resourceVersion;
      this.isLastSyncResourceVersionUnavailable = false;

      if (log.isDebugEnabled()) {
        log.info("{}#Start watching with {}...", apiTypeClass, lastSyncResourceVersion);
      }
      while (true) {
        if (!isActive.get()) {
          closeWatch();
          return;
        }

        try {
          if (log.isDebugEnabled()) {
            log.info(
                "{}#Start watch with resource version {}", apiTypeClass, lastSyncResourceVersion);
          }

          long jitteredWatchTimeoutSeconds =
              Double.valueOf(REFLECTOR_WATCH_CLIENTSIDE_TIMEOUT.getSeconds() * (1 + Math.random()))
                  .longValue();
          Watchable<ApiType> newWatch =
              listerWatcher.watch(
                  new CallGeneratorParams(
                      Boolean.TRUE,
                      lastSyncResourceVersion,
                      Long.valueOf(jitteredWatchTimeoutSeconds).intValue()));

          synchronized (this) {
            if (!isActive.get()) {
              newWatch.close();
              continue;
            }
            watch = newWatch;
          }
          watchHandler(newWatch);
        } catch (WatchExpiredException e) {
          // Watch calls were failed due to expired resource-version. Returning
          // to unwind the list-watch loops so that we can respawn a new round
          // of list-watching.
          log.info("{}#Watch expired, will re-list-watch soon", this.apiTypeClass);
          return;
        } catch (Throwable t) {
          if (isConnectException(t)) {
            // If this is "connection refused" error, it means that most likely
            // apiserver is not responsive. It doesn't make sense to re-list all
            // objects because most likely we will be able to restart watch where
            // we ended. If that's the case wait and resend watch request.
            log.info("{}#Watch get connect exception, retry watch", this.apiTypeClass);
            try {
              Thread.sleep(1000L);
            } catch (InterruptedException e) {
              // no-op
            }
            continue;
          }
          if ((t instanceof RuntimeException)
              && t.getMessage() != null
              && t.getMessage().contains("IO Exception during hasNext")) {
            log.info("{}#Read timeout retry list and watch", this.apiTypeClass);
            // IO timeout should be taken as a normal case
            return;
          }
          this.exceptionHandler.accept(apiTypeClass, t);
          return;
        } finally {
          closeWatch();
        }
      }
    } catch (ApiException e) {
      if (e.getCode() == HttpURLConnection.HTTP_GONE) {
        log.info(
            "ResourceVersion {} expired, will retry w/o resourceVersion at the next time",
            getRelistResourceVersion());
        isLastSyncResourceVersionUnavailable = true;
      } else {
        log.info("{}#Coming here - 2...", apiTypeClass);
        this.exceptionHandler.accept(apiTypeClass, e);
      }
    } catch (Throwable t) {
      log.info("{}#Coming here - 3...", apiTypeClass);
      this.exceptionHandler.accept(apiTypeClass, t);
    }
  }

Overriding the ExceptionHandler is not really happening because the default exception handler is being used from here. I am not sure how to check if I have not passed a exception handler as part of my parameters for creating Reflector Runnable from Controller.java. Can you please tell me which line would be responsible for doing that from the snippet of code in my listNamespacedVmiCall ? Sorry I am not exactly perfect with respect to my understanding of the code flow. But I am confident I have not overriden the Exception Handler.

[sharath-sri-chellappa]

@yue9944882 Sorry for repinging. Any idea what I am doing wrong here ?

[yue9944882]

log.info("{}#Coming here - 1...", apiTypeClass);

@sharath-sri-chellappa so the new log you added is also not shown right? which probably means that the thread is aborted right after log.info("{}#Start listing and watching - 0...", apiTypeClass);, do you get a chance to step-by-step debug into the program to find out what's thrown out there?

[yue9944882]

@sharath-sri-chellappa when i was running informer w/ java 11 in my local environment i find the following errors which is related w/ the incompatibility between gson library and java 11, see similar thread #1638:

17:10:31.673 [controller-reflector-io.kubernetes.client.openapi.models.V1Pod-1] ERROR i.k.c.i.cache.ReflectorRunnable - class io.kubernetes.client.openapi.models.V1Pod#Reflector loop failed unexpectedly
com.google.gson.JsonIOException: Unable to create instance of class io.kubernetes.client.util.Watch$Response; usage of JDK Unsafe is disabled. Registering an InstanceCreator or a TypeAdapter for this type, adding a no-args constructor, or enabling usage of JDK Unsafe may fix this problem.
	at com.google.gson.internal.ConstructorConstructor$16.construct(ConstructorConstructor.java:261)
	at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$Adapter.read(ReflectiveTypeAdapterFactory.java:211)
	at com.google.gson.Gson.fromJson(Gson.java:984)
	at com.google.gson.Gson.fromJson(Gson.java:949)
	at com.google.gson.Gson.fromJson(Gson.java:898)
	at io.kubernetes.client.openapi.JSON.deserialize(JSON.java:173)
	at io.kubernetes.client.util.Watch.parseLine(Watch.java:170)
	at io.kubernetes.client.util.Watch.next(Watch.java:126)
	at io.kubernetes.client.util.Watch.next(Watch.java:38)
	at io.kubernetes.client.informer.cache.ReflectorRunnable.watchHandler(ReflectorRunnable.java:234)
	at io.kubernetes.client.informer.cache.ReflectorRunnable.run(ReflectorRunnable.java:137)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
	at java.base/java.lang.Thread.run(Thread.java:831)
17:10:32.524 [controller-reflector-io.kubernetes.client.openapi.models.V1Pod-1] INFO  i.k.c.i.cache.ReflectorRunnable - class io.kubernetes.client.openapi.models.V1Pod#Start listing and watching...

i wonder u r hitting the same issue, currently we're work-arounding by adding a --illegal-access=warn --add-opens java.base/java.util=ALL-UNNAMED --add-opens java.base/java.time=ALL-UNNAMED jvm option to suppress the error

java/util/pom.xml

Line 175 in e2f2ef2

@{argLine} -Xms512m -Xmx1500m --illegal-access=warn --add-opens java.base/java.util=ALL-UNNAMED --add-opens java.base/java.time=ALL-UNNAMED

also we're working on a long-term fix at #2078

[sharath-sri-chellappa]

Hi @yue9944882 Making changes to pom.xml doesnt seem to solve the problem.

And about the log line, if you notice, the logs are not being displayed correctly in the first place. Meaning in the place where it displays

16:31:59.461 [controller-reflector-io.kubernetes.client.kubevirt.models.V1alpha3VirtualMachineInstance-1] INFO io.kubernetes.client.informer.cache.ReflectorRunnable - class io.kubernetes.client.kubevirt.models.V1alpha3VirtualMachineInstance#Start listing and watching...

it should have displayed something along the lines

log.info("{}#Start listing and watching - 0...", apiTypeClass);

This is what perplexes me. Which is why I raised the Issue in the first place since the code that I am adding to ReflectorRunnable is not being taken by the SharedIndexInformer running the code. Which is why I wanted to check if the code flow that I have thought to be correct might be mistaken (is ReflectorRunnable even called from that file ?)

[sharath-sri-chellappa]

@yue9944882 Hi. I am able to reproduce the error even on the following configurations -

Client Version
12.0.0

Kubernetes Version

Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.0", GitCommit:"e19964183377d0ec2052d1f1fa930c4d7575bd50", GitTreeState:"clean", BuildDate:"2020-08-26T14:30:33Z", GoVersion:"go1.15", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.1", GitCommit:"5e58841cce77d4bc13713ad2b91fa0d961e69192", GitTreeState:"clean", BuildDate:"2021-05-12T14:12:29Z", GoVersion:"go1.16.4", Compiler:"gc", Platform:"linux/amd64"}

Java Version

openjdk 11.0.11 2021-04-20
OpenJDK Runtime Environment (build 11.0.11+9-Ubuntu-0ubuntu2.20.04)
OpenJDK 64-Bit Server VM (build 11.0.11+9-Ubuntu-0ubuntu2.20.04, mixed mode, sharing)

Can you please help out here again wrt the issue being faced.

[yue9944882]

@sharath-sri-chellappa i wonder if you are hitting this problem, can you try the work-around here?

#2102 (comment)

[sharath-sri-chellappa]

@yue9944882 Sorry for the delay. According to the workaround, I changed the code to the following -

        Configuration.setDefaultApiClient(client);
        CoreV1Api coreV1Api = new CoreV1Api();
        ApiClient apiClient = coreV1Api.getApiClient();
        OkHttpClient httpClient =
                apiClient.getHttpClient().newBuilder().readTimeout(0, TimeUnit.SECONDS).build();
        apiClient.setHttpClient(httpClient);

        SharedInformerFactory factory = new SharedInformerFactory(apiClient);

The output still remains to be the same -

15:52:03.083 [informer-controller-V1alpha3VirtualMachineInstance] INFO io.kubernetes.client.informer.cache.Controller - informer#Controller: ready to run resync & reflector runnable
15:52:03.090 [informer-controller-V1alpha3VirtualMachineInstance] INFO io.kubernetes.client.informer.cache.Controller - informer#Controller: resync skipped due to 0 full resync period
15:52:03.104 [controller-reflector-io.kubernetes.client.kubevirt.models.V1alpha3VirtualMachineInstance-1] INFO io.kubernetes.client.informer.cache.ReflectorRunnable - class io.kubernetes.client.kubevirt.models.V1alpha3VirtualMachineInstance#Start listing and watching...
15:52:04.979 [controller-reflector-io.kubernetes.client.kubevirt.models.V1alpha3VirtualMachineInstance-1] INFO io.kubernetes.client.informer.cache.ReflectorRunnable - class io.kubernetes.client.kubevirt.models.V1alpha3VirtualMachineInstance#Start listing and watching...
15:52:06.109 [controller-reflector-io.kubernetes.client.kubevirt.models.V1alpha3VirtualMachineInstance-1] INFO io.kubernetes.client.informer.cache.ReflectorRunnable - class io.kubernetes.client.kubevirt.models.V1alpha3VirtualMachineInstance#Start listing and watching...
15:52:07.219 [controller-reflector-io.kubernetes.client.kubevirt.models.V1alpha3VirtualMachineInstance-1] INFO io.kubernetes.client.informer.cache.ReflectorRunnable - class io.kubernetes.client.kubevirt.models.V1alpha3VirtualMachineInstance#Start listing and watching...

I believe it would be best for me to provide you with the repository link to the file -

https://github.com/sharath-sri-chellappa/kubevirt-java-client/blob/main/examples/VMIInformerExample.java#L306

This contains the entire code that I am trying out here. Could you please take a look and help me out here.

[yue9944882]

@sharath-sri-chellappa thx, will try reproduce this issue from the repo u linked

[sharath-sri-chellappa]

@yue9944882 Any updates here ?

[sharath-sri-chellappa]

@yue9944882 Any updates here ?

Anyone can help here please ?

[yue9944882]

@sharath-sri-chellappa it's still in my list, will try reproduce this in this week, thanks!

[sharath-sri-chellappa]

@yue9944882 Any luck here ? Sorry to repeatedly push on this.