Prepare Nextflow for the CWS Plugin

[Lehmann-Fabian]

This PR is necessary to publish my CWS Plugin finally. The plugin is finished, and you find it here.
In the K8sResponseException, I introduce a new field to store the error code.
For GroupKey, I need to get the target to submit all input data to the scheduler.
Besides that, all changes are only about access modifiers to allow inheriting classes access to required fields.

None of the changes influences the current Nextflow behavior or features. Nor are there any side effects.

[pditommaso]

These are all small changes. However, the side effect of them, internal APIs will be made accessible to components that are not meant to consume them (especially plugins).

What's the value for the scheduling sub-system to access rateLimiter or K8s client error code (and not other executor errors)? As a side thought, maybe that information should be accessed with a dedicated API extension

[Lehmann-Fabian]

Thanks for your feedback. Let me explain why I think the following changes are the right way to go.
rateLimiter:
https://github.com/CommonWorkflowScheduler/nf-cws/blob/master/plugins/nf-cws/src/main/nextflow/cws/processor/CWSTaskPollingMonitor.groovy#L50-L77
Here, I overwrite the submitPendingTask method to implement a batching strategy, not to start scheduling before all ready-to-run tasks are submitted. Therefore, I copied the original code and only added three lines for batching. In the original code, you use the submitRateLimiter, which I also want to support. However, I cannot access it because it is private. For sure, I can create a second submitRateLimiter, as the submitRateLimit() method is protected.
https://github.com/nextflow-io/nextflow/blob/master/modules/nextflow/src/main/groovy/nextflow/processor/TaskPollingMonitor.groovy#L285
But using the same instance as the superclass for two reasons would be better.
First, the submitRateLimiter is currently only used in the submitPendingTask method, which I overwrite.
And second, if the TaskPollingMonitor is later adjusted, it might cause problems.

ErrorCode:
It is in general useful to have the errorCode in the Kubernetes Exception.
Nextflow could handle 404 (not found) problems differently than other Exceptions. I often saw whole pipelines failing if Nextflow couldn't delete a pod after a task was successfully finished because of a 404 error. But this is another topic.
However, I deal with 404 differently than with other errors in my plugin.
https://github.com/CommonWorkflowScheduler/nf-cws/blob/master/plugins/nf-cws/src/main/nextflow/cws/k8s/K8SSchedulerClient.groovy#L66
But if you don't allow me to save the error code into the exception, I can parse it from the Exception message. However, this could cause errors if the message is changed in the future.
https://github.com/nextflow-io/nextflow/blob/master/modules/nextflow/src/main/groovy/nextflow/k8s/client/K8sClient.groovy#L675

So it is up to you. If you decline the changes, I will work around it to get my plugin working anyway, but having the changes will prevent me from using redundant values or reflections and String parsing.

[pditommaso]

Here, I overwrite the submitPendingTask method to implement a batching strategy

I see, but using this approach, CWS is strictly coupled with the nextflow implementation. I believe the CWS scheduler should manage the batching independently on the workflow engine. Don't you agree?

The same for the K8s error code

[Lehmann-Fabian]

I got your point, and I have an idea of how to avoid copying the code of the TaskPollingMonitor.

The error code is not used for anything scheduling-specific. But as Nextflow is built around the paradigm "Do not install anything," I start the scheduler if necessary at runtime, and there I want to catch more details about the problems. Here, I don't see that I couple to close on Nextflow, but this is definitely an unimportant aspect of my CWS Plugin.

I removed both changes from this PR so that you can merge it, and I can publish the Plugin soon.

[Lehmann-Fabian]

I have further reduced the PR. Now there are only three unavoidable changes.

[pditommaso]

It seems reasonable.

[Lehmann-Fabian]

Thanks for merging. I'll publish the plugin later today.

[pditommaso]

You are welcome