Configure Nextflow with a local K3S cluster.

[Teachh]

Discussed in #4489

^{Originally posted by Teachh November 7, 2023}
Hello guys!

Recently, I have been testing Nextflow with some public pipelines, and after playing a bit, I decided to run this pipelines to Kubernetes a local Kubernetes Cluster (K3S Dist.) that is more powerful.
The problem comes when following this tutorial from the official documentation doesn't work.
The logs show Invalid DER: object is not integer, more specifically:

java.io.IOException: Invalid DER: object is not integer
	at nextflow.k8s.client.SSLUtils$Asn1Object.getInteger(SSLUtils.java:235)
	at nextflow.k8s.client.SSLUtils.next(SSLUtils.java:166)
	at nextflow.k8s.client.SSLUtils.decodePKCS1(SSLUtils.java:161)
	at nextflow.k8s.client.SSLUtils.createKeyStore(SSLUtils.java:106)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
	at java.base/java.lang.reflect.Method.invoke(Method.java:578)
	at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:107)
	at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:323)
	at org.codehaus.groovy.runtime.callsite.StaticMetaMethodSite.invoke(StaticMetaMethodSite.java:44)
	at org.codehaus.groovy.runtime.callsite.StaticMetaMethodSite.call(StaticMetaMethodSite.java:89)
	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:125)
	at nextflow.k8s.client.ConfigDiscovery.createKeyStore0(ConfigDiscovery.groovy:139)
	at nextflow.k8s.client.ConfigDiscovery.createKeyStore(ConfigDiscovery.groovy:145)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
	at java.base/java.lang.reflect.Method.invoke(Method.java:578)
	at org.codehaus.groovy.runtime.callsite.PlainObjectMetaMethodSite.doInvoke(PlainObjectMetaMethodSite.java:48)
	at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSiteNoUnwrapNoCoerce.invoke(PogoMetaMethodSite.java:189)
	at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.callCurrent(PogoMetaMethodSite.java:57)
	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:51)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:171)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:203)
	at nextflow.k8s.client.ConfigDiscovery.createKeyManagers(ConfigDiscovery.groovy:161)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
	at java.base/java.lang.reflect.Method.invoke(Method.java:578)
	at org.codehaus.groovy.runtime.callsite.PlainObjectMetaMethodSite.doInvoke(PlainObjectMetaMethodSite.java:48)
	at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSiteNoUnwrapNoCoerce.invoke(PogoMetaMethodSite.java:189)
	at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.callCurrent(PogoMetaMethodSite.java:57)
	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:51)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:171)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:194)
	at nextflow.k8s.client.ConfigDiscovery.fromKubeConfig(ConfigDiscovery.groovy:127)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
	at java.base/java.lang.reflect.Method.invoke(Method.java:578)
	at org.codehaus.groovy.runtime.callsite.PlainObjectMetaMethodSite.doInvoke(PlainObjectMetaMethodSite.java:48)
	at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSiteNoUnwrapNoCoerce.invoke(PogoMetaMethodSite.java:189)
	at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.callCurrent(PogoMetaMethodSite.java:57)
	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:51)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:171)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:212)
	at nextflow.k8s.client.ConfigDiscovery.discover(ConfigDiscovery.groovy:61)
	at nextflow.k8s.client.ClientConfig.discover(ClientConfig.groovy:98)
	at nextflow.k8s.K8sConfig.clientDiscovery(K8sConfig.groovy:260)
	at nextflow.k8s.K8sConfig.memoizedMethodPriv$getClient(K8sConfig.groovy:213)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
	at java.base/java.lang.reflect.Method.invoke(Method.java:578)
	at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:107)
	at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:323)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1258)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1035)
	at org.codehaus.groovy.runtime.InvokerHelper.invokePogoMethod(InvokerHelper.java:1036)
	at org.codehaus.groovy.runtime.InvokerHelper.invokeMethod(InvokerHelper.java:1019)
	at org.codehaus.groovy.runtime.InvokerHelper.invokeMethodSafe(InvokerHelper.java:97)
	at nextflow.k8s.K8sConfig$_closure2.doCall(K8sConfig.groovy)
	at nextflow.k8s.K8sConfig$_closure2.doCall(K8sConfig.groovy)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
	at java.base/java.lang.reflect.Method.invoke(Method.java:578)
	at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:107)
	at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:323)
	at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:274)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1035)
	at groovy.lang.Closure.call(Closure.java:412)
	at org.codehaus.groovy.runtime.memoize.Memoize$MemoizeFunction.lambda$call$0(Memoize.java:137)
	at org.codehaus.groovy.runtime.memoize.ConcurrentCommonCache.getAndPut(ConcurrentCommonCache.java:137)
	at org.codehaus.groovy.runtime.memoize.ConcurrentCommonCache.getAndPut(ConcurrentCommonCache.java:113)
	at org.codehaus.groovy.runtime.memoize.Memoize$MemoizeFunction.call(Memoize.java:136)
	at groovy.lang.Closure.call(Closure.java:406)
	at nextflow.k8s.K8sConfig.getClient(K8sConfig.groovy)
	at nextflow.k8s.K8sExecutor.register(K8sExecutor.groovy:66)
	at nextflow.executor.Executor.init(Executor.groovy:67)
	at nextflow.executor.ExecutorFactory.createExecutor(ExecutorFactory.groovy:210)
	at nextflow.executor.ExecutorFactory.getExecutor(ExecutorFactory.groovy:201)
	at nextflow.script.ProcessDef.run(ProcessDef.groovy:206)
	at nextflow.script.BindableDef.invoke_a(BindableDef.groovy:51)
	at nextflow.script.ComponentDef.invoke_o(ComponentDef.groovy:40)
	at nextflow.extension.ChannelEx.or(ChannelEx.groovy:132)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
	at java.base/java.lang.reflect.Method.invoke(Method.java:578)
	at org.codehaus.groovy.runtime.metaclass.ReflectionMetaMethod.invoke(ReflectionMetaMethod.java:54)
	at org.codehaus.groovy.runtime.metaclass.NewInstanceMetaMethod.invoke(NewInstanceMetaMethod.java:54)
	at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:323)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1258)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1035)
	at groovy.runtime.metaclass.NextflowDelegatingMetaClass.invokeMethod(NextflowDelegatingMetaClass.java:64)
	at org.codehaus.groovy.runtime.callsite.PojoMetaClassSite.call(PojoMetaClassSite.java:44)
	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:125)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:139)
	at Script_4518c005$_runScript_closure2$_closure4.doCall(Script_4518c005:16)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
	at java.base/java.lang.reflect.Method.invoke(Method.java:578)
	at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:107)
	at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:323)
	at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:274)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1035)
	at groovy.lang.Closure.call(Closure.java:412)
	at groovy.lang.Closure.call(Closure.java:406)
	at nextflow.script.WorkflowDef.run0(WorkflowDef.groovy:204)
	at nextflow.script.WorkflowDef.run(WorkflowDef.groovy:188)
	at nextflow.script.BindableDef.invoke_a(BindableDef.groovy:51)
	at nextflow.script.ChainableDef$invoke_a.call(Unknown Source)
	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:125)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:139)
	at nextflow.script.BaseScript.run0(BaseScript.groovy:183)
	at nextflow.script.BaseScript.run(BaseScript.groovy:192)
	at nextflow.script.ScriptParser.runScript(ScriptParser.groovy:229)
	at nextflow.script.ScriptRunner.run(ScriptRunner.groovy:224)
	at nextflow.script.ScriptRunner.execute(ScriptRunner.groovy:130)
	at nextflow.cli.CmdRun.run(CmdRun.groovy:368)
	at nextflow.cli.Launcher.run(Launcher.groovy:494)
	at nextflow.cli.Launcher.main(Launcher.groovy:653)

Some information:

• Executing kubectl get nodes returns me the actual cluster correctly.
• My nextflow.conf is:

wave {
  enabled = true
}

fusion {
  enabled = true
}

process {
  executor = 'k8s'
}

k8s {
  context = 'default'
  namespace = 'nextflow'
  serviceAccount = 'default'
}

• The pipeline is:

#!/usr/bin/env nextflow
nextflow.enable.dsl=2 

process sayHello {
  input: 
    val x
  output:
    stdout
  script:
    """
    echo '$x world!'
    """
}

workflow {
  Channel.of('Bonjour', 'Ciao', 'Hello', 'Hola') | sayHello | view
}

• Using this command: nextflow run hello -work-dir s3://mybucket/scratch

Thanks in advance!

[Teachh]

For anyone that is using a local K3S cluster, generate a new config file using your new Service Account instead of using the default config file generated by the master node. The decryption is generating this problem and it's not a good practice to share the admin config file.