Custom Job Stages broken after upgrade to 1.22.x

[dil-dbakker]

Issue Summary:

We are currently unable to upgrade our prod spinnaker instance to 1.22.x due to issues with our custom job stages.

Cloud Provider(s):

On Prem - Kubernetes

Environment:

Self hosted spinnaker on Kubernetes

Feature Area:

Orca

Description:

Expected: After upgrade our custom job stages in our templates can be saved and run
Actual: When running a custom job stage we get the following error message:

Exception
Cannot invoke method split() on null object

Steps to Reproduce:

Create a Custom Job Stage containing a parameter with no mapping attribute.
Run this stage in a pipeline.

Additional Details:

Here is an example of one of our existing Custom Jobs that works in Spinnaker 1.20.x but is broken in 1.22.x

- label: Send RFC Email
        type: rfcEmail
        description: Send a RFC for a deployment
        cloudProvider: kubernetes
        account: test-account
        credentials: test-account
        application: ${#currentStage()['context']['parameters']['application']}
        waitforCompletion: true
        parameters:
          - name: application
            label: Spinnaker app
            description: Spinnaker application
          - name: appVersion
            label: Application version
            description: The version of the application being deployed
          - name: ccAddress
            label: CC recipient for the RFC email
            description: Email address to CC the RFC email to
        manifest:
          apiVersion: batch/v1
          kind: Job
          metadata:
            labels:
              type: tests
            generateName: send-rfc-email-${#currentStage()['context']['parameters']['application']}-
            namespace: rfc-email
          spec:
            backoffLimit: 2
            template:
              metadata:
                labels:
                  type: tests
              spec:
                containers:
                  - command:
                      - /script/send-rfc-email.sh
                    env:
                      - name: SCRIPT_APP_NAME
                        value: ${#currentStage()['context']['parameters']['application']}
                      - name: SCRIPT_APP_VERSION
                        value: ${#currentStage()['context']['parameters']['appVersion']}
                      - name: SCRIPT_CC_ADDRESS
                        value: ${#currentStage()['context']['parameters']['ccAddress']}
                    image: "namshi/smtp:latest"
                    name: send-rfc-email
                    volumeMounts:
                      - mountPath: /script/send-rfc-email.sh
                        name: send-rfc-email
                        subPath: send-rfc-email.sh
                restartPolicy: Never
                volumes:
                  - configMap:
                      defaultMode: 511
                      name: send-rfc-email
                    name: send-rfc-email

Here is a previous discussion related to this issue #5147

[ezimanyi]

@marchello2000 : This looks likely to have been caused by spinnaker/orca#3760; mind taking a look? Thanks!

[ezimanyi]

spinnaker/orca#3830 also possibly related

[marchello2000]

I will take a look this week

[marchello2000]

@dil-dbakker are you able to post the whole log/callstack for this exception?

[marchello2000]

Also, what your stage JSON looks like

[dil-dbakker]

@marchello2000 I'm a bit late back here, but just in case you are still interested, here is our stage JSON referencing the above job:

"stages": [
--
   {
   "alias": "preconfiguredJob",
   "name": "Submit RFC",
   "parameters": {
   "appVersion": "${parameters['version']}",
   "application": "${ templateVariables.appName }",
   "ccAddress": "${ templateVariables.teamEmailAddress }",
   "delegatedApprovalRfcNumber": "${ templateVariables.delegatedApprovalRfcNumber }",
   "executionId": "${execution['id']}",
   "pipelineParameters": "${'${'}parameters.toString()}",
   "changelogLink": "${ templateVariables.changelogLink ?: \"${#stage('Publish Jira Release').status == 'SUCCEEDED' ? ('https://xxxxxxx.atlassian.net/wiki/display/DDT/' + project_name + '+' + version_name + '+-+' + version_date).replaceAll(' ', '+') : ''' }\" }",
   "sendToTestProject": "false",
   "startTime": "${'${'}new java.text.SimpleDateFormat(\"MM/dd/yyyy HH:mm\").format(new java.util.Date())}"
   },
   "type": "rfcEmail",
   "refId": "rfcEmail",
   "requisiteStageRefIds": ["arnPublishCheck"]
   },

And here is the full stack trace of the error:

"stackTrace": "java.lang.NullPointerException: Cannot invoke method split() on null object\n\tat org.codehaus.groovy.runtime.NullObject.invokeMethod(NullObject.java:91)\n\tat org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:43)\n\tat org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47)\n\tat org.codehaus.groovy.runtime.callsite.NullCallSite.call(NullCallSite.java:34)\n\tat org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47)\n\tat org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:52)\n\tat org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:127)\n\tat com.netflix.spinnaker.orca.clouddriver.pipeline.job.PreconfiguredJobStage.setNestedValue(PreconfiguredJobStage.groovy:97)\n\tat jdk.internal.reflect.GeneratedMethodAccessor1160.invoke(Unknown Source)\n\tat java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat java.base/java.lang.reflect.Method.invoke(Method.java:566)\n\tat org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:101)\n\tat org.codehaus.groovy.runtime.callsite.StaticMetaMethodSite$StaticMetaMethodSiteNoUnwrapNoCoerce.invoke(StaticMetaMethodSite.java:149)\n\tat org.codehaus.groovy.runtime.callsite.StaticMetaMethodSite.callStatic(StaticMetaMethodSite.java:100)\n\tat org.codehaus.groovy.runtime.callsite.AbstractCallSite.callStatic(AbstractCallSite.java:224)\n\tat com.netflix.spinnaker.orca.clouddriver.pipeline.job.PreconfiguredJobStage$_overrideIfNotSetInContextAndOverrideDefault_closure4.doCall(PreconfiguredJobStage.groovy:88)\n\tat jdk.internal.reflect.GeneratedMethodAccessor1159.invoke(Unknown Source)\n\tat java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat java.base/java.lang.reflect.Method.invoke(Method.java:566)\n\tat org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:101)\n\tat groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:323)\n\tat org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:263)\n\tat groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1041)\n\tat groovy.lang.Closure.call(Closure.java:405)\n\tat org.codehaus.groovy.runtime.DefaultGroovyMethods.callClosureForMapEntry(DefaultGroovyMethods.java:5961)\n\tat org.codehaus.groovy.runtime.DefaultGroovyMethods.each(DefaultGroovyMethods.java:2407)\n\tat org.codehaus.groovy.runtime.dgm$187.invoke(Unknown Source)\n\tat org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoMetaMethodSiteNoUnwrapNoCoerce.invoke(PojoMetaMethodSite.java:244)\n\tat org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:53)\n\tat org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:127)\n\tat com.netflix.spinnaker.orca.clouddriver.pipeline.job.PreconfiguredJobStage.overrideIfNotSetInContextAndOverrideDefault(PreconfiguredJobStage.groovy:85)\n\tat jdk.internal.reflect.GeneratedMethodAccessor1046.invoke(Unknown Source)\n\tat java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat java.base/java.lang.reflect.Method.invoke(Method.java:566)\n\tat org.codehaus.groovy.runtime.callsite.PlainObjectMetaMethodSite.doInvoke(PlainObjectMetaMethodSite.java:43)\n\tat org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSiteNoUnwrapNoCoerce.invoke(PogoMetaMethodSite.java:190)\n\tat org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.callCurrent(PogoMetaMethodSite.java:58)\n\tat org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:184)\n\tat com.netflix.spinnaker.orca.clouddriver.pipeline.job.PreconfiguredJobStage.taskGraph(PreconfiguredJobStage.groovy:54)\n\tat com.netflix.spinnaker.orca.api.pipeline.graph.StageDefinitionBuilder.buildTaskGraph(StageDefinitionBuilder.java:48)\n\tat com.netflix.spinnaker.orca.q.StageDefinitionBuildersKt.buildTasks(StageDefinitionBuilders.kt:36)\n\tat com.netflix.spinnaker.orca.q.handler.StartStageHandler.plan(StartStageHandler.kt:171)\n\tat com.netflix.spinnaker.orca.q.handler.StartStageHandler.access$plan(StartStageHandler.kt:60)\n\tat com.netflix.spinnaker.orca.q.handler.StartStageHandler$handle$1$1.invoke(StartStageHandler.kt:97)\n\tat com.netflix.spinnaker.orca.q.handler.StartStageHandler$handle$1$1.invoke(StartStageHandler.kt:60)\n\tat com.netflix.spinnaker.orca.q.handler.AuthenticationAware$sam$java_util_concurrent_Callable$0.call(AuthenticationAware.kt)\n\tat com.netflix.spinnaker.security.AuthenticatedRequest.lambda$wrapCallableForPrincipal$0(AuthenticatedRequest.java:272)\n\tat com.netflix.spinnaker.orca.q.handler.AuthenticationAware$DefaultImpls.withAuth(AuthenticationAware.kt:53)\n\tat com.netflix.spinnaker.orca.q.handler.StartStageHandler.withAuth(StartStageHandler.kt:60)\n\tat com.netflix.spinnaker.orca.q.handler.StartStageHandler$handle$1.invoke(StartStageHandler.kt:79)\n\tat com.netflix.spinnaker.orca.q.handler.StartStageHandler$handle$1.invoke(StartStageHandler.kt:60)\n\tat com.netflix.spinnaker.orca.q.handler.OrcaMessageHandler$withStage$1.invoke(OrcaMessageHandler.kt:85)\n\tat com.netflix.spinnaker.orca.q.handler.OrcaMessageHandler$withStage$1.invoke(OrcaMessageHandler.kt:46)\n\tat com.netflix.spinnaker.orca.q.handler.OrcaMessageHandler$DefaultImpls.withExecution(OrcaMessageHandler.kt:95)\n\tat com.netflix.spinnaker.orca.q.handler.StartStageHandler.withExecution(StartStageHandler.kt:60)\n\tat com.netflix.spinnaker.orca.q.handler.OrcaMessageHandler$DefaultImpls.withStage(OrcaMessageHandler.kt:74)\n\tat com.netflix.spinnaker.orca.q.handler.StartStageHandler.withStage(StartStageHandler.kt:60)\n\tat com.netflix.spinnaker.orca.q.handler.StartStageHandler.handle(StartStageHandler.kt:77)\n\tat com.netflix.spinnaker.orca.q.handler.StartStageHandler.handle(StartStageHandler.kt:60)\n\tat com.netflix.spinnaker.q.MessageHandler$DefaultImpls.invoke(MessageHandler.kt:36)\n\tat com.netflix.spinnaker.orca.q.handler.OrcaMessageHandler$DefaultImpls.invoke(OrcaMessageHandler.kt)\n\tat com.netflix.spinnaker.orca.q.handler.StartStageHandler.invoke(StartStageHandler.kt:60)\n\tat com.netflix.spinnaker.orca.q.audit.ExecutionTrackingMessageHandlerPostProcessor$ExecutionTrackingMessageHandlerProxy.invoke(ExecutionTrackingMessageHandlerPostProcessor.kt:72)\n\tat com.netflix.spinnaker.q.QueueProcessor$callback$1$1.run(QueueProcessor.kt:89)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)\n\tat java.base/java.lang.Thread.run(Thread.java:834)\n",