Skip to content
Permalink
Browse files

[FIXED JENKINS-42753] Use AST transformation to generate runtime model

There've been any number of problems in the past due to the closure
translation approach for going from `pipeline { ... }` to the `Root`
object we need to actually run. I won't bother going into them here,
just...yeah. There's lots. Until this change goes in, there are 4
special CPS-transformed classes for the translation, plus truly
bizarre special-casing logic for handling environment and when
expressions via looping circuits of script.evaluate calls. I'm
serious. It's sad.

This replaces all that by instead generating Root and its children via
an AST transformation at parsing/validation time. It gets us away from
ever having to worry about translating or evaluating within the CPS
code, using lazily evaluated closures to support things like declaring
cross-referencing environment variables out of order, or environment
variables depending on credentials variables, etc...

Anyway, it works. It's more maintainable. It's cleaner. It's actually
very slightly faster (since we've eliminated the closure translation
stuff from CPS execution, and got rid of a now-redundant additional
parse/validate run to get the execution model). And it's completely
compatible syntax-wise (hopefully! It's not impossible that I missed
some weird expressions in environment variable values, but we'll cross
that bridge if we get to it).
  • Loading branch information...
abayer committed Jul 24, 2017
1 parent 7707b42 commit 963d58beea949d4d8eb468986b0908b075205b9c
Showing with 1,876 additions and 1,386 deletions.
  1. +10 −1 pipeline-model-definition/pom.xml
  2. +72 −155 ...line-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/Utils.groovy
  3. +8 −8 ...roovy/org/jenkinsci/plugins/pipeline/modeldefinition/model/AbstractBuildConditionResponder.groovy
  4. +7 −1 ...odel-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/model/Agent.groovy
  5. +55 −223 ...efinition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/model/Environment.groovy
  6. +4 −4 ...-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/model/Libraries.groovy
  7. +9 −2 ...el-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/model/Options.groovy
  8. +3 −1 ...definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/model/Parameters.groovy
  9. +5 −1 ...-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/model/PostBuild.groovy
  10. +5 −0 ...-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/model/PostStage.groovy
  11. +18 −53 ...model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/model/Root.groovy
  12. +13 −41 ...odel-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/model/Stage.groovy
  13. +6 −6 ...ion/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/model/StageConditionals.groovy
  14. +6 −2 ...del-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/model/Stages.groovy
  15. +6 −4 ...definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/model/StepsBlock.groovy
  16. +8 −14 ...odel-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/model/Tools.groovy
  17. +3 −1 ...l-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/model/Triggers.groovy
  18. +399 −0 ...ition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/parser/ASTParserUtils.groovy
  19. +1 −1 .../src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/parser/BlockStatementMatch.groovy
  20. +1 −1 ...definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/parser/Converter.groovy
  21. +110 −210 ...finition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/parser/ModelParser.groovy
  22. +603 −0 ...rc/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/parser/RuntimeASTTransformer.groovy
  23. +0 −5 ...odel-definition/src/main/java/org/jenkinsci/plugins/pipeline/modeldefinition/ModelStepLoader.java
  24. +2 −2 ...ion/src/main/java/org/jenkinsci/plugins/pipeline/modeldefinition/SyntheticStageGraphListener.java
  25. +8 −5 ...src/main/java/org/jenkinsci/plugins/pipeline/modeldefinition/parser/GroovyShellDecoratorImpl.java
  26. +9 −0 ...tion/src/main/java/org/jenkinsci/plugins/pipeline/modeldefinition/when/impl/AllOfConditional.java
  27. +9 −0 ...tion/src/main/java/org/jenkinsci/plugins/pipeline/modeldefinition/when/impl/AnyOfConditional.java
  28. +8 −5 ...ion/src/main/java/org/jenkinsci/plugins/pipeline/modeldefinition/when/impl/BranchConditional.java
  29. +8 −8 ...rc/main/java/org/jenkinsci/plugins/pipeline/modeldefinition/when/impl/EnvironmentConditional.java
  30. +55 −2 ...src/main/java/org/jenkinsci/plugins/pipeline/modeldefinition/when/impl/ExpressionConditional.java
  31. +10 −0 ...nition/src/main/java/org/jenkinsci/plugins/pipeline/modeldefinition/when/impl/NotConditional.java
  32. +0 −219 ...n/src/main/resources/org/jenkinsci/plugins/pipeline/modeldefinition/ClosureModelTranslator.groovy
  33. +0 −84 .../src/main/resources/org/jenkinsci/plugins/pipeline/modeldefinition/MethodsToListTranslator.groovy
  34. +65 −57 ...inition/src/main/resources/org/jenkinsci/plugins/pipeline/modeldefinition/ModelInterpreter.groovy
  35. +0 −89 ...nition/src/main/resources/org/jenkinsci/plugins/pipeline/modeldefinition/OptionsTranslator.groovy
  36. +0 −87 ...rc/main/resources/org/jenkinsci/plugins/pipeline/modeldefinition/PropertiesToMapTranslator.groovy
  37. +5 −0 ...urces/org/jenkinsci/plugins/pipeline/modeldefinition/when/impl/ExpressionConditionalScript.groovy
  38. +34 −0 ...ne-model-definition/src/test/java/org/jenkinsci/plugins/pipeline/modeldefinition/OptionsTest.java
  39. +21 −0 ...model-definition/src/test/java/org/jenkinsci/plugins/pipeline/modeldefinition/ParametersTest.java
  40. +9 −0 ...line-model-definition/src/test/java/org/jenkinsci/plugins/pipeline/modeldefinition/ToolsTest.java
  41. +24 −0 ...e-model-definition/src/test/java/org/jenkinsci/plugins/pipeline/modeldefinition/TriggersTest.java
  42. +20 −0 ...src/test/java/org/jenkinsci/plugins/pipeline/modeldefinition/steps/CredentialWrapperStepTest.java
  43. +28 −24 ...definition/model/PropertyOptionContainer.groovy → test/resources/credentialsUsedInWhenEnv.groovy}
  44. +53 −0 pipeline-model-definition/src/test/resources/credentialsUsedInWhenExpression.groovy
  45. +17 −19 ...gins/pipeline/modeldefinition/model/MethodsToList.groovy → test/resources/envVarInOptions.groovy}
  46. +15 −24 ...s/pipeline/modeldefinition/LibrariesTranslator.groovy → test/resources/envVarInParameters.groovy}
  47. +45 −0 pipeline-model-definition/src/test/resources/envVarInTools.groovy
  48. +18 −9 ...ine/modeldefinition/model/StepBlockWithOtherArgs.groovy → test/resources/envVarInTriggers.groovy}
  49. +19 −7 ...ns/pipeline/modeldefinition/model/PropertiesToMap.groovy → test/resources/envVarInWrapper.groovy}
  50. +14 −1 pipeline-model-extensions/pom.xml
  51. +9 −2 ...va/org/jenkinsci/plugins/pipeline/modeldefinition/when/DeclarativeStageConditionalDescriptor.java
  52. +19 −8 pom.xml
@@ -44,7 +44,8 @@
<artifactId>maven-hpi-plugin</artifactId>
<extensions>true</extensions>
<configuration>
<compatibleSinceVersion>0.8</compatibleSinceVersion>
<!-- TODO: Update this to whatever we end up releasing as -->
<compatibleSinceVersion>1.1.9-SNAPSHOT</compatibleSinceVersion>
</configuration>
</plugin>
<plugin>
@@ -134,6 +135,14 @@
<groupId>org.jenkins-ci.plugins.workflow</groupId>
<artifactId>workflow-cps-global-lib</artifactId>
</dependency>
<dependency>
<groupId>org.jenkins-ci.plugins</groupId>
<artifactId>mailer</artifactId>
</dependency>
<dependency>
<groupId>org.jenkins-ci.plugins</groupId>
<artifactId>git-client</artifactId>
</dependency>
<!-- Dependency on tombstoned declarative agent plugin to make sure we don't end up with a
bad old version installed -->
<dependency>
@@ -41,31 +41,27 @@ import hudson.model.Result
import hudson.triggers.Trigger
import org.apache.commons.codec.digest.DigestUtils
import org.apache.commons.lang.StringUtils
import org.codehaus.groovy.ast.ASTNode
import org.codehaus.groovy.ast.stmt.BlockStatement
import org.codehaus.groovy.ast.stmt.Statement
import org.codehaus.groovy.control.SourceUnit
import org.jenkinsci.plugins.pipeline.StageStatus
import org.jenkinsci.plugins.pipeline.StageTagsMetadata
import org.jenkinsci.plugins.pipeline.SyntheticStage
import org.jenkinsci.plugins.pipeline.modeldefinition.actions.ExecutionModelAction
import org.jenkinsci.plugins.pipeline.modeldefinition.actions.DeclarativeJobPropertyTrackerAction
import org.jenkinsci.plugins.pipeline.modeldefinition.ast.ModelASTEnvironment
import org.jenkinsci.plugins.pipeline.modeldefinition.ast.ModelASTInternalFunctionCall
import org.jenkinsci.plugins.pipeline.modeldefinition.ast.ModelASTPipelineDef
import org.jenkinsci.plugins.pipeline.modeldefinition.ast.ModelASTStage
import org.jenkinsci.plugins.pipeline.modeldefinition.ast.ModelASTStages
import org.jenkinsci.plugins.pipeline.modeldefinition.ast.ModelASTValue
import org.jenkinsci.plugins.pipeline.modeldefinition.ast.ModelASTWhenCondition
import org.jenkinsci.plugins.pipeline.modeldefinition.ast.ModelASTWhenContent
import org.jenkinsci.plugins.pipeline.modeldefinition.ast.ModelASTWhenExpression
import org.jenkinsci.plugins.pipeline.modeldefinition.model.Environment
import org.jenkinsci.plugins.pipeline.modeldefinition.model.MethodsToList
import org.jenkinsci.plugins.pipeline.modeldefinition.model.StageConditionals
import org.jenkinsci.plugins.pipeline.modeldefinition.model.Root
import org.jenkinsci.plugins.pipeline.modeldefinition.model.Stage
import org.jenkinsci.plugins.pipeline.modeldefinition.model.StepsBlock
import org.jenkinsci.plugins.pipeline.modeldefinition.parser.Converter
import org.jenkinsci.plugins.pipeline.modeldefinition.steps.CredentialWrapper
import org.jenkinsci.plugins.pipeline.modeldefinition.when.DeclarativeStageConditional
import org.jenkinsci.plugins.pipeline.modeldefinition.when.DeclarativeStageConditionalDescriptor
import org.jenkinsci.plugins.scriptsecurity.sandbox.whitelists.Whitelisted
import org.jenkinsci.plugins.structs.SymbolLookup
import org.jenkinsci.plugins.structs.describable.DescribableModel
import org.jenkinsci.plugins.structs.describable.UninstantiatedDescribable
import org.jenkinsci.plugins.workflow.actions.LabelAction
import org.jenkinsci.plugins.workflow.actions.TagsAction
@@ -94,7 +90,6 @@ import javax.annotation.Nonnull
import javax.annotation.Nullable
import javax.lang.model.SourceVersion
import java.lang.reflect.ParameterizedType
import java.lang.reflect.Type
import java.util.concurrent.TimeUnit

// TODO: Prune like mad once we have step-in-groovy and don't need these static whitelisted wrapper methods.
@@ -172,27 +167,6 @@ public class Utils {

}

/**
* Finds the parameterized type argument for a {@link MethodsToList} class and returns it.
*
* @param c A class.
* @return The parameterized type argument for the class, if it's a {@link MethodsToList} class. Null otherwise.
*/
public static Class<Describable> getMethodsToListType(Class c) {
Class retClass
c.genericInterfaces.each { Type t ->
if (t instanceof ParameterizedType) {
if (t.getRawType() instanceof Class
&& MethodsToList.class.isAssignableFrom((Class)t.getRawType())
&& t.getActualTypeArguments().first() instanceof Class) {
retClass = (Class)t.actualTypeArguments.first()
}
}
}

return retClass
}

/**
* Simple wrapper for isInstance to avoid whitelisting issues.
*
@@ -226,6 +200,14 @@ public class Utils {
}
}

static Object getScriptPropOrParam(CpsScript script, String name) {
try {
return script.getProperty(name)
} catch (MissingPropertyException e) {
return script.params?.get(name)
}
}

static boolean hasJobProperties(CpsScript script) {
WorkflowRun r = script.$build()

@@ -247,96 +229,6 @@ public class Utils {
}
}

static Root attachDeclarativeActions(@Nonnull Root root, CpsScript script) {
WorkflowRun r = script.$build()
ModelASTPipelineDef model = Converter.parseFromWorkflowRun(r)

if (model != null) {
ModelASTStages stages = model.stages

stages.removeSourceLocation()
if (r.getAction(ExecutionModelAction.class) == null) {
r.addAction(new ExecutionModelAction(stages))
}

if (root != null) {
root = populateFromModel(r, root, model)
}
}

return root
}

static Root populateFromModel(@Nonnull WorkflowRun r, @Nonnull Root root, @Nonnull ModelASTPipelineDef model) {
root.environment = environmentFromAST(r, model.environment)

List<Stage> stagesWithEnvs = []

root.stages.stages.each { s ->
stagesWithEnvs.add(populateSingleStageFromModel(r, model.stages, s))
}

root.stages.stages = stagesWithEnvs

return root
}

private static Stage populateSingleStageFromModel(@Nonnull WorkflowRun r, @Nonnull ModelASTStages allStages,
@Nonnull Stage stage) {
ModelASTStage astStage = allStages.stages.find { it.name == stage.name }
stage.environment = environmentFromAST(r, astStage.environment)

if (astStage.when != null) {
List<DeclarativeStageConditional<? extends DeclarativeStageConditional>> processedConditions =
astStage.when.conditions.collect { c ->
stageConditionalFromAST(c)
}

stage.when(new StageConditionals(processedConditions))
}

if (stage.parallel != null && !stage.parallel.stages.isEmpty()) {
List<Stage> nestedStages = []
stage.parallel.stages.each { s ->
nestedStages.add(populateSingleStageFromModel(r, astStage.parallel, s))
}
stage.parallel.stages = nestedStages
}

return stage
}

/**
* Translates the {@link ModelASTWhenContent} into a {@link DeclarativeStageConditional}.
*
* @param w
* @return A populated {@link DeclarativeStageConditional}
*/
private static DeclarativeStageConditional stageConditionalFromAST(ModelASTWhenContent w) {
DeclarativeStageConditional c = null
DeclarativeStageConditionalDescriptor desc = DeclarativeStageConditionalDescriptor.byName(w.name)

if (w instanceof ModelASTWhenCondition) {
if (desc.allowedChildrenCount == 0) {
Object[] arg = new Object[1]
arg[0] = w.args?.argListToMap()
c = (DeclarativeStageConditional)getDescribable(w.name, desc.clazz, arg).instantiate()
} else if (desc.allowedChildrenCount == 1) {
DeclarativeStageConditional single = stageConditionalFromAST(w.children.first())
c = (DeclarativeStageConditional)getDescribable(w.name, desc.clazz, single).instantiate()
} else {
List<DeclarativeStageConditional> nested = w.children.collect { stageConditionalFromAST(it) }
c = (DeclarativeStageConditional)getDescribable(w.name, desc.clazz, nested).instantiate()
}
} else if (w instanceof ModelASTWhenExpression) {
ModelASTWhenExpression expr = (ModelASTWhenExpression)w

c = (DeclarativeStageConditional)getDescribable(w.name, desc.clazz, expr.codeBlockAsString()).instantiate()
}

return c
}

/**
* Takes a string and makes sure it starts/ends with double quotes so that it can be evaluated correctly.
*
@@ -373,16 +265,21 @@ public class Utils {
}

static String unescapeDollars(String s) {
return StringUtils.replace(s, Environment.DOLLAR_PLACEHOLDER, '$')
return s
}

static List<List<String>> getEnvCredentials(Environment environment, CpsScript script, Environment parent = null) {
List<List<String>> credsTuples = new ArrayList<>()
static Map<String,Closure> getCredsFromResolver(Environment environment, CpsScript script) {
if (environment != null) {
credsTuples.addAll(environment.getCredsMap(script, parent)?.collect { k, v ->
[k, v]
})
environment.credsResolver.setScript(script)
return environment.credsResolver.closureMap
} else {
return [:]
}
}

static List<List<String>> getEnvCredentials(Environment environment, CpsScript script, Environment parent = null) {
List<List<String>> credsTuples = new ArrayList<>()
// TODO: Creds
return credsTuples
}

@@ -415,33 +312,6 @@ public class Utils {
}
}

static Environment environmentFromAST(WorkflowRun r, ModelASTEnvironment inEnv) {
if (inEnv != null) {
Environment env = new Environment()

Map<String, Environment.EnvValue> inMap = [:]
Map<String, Environment.EnvValue> credMap = [:]
inEnv.variables.each { k, v ->
if (v instanceof ModelASTInternalFunctionCall) {
ModelASTInternalFunctionCall func = (ModelASTInternalFunctionCall)v
// TODO: JENKINS-41759 - look up the right method and dispatch accordingly, with the right # of args
Environment.EnvValue envVal = new Environment.EnvValue(isLiteral: func.args.first().isLiteral(),
value: func.args.first().value)
credMap.put(k.key, envVal)
} else {
ModelASTValue val = (ModelASTValue)v
Environment.EnvValue envVal = new Environment.EnvValue(isLiteral: val.isLiteral(), value: val.value)
inMap.put(k.key, envVal)
}
}

env.setValueMap(inMap)
env.setCredsMap(credMap)
return env
} else {
return null
}
}

static Predicate<FlowNode> endNodeForStage(final BlockStartNode startNode) {
return new Predicate<FlowNode>() {
@@ -619,6 +489,12 @@ public class Utils {
return knownTypes
}

@Whitelisted
static <T> T instantiateDescribable(Class<T> c, Map<String, ?> args) {
DescribableModel<T> model = new DescribableModel<>(c)
return model?.instantiate(args)
}

/**
* Determines whether a given {@link UninstantiatedDescribable} is of a given type.
*
@@ -634,6 +510,7 @@ public class Utils {
/**
* @param c The closure to wrap.
*/
@Whitelisted
public static StepsBlock createStepsBlock(Closure c) {
// Jumping through weird hoops to get around the ejection for cases of JENKINS-26481.
StepsBlock wrapper = new StepsBlock()
@@ -1000,4 +877,44 @@ public class Utils {
return existing
}


/**
* Obtains the source text of the given {@link org.codehaus.groovy.ast.ASTNode}.
*/
public static String getSourceTextForASTNode(@Nonnull ASTNode n, @Nonnull SourceUnit sourceUnit) {
def result = new StringBuilder();
int beginLine = n.getLineNumber()
int endLine = n.getLastLineNumber()
int beginLineColumn = n.getColumnNumber()
int endLineLastColumn = n.getLastColumnNumber()

//The node seems to be lying about the last line, so go through each statement to try to make sure
if (n instanceof BlockStatement) {
for (Statement s : n.statements) {
if (s.lineNumber < beginLine) {
beginLine = s.lineNumber
beginLineColumn = s.columnNumber
}
if (s.lastLineNumber > endLine) {
endLine = s.lastLineNumber
endLineLastColumn = s.lastColumnNumber
}
}
}
for (int x = beginLine; x <= endLine; x++) {
String line = sourceUnit.source.getLine(x, null);
if (line == null)
throw new AssertionError("Unable to get source line"+x);

if (x == endLine) {
line = line.substring(0, endLineLastColumn - 1);
}
if (x == beginLine) {
line = line.substring(beginLineColumn - 1);
}
result.append(line).append('\n');
}

return result.toString().trim();
}
}
@@ -25,21 +25,21 @@ package org.jenkinsci.plugins.pipeline.modeldefinition.model

import edu.umd.cs.findbugs.annotations.SuppressFBWarnings

import org.jenkinsci.plugins.workflow.job.WorkflowRun
import org.jenkinsci.plugins.workflow.support.steps.build.RunWrapper


/**
* Parent for {@link PostStage} and {@link PostBuild} - containers for condition name/step block pairs.
*
* @author Andrew Bayer
*/
@SuppressFBWarnings(value="SE_NO_SERIALVERSIONID")
public abstract class AbstractBuildConditionResponder<T extends AbstractBuildConditionResponder<T>>
abstract class AbstractBuildConditionResponder<T extends AbstractBuildConditionResponder<T>>
extends MappedClosure<StepsBlock,T> {

AbstractBuildConditionResponder(Map<String,StepsBlock> m) {
super(m)
}

@Override
public void modelFromMap(Map<String,Object> inMap) {
void modelFromMap(Map<String,Object> inMap) {

inMap.each { conditionName, conditionClosure ->
if (conditionName in BuildCondition.getConditionMethods().keySet()) {
@@ -51,7 +51,7 @@ public abstract class AbstractBuildConditionResponder<T extends AbstractBuildCon
}
}

public Closure closureForSatisfiedCondition(String conditionName, Object runWrapperObj) {
Closure closureForSatisfiedCondition(String conditionName, Object runWrapperObj) {
if (getMap().containsKey(conditionName)) {
BuildCondition condition = BuildCondition.getConditionMethods().get(conditionName)
if (condition != null && condition.meetsCondition(runWrapperObj)) {
@@ -62,7 +62,7 @@ public abstract class AbstractBuildConditionResponder<T extends AbstractBuildCon
return null
}

public boolean satisfiedConditions(Object runWrapperObj) {
boolean satisfiedConditions(Object runWrapperObj) {
Map<String,BuildCondition> conditions = BuildCondition.getConditionMethods()

return BuildCondition.orderedConditionNames.any { conditionName ->

0 comments on commit 963d58b

Please sign in to comment.
You can’t perform that action at this time.