New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat!: Rewrite templating. #6442
Conversation
Signed-off-by: Alex Collins <alex_collins@intuit.com>
|
The following tests are failing: TestNestedTemplateParallelismLimit TestParamAggregation TestStepsFailFast TestInputParametersAsJson TestExpandWithItemsMap TestSubstituteGlobalVariables |
|
@mac9416 I'd love to get your thoughts on this. |
|
Will take a look! |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Having only really looked at replace.go, I love the approach. Will read some more and try to see why the tests might be failing / how to fix that.
| if err != nil { | ||
| return err | ||
| } | ||
| data, err = json.Marshal(y) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think you could equivalently call json.Valid and return y directly. But I guess then you wouldn't be able to bubble up the marshalling error.
|
I haven't validated this yet, but I wonder if the test failures are because I'm not sure what issues the |
Signed-off-by: Alex Collins <alex_collins@intuit.com>
|
@alexec is that a real test failure or a flake test failure? I get the same thing consistently locally. |
|
|
|
It seems to me that we have some edge case behavior going on that it might be hard to compensate for. |
yes this edge case scenario, but it is a real usecase. you can add below code will work. Templates substitution will happen in execWfSpec := woc.execWf.Spec
// To Avoid the stale Global parameter value substitution to templates.
// Updated Global parameter values will be substituted in 'executetemplate' for templates.
execWfSpec.Templates = nil |
workflow/controller/operator.go
Outdated
| return err | ||
| } | ||
| return nil | ||
| return template.Replace(&woc.execWf.Spec, woc.globalParams, true) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
| return template.Replace(&woc.execWf.Spec, woc.globalParams, true) | |
| execWfSpec := woc.execWf.Spec | |
| // To Avoid the stale Global parameter value substitution to templates. | |
| // Updated Global parameter values will be substituted in 'executetemplate' for templates. | |
| execWfSpec.Templates = nil | |
| return template.Replace(&execWfSpec, woc.globalParams, true) |
@sarabala1979 like this?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yes
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
return template.Replace(&execWfSpec, woc.globalParams, true)
|
I'm closing this PR. I think it is too risky. |
|
I really don't know what the best solution is with templates. Bugs are a game of whack-a-mole. |
|
Given:
I feel like it's worth either playing a bit more whack-a-mole or removing the expression template feature. Is there any set of tests/validations we could perform on #6285 which would provide sufficient confidence that it is safe? Would be happy to schedule a call to discuss what that would look like. |
|
It was my understanding from you email that the vulnerability (#6441) exists for any template. It sound like you think it only exists for expression tag templates. Users can remove the vulnerability by starting the controller using Can you please double-confirm if this is only a problem with expression templates? That given, if this is a new issue, or an existing issue, the we should have a plan to fix it. That doesn't need to be in v3.1, but should probably be in v3.2 (releases take 3 months to rollout). This PR (#6442) is a big change, and therefore risky, so I'd want to do more testing. I'm not sure that'd make it in time for v3.2 Your PR (#6285) isolates changes to the expression code. So this is low risk and suitable for back-porting to v3.1. |
That is all correct. I should have made the email more clear. I believe the non-expression tag template code avoids the vulnerability by running the evaluated expression value through
Yep. I like this PR more because it clearly isolates expressions (and their evaluated values) to the strings in which they're defined. But I agree, it's riskier.
In that case, I'll spend some time validating the metrics template behavior (I'm just slightly uneasy about that code) and then un-mark it as a draft. Still happy to hop on a call if anything here could use clarification or you have any guidance on additional work you'd like to see done for #6285. |
Signed-off-by: Alex Collins <alex_collins@intuit.com>
|
Lets go with your PR for v3.1, and maybe in v3.2 or v3.3 we can consider this more riskly change. |
Signed-off-by: Alex Collins <alex_collins@intuit.com>

Signed-off-by: Alex Collins alex_collins@intuit.com
Our current templating system is a source of many edge-case bugs and is full of sticking-plasters to deal with these edge-cases. It also has a security issue that allows users to re-write a workflow using input parameters (see #6441) that cannot be fixed with the current solution.
I expect some users are probably using the ability to re-write templates to insert or change workflows, so this is a breaking change.
This PoC changes the solution, so that rather than templating the whole object, we only template values that are strings.