Change variable interpolation braces #1172

bobcatfish · 2019-08-06T23:00:30Z

Changes

In #850 we decided that to make our syntax more similar to k8s style
syntax, we will use $() for variable substitution instead of ${}. In a
later iteration we may also want to make it so that anything that can be
accesssed as a variable is also available as an env var to the running
container, but that is TBD.

In the context of this discussion we also considered removing variable
substitution entirely, that conversation is continuing.

We have also tried to start calling this functionality "variable
substitution" instead of "templating" to make it clear that we do not
intend to support more complicated templating functionality.

In order to make it clear that we are not intending to support more
complicated templating (via #850), rename the "template" related logic
to "substitution".

@vdemeester suggested "variable interpolation" which seems to be
completely correct but I am hoping that "substitution" is a bit
simpler than "interpolation" (might be less correct tho!)

Fixes #850

Submitter Checklist

These are the criteria that every PR should meet, please check them off as you
review them:

Includes tests (if functionality changed/added)
Includes docs (if user facing)
Commit messages follow commit message best practices

See the contribution guide for more details.

Double check this list of stuff that's easy to miss:

If you are adding a new binary/image to the cmd dir, please update
the release Task and TaskRun to build and release this image

Release Notes

🚨First step in changing ${} syntax to $() 🚨
Adds support for $() syntax in addition to ${}; in #1170 we will remove support for ${}. Please migrate to $()!

tekton-robot · 2019-08-06T23:00:34Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: bobcatfish

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [bobcatfish]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

tekton-robot · 2019-08-06T23:02:44Z

The following is the coverage report on pkg/.
Say /test pull-tekton-pipeline-go-coverage to re-run this coverage report

File	Old Coverage	New Coverage
pkg/reconciler/v1alpha1/pipelinerun/pipelinerun.go	Do not exist	82.4%
pkg/substitution/container_replacements.go	Do not exist	100.0%
pkg/substitution/substitution.go	Do not exist	62.3%

vdemeester

The $() syntax looks like bash variable replacement syntax. Folks actually using that syntax will need a way to escape it.
That was one of the cons. I thought it was a big one but actually, ${…} was one too, and is usually more used than $(…) (that starts a subshell and execute the comand)… so I think we're all good on that.

I also do like substitution more than interpolation, it is indeed simpler 😉

vdemeester · 2019-08-07T05:26:37Z

examples/pipelineruns/output-pipelinerun.yaml

  steps:
  - name: read
    image: ubuntu
    command: ["/bin/bash"]
-    args: ['-c', 'cat /workspace/newworkspace/stuff'] # tests that new targetpath and previous task output is dumped
+    args: ['$(inputs.params.args)'] # tests that new targetpath and previous task output is dumped


ghost · 2019-08-07T12:02:55Z

docs/resources.md

@@ -419,6 +419,7 @@ spec:
  type: cluster
  params:
    - name: url
+


nit: intentional extra line?

whoops thanks :D

ghost · 2019-08-07T12:31:38Z

pkg/substitution/substitution.go

@@ -76,7 +76,8 @@ func ValidateVariableIsolated(name, value, prefix, contextPrefix, locationName,
 // Extract a the first full string expressions found (e.g "${input.params.foo}"). Return
 // "" and false if nothing is found.
 func extractExpressionFromString(s, prefix string) (string, bool) {
-	pattern := fmt.Sprintf("\\$({%s.(?P<var>%s)})", prefix, parameterSubstitution)
+	//TODO(1170): Regex matches both ${} and $(), we will need to remove ${} compatibility.


The most minor of nits: These todos are labelled differently from others elsewhere in the diff: TODO(1170) vs TODO(#1170). The only reason this is noteworthy is because I assume we're going to want to grep for the pattern and remove all of them in future. Might be helpful to future us if all the todos are the same string.

whoops, thanks for catching this! :D

i think while im here i'm going to put that regex into a constant ... 😇

ghost · 2019-08-07T12:33:56Z

test/pipelinerun_test.go

 				// Reference build: https://github.com/knative/build/tree/master/test/docker-basic
 				tb.Step("config-docker", "quay.io/rhpipeline/skopeo:alpine",
 					tb.Command("skopeo"),
-					tb.Args("copy", "docker://gcr.io/build-crd-testing/secret-sauce", "dir:///tmp/"),
+					tb.Args("copy", "${inputs.params.path}", "${inputs.params.dest}"),


Intentionally using deprecated format here? If so, worth adding a TODO(#1170)?

good catch - and actually I meant to mix the formatting but apparently forgot to 🤦‍♀ !

ghost · 2019-08-07T12:35:34Z

I added some v v small quibbles. Otherwise this looks good to me.

bobcatfish · 2019-08-07T19:15:57Z

They were excellent quibbles @sbwsg :D

bobcatfish · 2019-08-07T19:18:32Z

tfw the integration tests passed on the first run and you become nervous they aren't actually being run ... 🤔 🕵️‍♀️ 😅

bobcatfish · 2019-08-07T19:21:14Z

Ready for another look!

tekton-robot · 2019-08-07T19:24:26Z

The following is the coverage report on pkg/.
Say /test pull-tekton-pipeline-go-coverage to re-run this coverage report

File	Old Coverage	New Coverage
pkg/reconciler/v1alpha1/pipelinerun/pipelinerun.go	Do not exist	82.4%
pkg/substitution/container_replacements.go	Do not exist	100.0%
pkg/substitution/substitution.go	Do not exist	62.3%

ghost · 2019-08-07T19:48:42Z

/lgtm

mgreau · 2019-08-07T21:34:04Z

docs/tasks.md


-`Tasks` support templating using values from all [`inputs`](#inputs) and
+`Tasks` support string replcement using values from all [`inputs`](#inputs) and


typo: replcement -> replacement

tekton-robot · 2019-08-07T22:02:26Z

The following is the coverage report on pkg/.
Say /test pull-tekton-pipeline-go-coverage to re-run this coverage report

File	Old Coverage	New Coverage
pkg/reconciler/v1alpha1/pipelinerun/pipelinerun.go	Do not exist	82.4%
pkg/substitution/container_replacements.go	Do not exist	100.0%
pkg/substitution/substitution.go	Do not exist	62.3%

ghost · 2019-08-08T12:07:00Z

/lgtm

Using the path replacement makes dealing with paths much easier - you don't need to know about `/workspace` or about when the path has been overridden or any of that. I'm also doing this to set the stage for tektoncd#850 - so that when we change from `${}` syntax to `$()` we can use this test to test for both types (to make sure the change is backward compatible).

In tektoncd#850 we want to make sure to support both $() and ${} syntax for all variable substitution, including pipeline params, so I've added these values to the test so we can update them to try both syntaxes.

When changing the ${} syntax to $() in tektoncd#850 I'd like to be able to test all reasonable instances of the syntax, of which the array type is one. Turns out we didn't have any examples or end to end tests using this syntax (until now!)

bobcatfish · 2019-08-08T17:59:21Z

Rebased!

tekton-robot · 2019-08-08T18:02:00Z

The following is the coverage report on pkg/.
Say /test pull-tekton-pipeline-go-coverage to re-run this coverage report

File	Old Coverage	New Coverage
pkg/reconciler/v1alpha1/pipelinerun/pipelinerun.go	Do not exist	82.4%
pkg/substitution/container_replacements.go	Do not exist	100.0%
pkg/substitution/substitution.go	Do not exist	62.3%

In tektoncd#850 we decided that to make our syntax more similar to k8s style syntax, we will use $() for variable substitution instead of ${}. In a later iteration we may also want to make it so that anything that can be acesssed as a variable is also available as an env var to the running container, but that is TBD. In the context of this discussion we also considered removing variable substitution entirely, that conversation is continuing. We have also tried to start calling this functionality "variable substitution" instead of "templating" to make it clear that we do not intend to support more complicated templating functionality. Co-authored-by: Eli Zucker <thezuck@google.com>

@vdemeester

In order to make it clear that we are not intending to support more complicated templating (via tektoncd#850), rename the "template" related logic to "substitution". @vdemeester suggested "variable interpolation" which seems to be completely correct but I am _hoping_ that "substitution" is a bit simpler than "interpolation" (might be less correct tho!) Fixes tektoncd#850

tekton-robot · 2019-08-12T15:13:49Z

The following is the coverage report on pkg/.
Say /test pull-tekton-pipeline-go-coverage to re-run this coverage report

File	Old Coverage	New Coverage
pkg/reconciler/v1alpha1/pipelinerun/pipelinerun.go	Do not exist	82.4%
pkg/substitution/container_replacements.go	Do not exist	100.0%
pkg/substitution/substitution.go	Do not exist	62.3%

dlorenc · 2019-08-12T16:34:09Z

/lgtm

@moficodes

In tektoncd#1172 I tried to update these docs to use the new brace syntax but I only half updated this example :( And we have no tests for examples embedded in our docs so we can't catch this. Thanks for the bug report @moficodes !!!

@moficodes

In tektoncd#1172 I tried to update these docs to use the new brace syntax but I only half updated this example :( And we have no tests for examples embedded in our docs so we can't catch this. Thanks for the bug report @moficodes !!! Fixes tektoncd#1199

@moficodes

In #1172 I tried to update these docs to use the new brace syntax but I only half updated this example :( And we have no tests for examples embedded in our docs so we can't catch this. Thanks for the bug report @moficodes !!! Fixes #1199

@moficodes

In #1172 I tried to update these docs to use the new brace syntax but I only half updated this example :( And we have no tests for examples embedded in our docs so we can't catch this. Thanks for the bug report @moficodes !!! Fixes #1199 (cherry picked from commit 129a8ed)

@moficodes

In tektoncd#1199 @moficodes reported that the tutorial wasn't working which was b/c in tektoncd#1172 when I updated the braces, I only half updated the tutorial example. I wanted to fix this for the v0.6.0 docs as well, but since the released binary and yaml files don't need an update, it didn't make sense to me to create v0.6.1 for this (and leave the v0.6.0 docs broken!) so I created a 0.6 branch and cherry picked the docs fix into it only, and am now updating the README to point at that.

@moficodes

In #1199 @moficodes reported that the tutorial wasn't working which was b/c in #1172 when I updated the braces, I only half updated the tutorial example. I wanted to fix this for the v0.6.0 docs as well, but since the released binary and yaml files don't need an update, it didn't make sense to me to create v0.6.1 for this (and leave the v0.6.0 docs broken!) so I created a 0.6 branch and cherry picked the docs fix into it only, and am now updating the README to point at that.

In tektoncd#850 we decided that to make our syntax more similar to k8s style syntax, we will use $() for variable substitution instead of ${}. In a later iteration we may also want to make it so that anything that can be acesssed as a variable is also available as an env var to the running container, but that is TBD. In tektoncd#1172 we added support for the new syntax, $(), and continued support for ${}, which was released in 0.6. This commit removes support for ${}. Fixes tektoncd#1170

In #850 we decided that to make our syntax more similar to k8s style syntax, we will use $() for variable substitution instead of ${}. In a later iteration we may also want to make it so that anything that can be acesssed as a variable is also available as an env var to the running container, but that is TBD. In #1172 we added support for the new syntax, $(), and continued support for ${}, which was released in 0.6. This commit removes support for ${}. Fixes #1170

tekton-robot requested review from abayer and dlorenc August 6, 2019 23:00

tekton-robot added approved Indicates a PR has been approved by an approver from all required OWNERS files. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Aug 6, 2019

googlebot added the cla: yes Trying to make the CLA bot happy with ppl from different companies work on one commit label Aug 6, 2019

vdemeester reviewed Aug 7, 2019

View reviewed changes

ghost reviewed Aug 7, 2019

View reviewed changes

bobcatfish force-pushed the change_variable_interpolation_braces branch from 3022582 to 9fd9cec Compare August 7, 2019 19:20

tekton-robot assigned ghost Aug 7, 2019

tekton-robot added the lgtm Indicates that a PR is ready to be merged. label Aug 7, 2019

mgreau reviewed Aug 7, 2019

View reviewed changes

bobcatfish force-pushed the change_variable_interpolation_braces branch from 9fd9cec to 7608519 Compare August 7, 2019 22:00

tekton-robot removed the lgtm Indicates that a PR is ready to be merged. label Aug 7, 2019

tekton-robot added the lgtm Indicates that a PR is ready to be merged. label Aug 8, 2019

bobcatfish added 3 commits August 8, 2019 10:57

Add Pipeline params to pipeline end to end test

5a473af

In tektoncd#850 we want to make sure to support both $() and ${} syntax for all variable substitution, including pipeline params, so I've added these values to the test so we can update them to try both syntaxes.

Add example that tests array type param

a77df59

When changing the ${} syntax to $() in tektoncd#850 I'd like to be able to test all reasonable instances of the syntax, of which the array type is one. Turns out we didn't have any examples or end to end tests using this syntax (until now!)

bobcatfish force-pushed the change_variable_interpolation_braces branch from 7608519 to fcc2c3c Compare August 8, 2019 17:58

tekton-robot removed the lgtm Indicates that a PR is ready to be merged. label Aug 8, 2019

bobcatfish and others added 2 commits August 12, 2019 11:09

bobcatfish force-pushed the change_variable_interpolation_braces branch from fcc2c3c to b2469f0 Compare August 12, 2019 15:10

tekton-robot assigned dlorenc Aug 12, 2019

tekton-robot added the lgtm Indicates that a PR is ready to be merged. label Aug 12, 2019

tekton-robot merged commit cf40d0a into tektoncd:master Aug 12, 2019

bobcatfish mentioned this pull request Aug 16, 2019

Fix braces in tutorial 😭 #1206

Merged

2 tasks

bobcatfish mentioned this pull request Aug 16, 2019

Update v0.6.0 docs to include fixed tutorial 🤒 #1208

Merged

2 tasks

bobcatfish mentioned this pull request Sep 14, 2019

Remove support for ${} syntax 🗑️ #1311

Merged

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Change variable interpolation braces #1172

Change variable interpolation braces #1172

bobcatfish commented Aug 6, 2019 •

edited

Loading

tekton-robot commented Aug 6, 2019

tekton-robot commented Aug 6, 2019

vdemeester left a comment

vdemeester Aug 7, 2019

ghost Aug 7, 2019

bobcatfish Aug 7, 2019

ghost Aug 7, 2019

bobcatfish Aug 7, 2019

bobcatfish Aug 7, 2019

ghost Aug 7, 2019

bobcatfish Aug 7, 2019

ghost commented Aug 7, 2019

bobcatfish commented Aug 7, 2019

bobcatfish commented Aug 7, 2019

bobcatfish commented Aug 7, 2019

tekton-robot commented Aug 7, 2019

ghost commented Aug 7, 2019

mgreau Aug 7, 2019

bobcatfish Aug 7, 2019

tekton-robot commented Aug 7, 2019

ghost commented Aug 8, 2019

bobcatfish commented Aug 8, 2019

tekton-robot commented Aug 8, 2019

tekton-robot commented Aug 12, 2019

dlorenc commented Aug 12, 2019


		`Tasks` support templating using values from all [`inputs`](#inputs) and
		`Tasks` support string replcement using values from all [`inputs`](#inputs) and

Change variable interpolation braces #1172

Change variable interpolation braces #1172

Conversation

bobcatfish commented Aug 6, 2019 • edited Loading

Changes

Submitter Checklist

Release Notes

tekton-robot commented Aug 6, 2019

tekton-robot commented Aug 6, 2019

vdemeester left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ghost commented Aug 7, 2019

bobcatfish commented Aug 7, 2019

bobcatfish commented Aug 7, 2019

bobcatfish commented Aug 7, 2019

tekton-robot commented Aug 7, 2019

ghost commented Aug 7, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tekton-robot commented Aug 7, 2019

ghost commented Aug 8, 2019

bobcatfish commented Aug 8, 2019

tekton-robot commented Aug 8, 2019

tekton-robot commented Aug 12, 2019

dlorenc commented Aug 12, 2019

bobcatfish commented Aug 6, 2019 •

edited

Loading