proposal: text/template: ability to resolve values from a dataset using templates

[pd93]

Proposal Details

Background

Given a template string and a dataset, the text/template package allows a user to write data from the dataset as a formatted string to an io.Writer. However, currently, it is not possible to use a template to resolve and return a value and preserve its type from a dataset.

For example, suppose we set up a template and dataset as follows:

import "text/template"

...

// Define a template
tplStr := `{{index .MySlice 3}}`

// Define a dataset
data := map[string]interface{}{
    "MySlice": []int{1, 2, 3, 4, 5},
}

// Create the template
tpl, _ := template.New("").Parse(tplStr)

We are now able to execute the template with our dataset by calling the Execute method. This will write the formatted string to the given io.Writer.

buf := &bytes.Buffer{}
tpl.Execute(buf, data)
fmt.Println(buf.String()) // Prints "4"

However, the output of the template is always a string and not the actual value from the dataset. There are a variety of use cases where it would be useful to fetch and mutate data while preserving types from a dataset using templating syntax - particularly when a template is given at runtime in software that surfaces Go's templating syntax.

Proposal

Add a new method to the Template type in the text/template package that allows the user to return a variable with its underlying type given a template and dataset. More precisely, a method with the following (or similar) signature:

func (t *Template) Resolve(data any) (val any, err error)

Using the same template and dataset example as before, we can now resolve the value from the dataset and preserve its type:

v, _ := tpl.Resolve(data) // v is of type any (int)
fmt.Println(v) // Prints 4

More interestingly, this allows us to do things like this:

import (
    "text/template"
    "github.com/Masterminds/sprig/v3"
)

...

// Use sprig's len function to get the length of a slice and return that instead
tpl, _ := template.New("").Funcs(sprig.FuncMap()).Parse(`{{len .MySlice}}`)
v, _ := tpl.Resolve(data)
fmt.Println(v) // Prints 5

It may be tempting to ask why we need templates to do this instead of doing something simple like this:

fmt.Println(len(data["MySlice"])) // Prints 5

However, as I mentioned in the background, in cases where a template is being supplied at runtime by a user of the software, we need to use the templating engine to resolve the value. The case study below describes a real-world example of this.

Since this only really makes sense with a single ActionNode, calling Resolve on a template with multiple nodes or a non-ActionNode should return an error.

Case Study

I am one of the maintainers of Task, which is a popular task runner and build tool written in Go. One of the useful features of Task is the ability to use Go's templating engine to template tasks. Until recently, all variables in Task were strings, so using one variable inside another variable was as simple as referencing it in a Go template:

tasks:
  default:
    vars:
      FOO: "foo"
      BAR: "{{.FOO}} bar"
    cmds:
      - echo "{{.BAR}}" # prints "foo bar"

However, we recently added made our "Any Variables" experiment generally available. This allows users to define variables of "any" type (i.e. string, int slice etc.). This is great because users now have much more flexibility in how they define/process their task data. However, passing these variables from one task to another was only possible using templating and this caused variables to be stringified.

To resolve this, we added a new ref keyword which will allow users to directly reference a variable and maintain its type:

tasks:
  task-1:
    vars:
      FOO: [1, 2, 3, 4, 5]
    cmds:
      - task: task-2
        vars:
          FOO:
            ref: .FOO # <-- Sends a reference to the variable FOO

  task-2:
    cmds:
      - echo "{{index .FOO 3}}" # prints "4" because we can still use FOO as a slice

We decided that we wanted to keep using Go's templating engine to do the reference resolving. This has a couple of advantages:

1. It keeps the syntax familiar.
2. It allows users to use Go's templating syntax/functions/pipes to manipulate the data as it is passed.

Since this wasn't possible to do using the public API of the standard library's text/template package, we have forked the package and implemented the proposed API. The two additional methods can be viewed below. This is a provisional implementation specifically for Task. However, it is in our latest release and working well for us so far.

• Template.Resolve - https://github.com/go-task/template/blob/960e6f576656d7e42a83a98307241c86a660f357/exec.go#L234-L253
• state.actionValue - https://github.com/go-task/template/blob/960e6f576656d7e42a83a98307241c86a660f357/exec.go#L286-L301

Since there aren't often significant changes to this package, we are happy to maintain this fork. However, we feel like this change would be a useful addition to the standard library and could benefit other users too.

[ianlancetaylor]

CC @robpike

[robpike]

Templates are for formatting data into text. They are basically printf on steroids. I do not think that should change.

In any case I think this would be both rarely used and would require a lot of preconditions because a typical template processes many values. Yes, you have presented a case where it makes sense, but I still feel it is an unusual situation not worth changing the basics of the template package.

[pd93]

it is an unusual situation not worth changing the basics of the template package

While I generally agree that the specific use-case above is unusual and beyond the scope of the regular usage of the template package, there is simply no way to do what we are trying to do with the current public API because template.state is not exposed.

Go's templating syntax is extremely powerful, and it seems to be a shame to restrict it to outputting strings when it could also be used for parsing/manipulating data at runtime with a relatively small, non-breaking change. This would open up a lot of possibilities for software that wants to expose the templating syntax to its users.

Templates are for formatting data into text. They are basically printf on steroids. I do not think that should change.

If we want to make the distinction between a template and a "resolver", we could make a separate Resolver struct which would be constructed with a single ActionNode and return a value when a Resolve() method is called. This would also solve the concern you raised around templates having many values.

This would still need to be contained within the text/template package though as it needs access to the state to do the value resolution.

[earthboundkid]

I have wanted this before, but the reason I wanted it was that I wanted to use Go templates as the engine for my own new templating language. I think maybe exposing the ability to do t.Eval("add (len .) 1", mySlice) (no {{ }}!) and get an int would be useful for people who want to use Go templates as basically a PHP-like dynamic scripting language engine within Go, but it's also a big API commitment from the Go team for unclear benefits.

Could you just fork the template library and expose the internals in your fork?

[rsc]

This proposal has been declined as infeasible.
— rsc for the proposal review group