Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

html/template: document ability to modify FuncMap after template parse by calling Funcs again #34680

Open
michaelsafyan opened this issue Oct 3, 2019 · 20 comments

Comments

@michaelsafyan
Copy link

@michaelsafyan michaelsafyan commented Oct 3, 2019

Background

I have a bunch of templates that have a complex dependency structure like:

 A -> B, D, E
 B ->  E, F
 C -> E, G
 D -> F, H
 E -> H
 F -> nil
 G -> nil
 H -> nil

The contents of the corresponding files ('A.tmpl', 'B.tmpl', etc.) are static; however, the template.FuncMap that they reference contain implicit reference to captured state that varies by caller. That is, the FuncMap is created like so:

func createFuncMap(request *rpc.Request) {
   result := template.FuncMap {
     "Foo": createFooFunc(request),
     "Bar": createBarFunc(request),
      ...
   }
  return request
}

What I Want to Do

I'd like to be able to preparse all of the various files irrespective of the func map (or perhaps with a whitelist of functions that will be defined), and I'd like to parse the individual pieces separately (e.g. parse 'F' only once, even though it is a dependency of 'B' and 'D'), and then somehow combine the parsed templates together like:

   func getTemplate(name string) *template.Template {
     result := template.New(name).Funcs(getStubFunctionMap())
     for _, dep := range getDeps(name) {
        result.AddDep(getTemplate(dep))
      }
      result.Parse(getFileContent(name))
      return result
    }

I'd also like to trivially rebind the function map as in:

   func getBoundTemplate(name string, request *rpc.Request) *template.Template {
     cachedTemplate := getOrCreateCachedTemplate(name)
     boundTemplate := cachedTemplate.Clone().Funcs(funcMapFor(request))
     return boundTemplate
   }

Why I Can't Do It

The documentation of html/template makes it appear that parsing is context dependent and that parsing is linear (dependencies can be added linearly, but cannot be combined from multiple, already-parsed templates). This is a pretty major limitation of the interface.

Impact of This Limitation

This is a major efficiency problem; I want to parse a complex network of templates upfront before receiving requests and do very minimal work on each request. This structure is causing much more work to happen in the context of every request.

For more specifics of this use case, reach out to me directly from your @google.com account, and I can share more details of the specific code in question.

@ianlancetaylor ianlancetaylor changed the title Feature request: ability to combine/clone/rebind parsed html/template templates html/template: feature request: ability to combine/clone/rebind parsed html/template templates Oct 4, 2019
@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Oct 4, 2019

@andybons andybons changed the title html/template: feature request: ability to combine/clone/rebind parsed html/template templates proposal: html/template: add ability to combine/clone/rebind parsed html/template templates Oct 4, 2019
@gopherbot gopherbot added this to the Proposal milestone Oct 4, 2019
@gopherbot gopherbot added the Proposal label Oct 4, 2019
@empijei

This comment has been minimized.

Copy link
Contributor

@empijei empijei commented Oct 16, 2019

I think I need a little bit more context.

Why do you need to closure some state in the funcs of your templates? Why can't you pass an additional map of data to process?

To use your example: "Foo" func would accept an additional parameter that depends on the request being processed, but its code would stay the same.

What kind of request-scoped data are you currently capturing in your functions that can't be a parameter?

@michaelsafyan

This comment has been minimized.

Copy link
Author

@michaelsafyan michaelsafyan commented Oct 16, 2019

@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Oct 16, 2019

Something like proposal #31107 would let you pass data into the functions via a context.Context. Would that be another way for you to pass the data you need into the functions?

Can you explain why you want the ability to bind together parsed templates? What would the API in html/template look like?

It sounds like you can do what you need by passing values through the template. You say that making the input more explicit introduces needless cognitive burden to template authors. But Go in general does prefer to make things more explicit.

@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Oct 16, 2019

A simple self-contained concrete example might help clarify this. The description is somewhat abstract.

@michaelsafyan

This comment has been minimized.

Copy link
Author

@michaelsafyan michaelsafyan commented Oct 17, 2019

I've just sent a private email from my @google.com account to Ian's @google.com account with more detail concerning the original motivation.

@rsc

This comment has been minimized.

Copy link
Contributor

@rsc rsc commented Oct 21, 2019

@michaelsafyan It would be nice if you could say something publicly. We don't make decisions based on unpublished Google-internal motivations. Well, except "no, because it's too Google-specific".

@rsc

This comment has been minimized.

Copy link
Contributor

@rsc rsc commented Oct 21, 2019

The original comment says:

The documentation of html/template makes it appear that parsing is context dependent and that parsing is linear (dependencies can be added linearly, but cannot be combined from multiple, already-parsed templates). This is a pretty major limitation of the interface.

Yes, parsing is context dependent, so it is not really possible to reuse pieces the way you want. That is fundamental to the security analysis. I don't think what you are trying to do is sound with respect to the security analysis. We won't make unsound changes here.

@michaelsafyan

This comment has been minimized.

Copy link
Author

@michaelsafyan michaelsafyan commented Oct 23, 2019

@rsc

This comment has been minimized.

Copy link
Contributor

@rsc rsc commented Oct 30, 2019

The html/template docs give as a simple example <a href="/search?q={{.}}">{{.}}</a>. Those two substitutions need different escaping. In general those could be call-outs to other templates. So fundamentally the analysis of the template is context-dependent.

The original proposal here sounds to me like "don't make template analysis context-dependent". That's not possible (see example).

Is there a more limited proposal you were making, and I've misunderstood?

@michaelsafyan

This comment has been minimized.

Copy link
Author

@michaelsafyan michaelsafyan commented Nov 5, 2019

The html/template docs give as a simple example <a href="/search?q={{.}}">{{.}}</a>. Those two substitutions need different escaping. In general those could be call-outs to other templates. So fundamentally the analysis of the template is context-dependent.

I agree and am not suggesting that this change.

The original proposal here sounds to me like "don't make template analysis context-dependent".

My apologies for being unclear. No, I'm not asking that parsing cease being context-dependent. I'm asking that parsing not make a distinction regarding whether parameters come from the global scope (via the FuncMap) vs from the local scope. There is currently this inconsistency where FuncMap parameters and parameters that are passed into the template get treated differently insofar as non-FuncMap parameters can be changed while keeping the template the same, while FuncMap parameters cannot be changed after parsing.

What I'd like is for it to be possible to replace the FuncMap (the values, not the keys) on each invocation of the template, without the FuncMap being a shared resource that needs to get locked, so that different requests / instantiations of the template can use different values of the FuncMap parameters. This would make member functions (of whatever is supplied as data interface{}) and global functions (supplied through FuncMap) roughly interchangeable.

@rsc rsc changed the title proposal: html/template: add ability to combine/clone/rebind parsed html/template templates proposal: html/template: add ability to modify FuncMap after template parse Nov 6, 2019
@rsc

This comment has been minimized.

Copy link
Contributor

@rsc rsc commented Nov 6, 2019

I tried to retitle this issue to indicate my current understanding of the suggestion:
"add ability to modify FuncMap after template parse".

The original message says:

I'd also like to trivially rebind the function map as in:

  func getBoundTemplate(name string, request *rpc.Request) *template.Template {
    cachedTemplate := getOrCreateCachedTemplate(name)
    boundTemplate := cachedTemplate.Clone().Funcs(funcMapFor(request))
    return boundTemplate
   }

Now I don't understand "why I can't do it". As far as I can see that code should work fine. The "why I can't do it" talks about parsing, but there's no parsing in calling Clone and Funcs.

I'm sorry, but I'm still confused.

@michaelsafyan

This comment has been minimized.

Copy link
Author

@michaelsafyan michaelsafyan commented Nov 10, 2019

Per the documentation of func (t *Template) Funcs(funcMap FuncMap) *Template:
https://golang.org/pkg/html/template/#Template.Funcs

Funcs adds the elements of the argument map to the template's function map. It must be called before the template is parsed. It panics if a value in the map is not a function with appropriate return type. However, it is legal to overwrite elements of the map. The return value is the template, so calls can be chained.

In other words, one cannot call Funcs a second time after calling Clone. One can modify the existing map passed into Funcs before, but one cannot pass in a completely different map (which is important for thread-safety).

I'd like this to be relaxed; one should be able to call Funcs again with a new map, as long as the new map has all of the same keys as the previous one.

(And, yes, I am basing this on more than just documentation, but also trying it; when I tried the code above, the result was an error).

@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Nov 12, 2019

Out of curiousity:

Why do you want to call Funcs a second time after calling Clone? If you do, why not just call Clone again?

Or, could you write the functions as redirectors, so that they test whatever condition you would use to call Funcs again to decide themselves which function to call?

@michaelsafyan

This comment has been minimized.

Copy link
Author

@michaelsafyan michaelsafyan commented Nov 13, 2019

Or, could you write the functions as redirectors, so that they test whatever condition you would use to call Funcs again to decide themselves which function to call?

Doing this would require the introduction of locking, which is what I'm trying to avoid.

To clarify:

Startup / shared thread:
- (Re)load template text files
- (Re)parse text files into templates (using dummy FuncMap)

(In parallel): request1 thread A:
- Clone (shared) parsed template, creating thread-local copy
- Set copy to use request1's function map

(In parallel): request2 thread B:
- Clone (shared) parsed template, creating thread-local copy
- Set copy to use request2's function map

[Also thread C, D, etc. up to N request-serving threads]

The same semantics can be achieved with a function that is a "redirector", but then this requires the function to reference an actual thread local variable that it resolves on each invocation. It should be possible to simply allow each clone to reference a separate, independent map.

@kaey

This comment has been minimized.

Copy link

@kaey kaey commented Nov 13, 2019

It should be possible to simply allow each clone to reference a separate, independent map.

This works today and we're using it heavily.
What error are you seeing?

@rsc

This comment has been minimized.

Copy link
Contributor

@rsc rsc commented Nov 13, 2019

Per the documentation of func (t *Template) Funcs(funcMap FuncMap) *Template:
https://golang.org/pkg/html/template/#Template.Funcs

Funcs adds the elements of the argument map to the template's function map. It must be called before the template is parsed. It panics if a value in the map is not a function with appropriate return type. However, it is legal to overwrite elements of the map. The return value is the template, so calls can be chained.

In other words, one cannot call Funcs a second time after calling Clone. One can modify the existing map passed into Funcs before, but one cannot pass in a completely different map (which is important for thread-safety).

I think you are misreading the documentation. There are two maps in play.

Funcs copies entries from the map argument into a private map stored in the template.
If the caller modifies Funcs's map argument after the call returns, nothing changes in the template.
If you Clone the template, the clone gets its own private map initialized to be a copy of the original template (being cloned).
If you call Funcs again, more entries are added to the private map.

Have you tried to use Funcs after Clone and had it fail? @kaey says it works for them, for example, and I believe I've done this too. Can you provide a failing test case if it doesn't work for you?

Thanks.

@rsc

This comment has been minimized.

Copy link
Contributor

@rsc rsc commented Nov 20, 2019

@michaelsafyan, any response to my comment from last week? The code you are trying to write should work already. If not it would help to provide a simple test case. Thanks!

@michaelsafyan

This comment has been minimized.

Copy link
Author

@michaelsafyan michaelsafyan commented Nov 21, 2019

Sorry, I've not had an opportunity to confirm that it works. But, assuming that it works, this may then simply be a request to clarify/update the documentation than an implementation change.

@rsc rsc added this to Active in Proposals Nov 27, 2019
@rsc

This comment has been minimized.

Copy link
Contributor

@rsc rsc commented Nov 27, 2019

Will repurpose this issue for updating docs that Funcs can be called multiple times to replace earlier mappings.

@rsc rsc changed the title proposal: html/template: add ability to modify FuncMap after template parse html/template: document ability to modify FuncMap after template parse by calling Funcs again Nov 27, 2019
@rsc rsc removed the Proposal label Nov 27, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
6 participants
You can’t perform that action at this time.