-
Notifications
You must be signed in to change notification settings - Fork 114
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
support partials #207
support partials #207
Conversation
Hey @skaes ! I discussed this feature with my team, and we decided it is something we'd like to accept. In addition to the use cases you've shown in your tests, I think it would be great to be able to embed a common pod spec into multiple deployments. Before I look at the details, I'd like to request that the rendering functionality be extracted into its own class now that the logic required is growing. This is also going to be necessary in #228, so please coordinate with @maxlaverse (who isn't at Shopify either--it's great to have all these contributions! ❤️ ) about who will have time to do that piece first. |
@KnVerey Maxime and myself are working both at XING. He's in my team. We're using kubernetes-deploy in building an internal PaaS. Which explains the many pull requests you've seen. |
@KnVerey also, thrilled you like the idea. Will refactor tomorrow. |
Can we make some progress on this pull request? I would like complete this one, before integrating #228. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't think this currently handles indentation correctly. If you can get that working with the current syntax, great! Otherwise something like partial 'my_partial', locals: { var1: 'foo' }, indent: 5
would work for me too.
Please leave a comment saying that the code is ready for re-review once it's updated. 😄
README.md
Outdated
|
||
`kubernetes-deploy` supports composing templates from so called partials in order to reduce duplication in Kubernetes YAML files. Given a template directory `DIR`, partials are searched for in `DIR/partials`and in 'DIR/../partials', in that order. They can be embedded in other ERB templates using the helper method `partial`. For example, let's assume an application needs a number of different CronJob resources, on could place a template called `cron`in one of those directories and then use it in the main deployment.yaml.erb like so: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
typo: "on could place" -> "one could place"
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
fixed
README.md
Outdated
<%= partial "cron", name: "send-mail", schedule: "0 0 * * *", args: %w(send-mails), cpu: "200m", memory: "256Mi" %> | ||
``` | ||
|
||
Inside a partial, the parameters can be accesses as normal variables, or via a hash called `locals`. Thus, the `cron` template could like this: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
typo: "can be accesses" -> "can be accessed"
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
fixed
README.md
Outdated
containers: | ||
- name: cron-<%= name %> | ||
image: ... | ||
args: [ <%= args.map{|a| %Q("#{a}")}.join(", ") %> ] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I know this is just an example, but why all the transformations? This seems to generate [""cleanup""]
, vs. ["cleanup"]
from just <%= args %>
. Why is that desirable?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It was extracted from a more complex example, where part of the args entry was already present.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
changed
README.md
Outdated
restartPolicy: OnFailure | ||
``` | ||
|
||
Note that partial names must have the suffix `.yaml.erb`. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We should support .yml.erb
too... we originally supported only one in the main renderer, and it led to some confusing debugging for a few users.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
done
lib/kubernetes-deploy/deploy_task.rb
Outdated
rescue NameError => e | ||
@logger.summary.add_paragraph("Error from renderer:\n #{e.message.tr("\n", ' ')}") | ||
raise FatalDeploymentError, "Template '#{filename}' cannot be rendered" | ||
@renderer.render_template(filename, raw_template) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Let's inline this since render_template
is only called once.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
fine with me
bindings: { "cronjob_api_version" => api_version }, | ||
sha: nil | ||
) | ||
assert_deploy_success(result) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We should also include some assertions that expected resources were created. With malformed yaml documents (not saying your new ones are malformed now, but we could mess them up in the future), it is possible to have a successful deploy that simply never tried to render some of the expected templates.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
done
@@ -0,0 +1,20 @@ | |||
--- | |||
apiVersion: extensions/v1beta1 | |||
kind: Deployment |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'd also like to see a fixture that proves that your renderer correctly handles indentation. A common request I've heard is to be able to embed a full podspec (i.e. spec.template
) from a partial into multiple deployment files. Let's do that with this deployment; otherwise, it isn't really contributing much to the test itself.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The renderer does not handle indentation. If you have an idea how to implement this feature without rewriting or monkey patching ERB, please let me know.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It would be really nice to come up with a solution for this. I will think about how to do this in a nice way for a bit as well.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
When I implemented the first version of the code, I already thought about this problem. I couldn't think of any other way to support indenting other than adding an explicit indentation parameter to the partial method, just as @KnVerey suggested, or writing a renderer. I rejected the first idea as being to error prone and clumsy and rejected the second idea as way too costly and not backwards compatible.
Besides, you can always write partials using YAML flow style, which does not require indentation:
{
a: 1,
b: {
x: 1
}
}
lib/kubernetes-deploy/renderer.rb
Outdated
erb_binding = binding | ||
bind_template_variables(erb_binding) | ||
erb_binding.eval <<~EVA, __FILE__, __LINE__ + 1 | ||
def partial(partial, locals = {}) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Since our base binding comes from this class, I believe this method can be defined like any other--no need to eval
it in or to use self.
in calling methods within it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I wanted to keep the exposed interface available inside ERB as small as possible. The eval essentially reduces it to the method "partial" and "self", whereas your proposed implementation allows direct access to all instance variables, public and private methods of class Renderer.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Of course, anything can be accessed via self as well, but it requires a conscious effort to do so.
lib/kubernetes-deploy/renderer.rb
Outdated
def partial(partial, locals = {}) | ||
partial_binding = binding | ||
self.bind_template_variables(partial_binding) | ||
self.bind_template_variables(partial_binding, locals) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: I'd remove the default from the second arg and call it just once with template_variables.merge(locals)
so it's more clear what is going on.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
changed
lib/kubernetes-deploy/renderer.rb
Outdated
@id = current_sha[0...8] + "-#{SecureRandom.hex(4)}" if current_sha | ||
end | ||
|
||
def template_variables |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please make this method, bind_template_variables
and find_partial
private
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't think that improves anything.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It would make us happy :)
README.md
Outdated
@@ -120,6 +120,44 @@ All templates must be YAML formatted. You can also use ERB. The following local | |||
|
|||
You can add additional variables using the `--bindings=BINDINGS` option. For example, `kubernetes-deploy my-app cluster1 --bindings=color=blue,size=large` will expose `color` and `size` in your templates. | |||
|
|||
#### Using partials | |||
|
|||
`kubernetes-deploy` supports composing templates from so called partials in order to reduce duplication in Kubernetes YAML files. Given a template directory `DIR`, partials are searched for in `DIR/partials`and in 'DIR/../partials', in that order. They can be embedded in other ERB templates using the helper method `partial`. For example, let's assume an application needs a number of different CronJob resources, one could place a template called `cron`in one of those directories and then use it in the main deployment.yaml.erb like so: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Missing space: s/called cron
in/called cron
in/
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
fixed
end | ||
|
||
def find_partial(name) | ||
partial_names = [name + '.yaml.erb', name + '.yml.erb'] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think the function could be implemented nicer with Dir.glob. Something like:
Dir.glob("**/#{name}.y[a]ml")` This could would then also have an explicit way of choosing one file over another if there are two files with the same name but in different directories, instead of now implicitly using the first one it comes across.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think you might have missed that the README specifies a search order for the directories it's looking in, in order to support scoping.
I would consider the existence of two partials which differ only in the suffix a user error. We could of course check for that condition (within one directory).
BTW: I don't think niceness is objectively measurable.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I read the readme and that's why I said that the code just implicitly reflects the that. And also wasn't talking about the suffix, but a partial in DIR/partials
and DIR/../partials
.
More importantly: Dir.glob
removes 2 each
s, a File.join
, the need to do a File.exists
, as well as partial_names
, so basically the whole function. And would get rid of @partials_dirs
. I think that's a cleaner way of doing it, which is why I suggested it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@klautcomputing how about you show me the implementation you want and I just copy and paste it? I really don't have time to waste on minor details.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I changed the code, please find my PR here xing#1
@@ -0,0 +1,20 @@ | |||
--- | |||
apiVersion: extensions/v1beta1 | |||
kind: Deployment |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It would be really nice to come up with a solution for this. I will think about how to do this in a nice way for a bit as well.
@KnVerey @klautcomputing ready for further review. |
In my and @klautcomputing opinion, the ability to correctly handle indentation is an essential requirement for this feature. As I mentioned in my original comment in response to this PR, that is commonly the use case I've heard for requesting partials internally. IMO it would be considered a major bug if it did not work. You're right that this is not trivial to research and implement, and I don't know how to do it offhand either. One thing I'd definitely look into is the implementation in Rails. If you do not have the time to pursue this further, one of us will implement it branched off your work when we have the time, and then merge both pieces together. Either way, thank you for your work so far. |
@KnVerey @klautcomputing We have found a way to allow partials authors to use normal YAML notation when including YAML partials at arbitrary offsets from column zero. Since JSON is a subset of YAML, we simply convert the partial to a JSON string before including it in the outer template. I sincerely hope this satisfies your requirements now. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hey @skaes. I only did a partial review, since I noticed that your test doesn't actually cover your solution (I put in a debugger and it never got hit), and if I add a template that does (e.g. a partial for a pod spec or an environment section, as I mentioned before), the rendering doesn't seem to work. Please modify your fixtures to prove your solution works and re-ping if I'm wrong 😄 .
lib/kubernetes-deploy/renderer.rb
Outdated
}.merge(@bindings) | ||
end | ||
|
||
def bind_template_variables(binding, variables) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please rename the arg to erb_binding
or something to avoid confusion with the binding
method (this confusion is aggravated by syntax highlighting because of the special status of that method -- see L30).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Note that fact that editors highlight the variable name 'binding' is a mistake on behalf of syntax highlighters. There is nothing special about the name 'binding'. But I get your point, so I changed it.
lib/kubernetes-deploy/renderer.rb
Outdated
def render_template(filename, raw_template) | ||
return raw_template unless File.extname(filename) == ".erb" | ||
|
||
binding = TemplateContext.new(self).template_binding |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Same comment for this variable--please name it something other than binding
to avoid confusion
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done.
lib/kubernetes-deploy/renderer.rb
Outdated
bind_template_variables(binding, template_variables) | ||
|
||
src = ERB.new(raw_template).result(binding) | ||
if src =~ /^--- *\n/m |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It isn't safe to assume that multi-doc templates (or single-doc templates that have a header) do not contain partials.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please note that src is actually YAML code that already went though the ERB step and therefore partials inside it have been expanded. Apparently the choice of variable name was unfortunate, I'll rename it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ouch. I added the code to the wrong method. Will fix.
@@ -802,6 +802,23 @@ def test_ejson_secrets_respects_no_prune_flag | |||
ejson_cloud.assert_secret_present('ejson-keys', managed: false) | |||
end | |||
|
|||
def test_partials | |||
result = deploy_dir_without_profiling( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Our original discussion got lost, but I see why you can't use deploy_fixtures
here. The best helper for this case would be deploy_raw_fixtures("test-partials", bindings: { 'supports_partials' => 'true' })
. Incidentally, we should make a partial reference custom bindings (e.g. "supports_partials" in that command) to prove that works.
We also need another test about what happens when a referenced partial is not found (you can add at template to the 'invalid' fixture set for that purpose).
@KnVerey I think I have addressed all comments and known issues. I added a unit test class for testing the partials implementation, which should make it a lot simpler adding new features later on. I have explained the limitations of the partials system in the README and changed the partials integration test to show how a pod spec can be included via a partial. Please let me know if I have missed to address something. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This implementation in the current form works and I am tentatively approving it. It would be nice if the nit pick comments would be addressed and then we can merge this. Please rebase and fixup your commits into one after @KnVerey had a change to review and after addressing her comments, too.
Thank you for implementing this!
lib/kubernetes-deploy/renderer.rb
Outdated
@id = current_sha[0...8] + "-#{SecureRandom.hex(4)}" if current_sha | ||
end | ||
|
||
def template_variables |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It would make us happy :)
lib/kubernetes-deploy/renderer.rb
Outdated
|
||
template = @_renderer.find_partial(partial) | ||
expanded_template = ERB.new(template, nil, '-').result(erb_binding) | ||
if expanded_template =~ /^--- *\n/m |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Could you refactor this to return expanded_template if expanded_template =~ /^--- *\n/m
and thereby get rid of the else. Thanks
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done.
b: 2 | ||
``` | ||
|
||
This is a limitation of the current implementation. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for explaining this in the readme.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yw
@@ -0,0 +1,7 @@ | |||
value: <%= n %> |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
👍 test
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah. I like it too. :-)
@klautcomputing when you say rebase and merge into one commit, do you really want me to force push this onto this pull request? That would make it hard for us to merge this work with our own master and also impossible for anyone to follow the discussion on the pull request, won't it? |
can you please advise me how to fix the code style violations in the README? |
Rebase and fixup is how I usually do it, especially when there are so many commits. So, just one commit that has a meaningful message about what was implemented. But if that makes things difficult for you I leave it up to you and @KnVerey |
Sorry, I misread the Policial messages. It complains about strings not being internationalized in the renderer unit test. Nevertheless, I really don't know how to fix this. Please advise. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for the JSON solution and new tests! To make sure my comments are clear (and to make sure what I'm asking for works), I pushed a branch with the changes I have in mind here: d0be76f...partial-review.
The Policial failure was in fact a bug. I re-ran it on your commit after the bug was fixed, and it passed.
Re: squashing commits, our standard process at Shopify is to continue pushing new commits while review is ongoing, but rebase and squash before merging, especially if there are more than a few commits. The idea is for master to contain meaningful commits. Github does not lose the conversation or lines of code it was made on even after the squash. Why would squashing make it harder to pull this work into your own master than it would be to pull in any other changes we make upstream?
README.md
Outdated
|
||
Inside a partial, parameters can be accessed as normal variables, or via a hash called `locals`. Thus, the `cron` template could like this: | ||
|
||
```erb |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It looks like "erb" means "html/erb" to Github, which explains why nothing is getting syntax highlighted here. yaml
would be more useful (same comment about the other code blocks below).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Was this forgotten/dismissed?
README.md
Outdated
restartPolicy: OnFailure | ||
``` | ||
|
||
Supported file names for a given partial `p` are `p.yaml.erb` or `p.yml.erb`, where the first form is preferred. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This makes it sound like you can name the actual file "p" and possibly use partial: 'p.yaml.erb'
. How about something like:
Both
.yaml.erb
and.yml.erb
file extensions are supported. Templates must refer to the bare filename (e.g. usepartial: 'cron'
to referencecron.yaml.erb
)
lib/kubernetes-deploy/renderer.rb
Outdated
bind_template_variables(erb_binding, template_variables) | ||
|
||
ERB.new(raw_template).result(erb_binding) | ||
rescue StandardError => e |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This used to be NameError
, which is the error you get when a template references something undefined. We generally consider it a best practice to rescue specifically the errors we are expecting; in this case, we want to make NameError
pretty (by turning it into FatalDeploymentError
) because they're the end user's fault, but for errors that are gem problems that should be fixed, getting a stack trace would be more helpful. Are there additional errors you encountered in working on this?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Well, erb templates contain arbitrary ruby code, which can raise arbitrary exceptions. I don't see why NameError should be treated special here. Either rescue all non-fatal exceptions, or none at all, would be my strategy. But I'm happy to change it back to NameError.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That's a fair point. I'm fine with StandardError
.
lib/kubernetes-deploy/renderer.rb
Outdated
begin | ||
JSON.generate(YAML.load(expanded_template)) | ||
rescue Psych::SyntaxError => e | ||
raise "#{e.class}#{e}. Partial did not expand to valid YAML, source: #{expanded_template}" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
To get this formatted nicely for the user, we should do:
@logger.summary.add_paragraph("Rendered template content:\n#{expanded_template}")
raise FatalDeploymentError, "Partial #{partial} is not valid YAML"
(I realize the logger bit isn't possible inside this class; see my other request)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Refactored the class and changed.
binding | ||
end | ||
|
||
def partial(partial, locals = {}) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In my opinion, this method should simply delegate to a render_partial
method on the renderer that itself makes use of the existing render
method. This has several advantages:
- The ability to both keep the Renderer's public interface clean and prevent this private class from reaching into its internals (both
send
s and making methods public for the sake of an implementation detail are smells IMO). - The ability to share the rendering logic and error handling between partials and regular templates.
- The ability to use the
logger
to augment errors (see my other comment) - Clarity as to whether, given
template -> partial1 -> partial2
, variables set within partial1 will be available in partial2 (they shouldn't be, unless exposed purposefully throughlocals
, and I think they would be right now)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I agree.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Variables are not inherited from an embedding partial in the current implementation. The only variables available in all templates are the ones passed in via extra bindings plus current_sha
and deployment_id
. I would consider it an extreme nuisance to not have those available automatically everywhere.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I 100% agree that the extra bindings and standard variables should be available in all templates. You're right that local variables are not inherited, but instance variables are. Here's how I tested that on your branch:
# deployment.yaml.erb
<% @test = "hello" %>
<%= partial "pod", name: "pod1", args: ["echo", "foo"] %>
# pod.yaml.erb
...
args: <%= @test %>
Results in:
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, instance variables are available. Just like in Rails. In my view, this is a feature.
lib/kubernetes-deploy/renderer.rb
Outdated
|
||
# If we're at top level we don't need to worry about the result being | ||
# included in another partial. | ||
return expanded_template if expanded_template =~ /^--- *\n/m |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm always hesitant to use regexes to parse language tokens. For example, this version would miss document separator lines:
- that are followed by tabs
- that end with line breaks other than
\n
- that include comments
- probably other conditions that I'm not thinking of
The worst consequence of failing to return here would be that YAML.load
would silently drop documents if there are more than one, which is pretty bad. Imagine you've deployed your resources using partials, then someone accidentally commits a tab (or whatever thing it doesn't occur to us to handle), and this causes the deploy to prune some of the resources!
Incidentally, we should have a test for the case where a partial contains multiple documents.
The most pragmatic solution I could come up with is to parse the partial as a stream, return it verbatim if there are multiple documents, and always turn it into JSON if there's just one document. This has the limitation of requiring document separators to be placed in parent templates, but that seems more reasonable than risking resource loss. If you or @klautcomputing have a better idea, I'm all for it!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The change you are proposing would invalidate templates in dozens of our projects which have been happily using the originally proposed implementation of partials system without any complaints for two months now. I do not approve of it.
My suggestion is to continue using a regexp, but protect against potential document loss by using YAML.load_stream to detect whether we missed detecting multiple documents.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There's already a test which has a partial containing multiple documents. See partial 'stream.yaml.erb' in test_can_render_template_with_correct_indentation.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think I've found a solution that will work for everyone:
docs = YAML.parse_stream(expanded_template)
# If the partial contains multiple documents or has an explicit document header,
# we know it cannot validly be indented in the parent, so return it immediately.
return expanded_template unless docs.children.one? && docs.children.first.implicit
# Make sure indentation isn't a problem by producing a single line of parseable YAML.
# Note that JSON is a subset of YAML.
JSON.generate(docs.children.first.to_ruby)
Here's evidence that docs.children.first.implicit
is detecting exactly what we want:
[7] pry(main)> docs = {
[7] pry(main)* explicit_doc: YAML.parse_stream("---\nfoo: bar"),
[7] pry(main)* implicit_doc: YAML.parse_stream("foo: bar"),
[7] pry(main)* multiple_docs: YAML.parse_stream("foo: bar\n---\nbar: baz")
[7] pry(main)* }
[9] pry(main)> docs.each do |type, value|
[9] pry(main)* puts "#{type} -> Multiple docs: #{!value.children.one?} / First has explicit header: #{!value.children.first.implicit}"
[9] pry(main)* end
explicit_doc -> Multiple docs: false / First has explicit header: true
implicit_doc -> Multiple docs: false / First has explicit header: false
multiple_docs -> Multiple docs: true / First has explicit header: false
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Interesting. I'll study this tomorrow.
lib/kubernetes-deploy/renderer.rb
Outdated
@_renderer.__send__(:bind_template_variables, erb_binding, variables) | ||
|
||
template = @_renderer.__send__(:find_partial, partial) | ||
expanded_template = ERB.new(template, nil, '-').result(erb_binding) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The errors that can be raised by this are not currently handled (unlike in the main render
method)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You are right, the error is handled on the main method. I thought that would be fine. But I'm happy to change it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
To clarify why I'd like this, it's to be able to have the error message we show reflect the specific file the error happened in, have line number references (if any) make sense, and be able to include a dump of the relevant partial only.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Changed.
@KnVerey I think I have addressed all items of the latest code review, except sharing of instance variables during rendering. If you insist, I will change that too. I will rebase and force push once you are OK with the contents. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The implementation looks good now, but I noticed we're producing some pretty terrible output when there are errors in partials; see my comment and my suggested fix (9cf798f...partial_review3).
I think I have addressed all items of the latest code review, except sharing of instance variables during rendering. If you insist, I will change that too.
Please do make that change too. In the absence of a compelling use case for allowing instance variables to be passed around instead of using locals, I'd prefer to start with the more restrictive option. The fact that rails partials allow this doesn't strike me as particularly relevant, since they are typically set in controllers in that context, and those have no analogue here.
end | ||
|
||
def test_invalid_partial_raises | ||
assert_raises(KubernetesDeploy::FatalDeploymentError) do |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Let's assert on the actual error message her and in the test below (can use our custom assert_raises_message
or the return value of this method)
lib/kubernetes-deploy/renderer.rb
Outdated
bind_template_variables(erb_binding, template_variables) | ||
|
||
ERB.new(raw_template, nil, '-').result(erb_binding) | ||
rescue StandardError => err |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Gave the feature another spin, including with some invalid templates, and noticed that rescuing StandardError
both here and in render_partial
is leading to some confusing error output when a partial is invalid (first screenshot) or includes a missing partial (second screenshot):
Let's not only fix that, but also provide the render sequence to help people debug. Here's one way to do that: 9cf798f...partial_review3
Since logging is our main UI in this project, I'd also like to see some integration tests with thorough assertions for those error cases. That commit also shows what I have in mind there. For reference, here's the "after" output:
Merged your proposed changes and ensured instance variables are not shared between partials. Please let me know if this is good to go or not. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM and CI is green. Please rebase and squash as previously discussed, and I'll merge and release this. Thanks for all your work on this feature!
60e978f
to
a856be7
Compare
Rebased and squashed. Please note there are still random test failures when running against a minikube 1.8 cluster, butthey are unrelated to the PR. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looking forward to seeing that getting used in production!
I wasn't sure whether the yaml/erb
code highlighting in the makefile was overlooked or decided to keep it as is. But that's my only comment.
README.md
Outdated
|
||
Inside a partial, parameters can be accessed as normal variables, or via a hash called `locals`. Thus, the `cron` template could like this: | ||
|
||
```erb |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Was this forgotten/dismissed?
a856be7
to
a81d488
Compare
@klautcomputing Force pushed latest change request. Might have overlooked changing it. |
\o/ |
We have a lot of repetition in our YAML files, mostly due to cronjobs.
I attacked the problem by adding partials. Mind you, this a proposal, which is
why I haven't added documentation yet, but there are tests.
The implemented partials system is very simplistic, although it looks
a bit like Rails templating. There's only one function supported, called
"partial", which takes two arguments: the name of the partial and an
argument hash. Partials are expected to be placed in a "partials"
subdirectory of the template directory and must have a '.yaml.erb'
suffix. Keys in the arguments hash are converted into local variables
for the partial. They are also accessible via the argument hash which is
passed in as variable 'locals'.