New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for the Environment to optionally return native types. #708

Merged
merged 5 commits into from Oct 31, 2017

Conversation

Projects
None yet
6 participants
@jctanner
Contributor

jctanner commented Apr 27, 2017

This works by having an alternate CodeGenerator that avoids doing to_string
after the yield statement and a new version of concat that handles the returned
generator with a bit more "intelligence".

Related to ansible/ansible#23943

We use jinja heavily in the ansible project. Although it seems to target a text based destination for the renderers, our users have a desire to preserve the types of their templated vars. We also do a lot of internal intercept and post-processing to preserve those types but it's hit or miss and never obvious to the end user what will work and what won't. Therefore, I'm trying to extend jinja beyond what it's original usecase might have been.

This is a first pass and I hope to drive discussion with it and shape it into something the rest of the jinja community is happy with.

Show outdated Hide outdated tests/test_nativetypes.py Outdated
Show outdated Hide outdated jinja2/utils.py Outdated
Show outdated Hide outdated jinja2/utils.py Outdated
Show outdated Hide outdated jinja2/utils.py Outdated
@davidism

This comment has been minimized.

Show comment
Hide comment
@davidism

davidism May 1, 2017

Member

It took me a while to figure out what this was actually doing. Definitely needs some docs, but looks interesting.

The term "native" is a bit confusing, since I also associate it with discussion about what Python's str is in 2 vs 3.

How is Ansible using this? That is, when do users need access to the results of a render and don't have access to the original objects instead?

Does this need to be in Jinja, as opposed to a separate package providing a different Environment? Admittedly, there would be a little code duplication since the native checks happen in the middle of a couple methods. It just seems so radically different than what Jinja usually does.

Member

davidism commented May 1, 2017

It took me a while to figure out what this was actually doing. Definitely needs some docs, but looks interesting.

The term "native" is a bit confusing, since I also associate it with discussion about what Python's str is in 2 vs 3.

How is Ansible using this? That is, when do users need access to the results of a render and don't have access to the original objects instead?

Does this need to be in Jinja, as opposed to a separate package providing a different Environment? Admittedly, there would be a little code duplication since the native checks happen in the middle of a couple methods. It just seems so radically different than what Jinja usually does.

@jctanner

This comment has been minimized.

Show comment
Hide comment
@jctanner

jctanner May 1, 2017

Contributor

Hey @davidism

I'm working on addressing your inline code comments at the moment.

In terms of how Ansible will be using this... It's going to be transparent to our users, except that when they have chained template operations inside a playbook, types will be preserved all the way through.

Here's a basic example:

- hosts: el6host
  connection: local
  gather_facts: False
  vars:
      adict:
          foo: "1"
  tasks:
    - set_fact:
        tempres: "{{ adict.foo|int + 3 }}"
    - debug: var=tempres
    - debug: msg="{{ tempres | type_debug }}"

The type for "tempres" is -always- unicode in Ansible right now. With this feature, it will be an integer.

Contributor

jctanner commented May 1, 2017

Hey @davidism

I'm working on addressing your inline code comments at the moment.

In terms of how Ansible will be using this... It's going to be transparent to our users, except that when they have chained template operations inside a playbook, types will be preserved all the way through.

Here's a basic example:

- hosts: el6host
  connection: local
  gather_facts: False
  vars:
      adict:
          foo: "1"
  tasks:
    - set_fact:
        tempres: "{{ adict.foo|int + 3 }}"
    - debug: var=tempres
    - debug: msg="{{ tempres | type_debug }}"

The type for "tempres" is -always- unicode in Ansible right now. With this feature, it will be an integer.

Show outdated Hide outdated jinja2/utils.py Outdated
Show outdated Hide outdated jinja2/utils.py Outdated
Show outdated Hide outdated jinja2/utils.py Outdated
Show outdated Hide outdated jinja2/environment.py Outdated
Show outdated Hide outdated jinja2/environment.py Outdated
@ThiefMaster

This comment has been minimized.

Show comment
Hide comment
@ThiefMaster

ThiefMaster May 1, 2017

Member

Not sure if this really its in the Jinja core. Any chance this would be possible fully on the Ansible side by using a custom Environment/CodeGenerator/Runtime (I made the last two overridable on the Environment level some time ago)

Member

ThiefMaster commented May 1, 2017

Not sure if this really its in the Jinja core. Any chance this would be possible fully on the Ansible side by using a custom Environment/CodeGenerator/Runtime (I made the last two overridable on the Environment level some time ago)

Show outdated Hide outdated tests/test_nativetypes.py Outdated
@davidism

This comment has been minimized.

Show comment
Hide comment
@davidism

davidism May 1, 2017

Member

Possible concat implementation:

def concat(nodes):
    head = list(islice(nodes, 2))

    if not head:
        return None

    if len(head) == 1:
        out = head[0]
    else:
        out = u''.join([text_type(v) for v in nodes])

    try:
        return literal_eval(out)
    except (ValueError, SyntaxError, MemoryError):  # possibly RecursionError
        return out
Member

davidism commented May 1, 2017

Possible concat implementation:

def concat(nodes):
    head = list(islice(nodes, 2))

    if not head:
        return None

    if len(head) == 1:
        out = head[0]
    else:
        out = u''.join([text_type(v) for v in nodes])

    try:
        return literal_eval(out)
    except (ValueError, SyntaxError, MemoryError):  # possibly RecursionError
        return out
@jctanner

This comment has been minimized.

Show comment
Hide comment
@jctanner

jctanner May 1, 2017

Contributor

@davidism that concat example breaks this test ...

        native_env = Environment(native=True)
        native_tmpl = native_env.from_string("{% for x in listone %}{{ x }}{% endfor %}")
        result = native_tmpl.render(listone=['a', 'b', 'c', 'd'])
        assert isinstance(result, unicode)
        assert result == 'abcd'

The result is "cd", which I presume comes from having iterated over the first two values in the generator and not being able to seek backwards.

Contributor

jctanner commented May 1, 2017

@davidism that concat example breaks this test ...

        native_env = Environment(native=True)
        native_tmpl = native_env.from_string("{% for x in listone %}{{ x }}{% endfor %}")
        result = native_tmpl.render(listone=['a', 'b', 'c', 'd'])
        assert isinstance(result, unicode)
        assert result == 'abcd'

The result is "cd", which I presume comes from having iterated over the first two values in the generator and not being able to seek backwards.

@davidism

This comment has been minimized.

Show comment
Hide comment
@davidism

davidism May 1, 2017

Member

chain(head, nodes) fixes that.

Member

davidism commented May 1, 2017

chain(head, nodes) fixes that.

@jctanner

This comment has been minimized.

Show comment
Hide comment
@jctanner

jctanner May 2, 2017

Contributor

@davidism @ThiefMaster I think I've addressed everything suggested up till now. Any further thoughts?

Contributor

jctanner commented May 2, 2017

@davidism @ThiefMaster I think I've addressed everything suggested up till now. Any further thoughts?

@davidism

This comment has been minimized.

Show comment
Hide comment
@davidism

davidism May 2, 2017

Member

Needs documentation and changelog. Can be moved to a separate module, like the sandboxed env.

I still think this needs a clearer name than just "native". TypedOutputEnvironment? NativeTypeEnv? PythonTypeEnv? Something else? Or not?

Member

davidism commented May 2, 2017

Needs documentation and changelog. Can be moved to a separate module, like the sandboxed env.

I still think this needs a clearer name than just "native". TypedOutputEnvironment? NativeTypeEnv? PythonTypeEnv? Something else? Or not?

@jctanner

This comment has been minimized.

Show comment
Hide comment
@jctanner

jctanner May 2, 2017

Contributor

@davidism Would you want just the new code in environment.py moved to a separate file, or should I also move the new code in utils.py as well?

Contributor

jctanner commented May 2, 2017

@davidism Would you want just the new code in environment.py moved to a separate file, or should I also move the new code in utils.py as well?

@davidism

This comment has been minimized.

Show comment
Hide comment
@davidism

davidism May 2, 2017

Member

All of it.

Member

davidism commented May 2, 2017

All of it.

@jctanner

This comment has been minimized.

Show comment
Hide comment
@jctanner

jctanner May 2, 2017

Contributor

@davidism all code is in a separate "nativetypes.py" file now. I honestly don't know what makes sense in terms of naming the classes/files. I chose "native" because it espoused how I think about it, but I realize that's not how everyone thinks. If you or the rest of the team want to decide and pick a name, I'll do the renames.

For the docs, how much do you want in the docstrings versus in the "docs" directory? Is there a make script for the "docs" dir ... I didn't see anything obvious. How does the dev team build and examine docs prior to push?

Contributor

jctanner commented May 2, 2017

@davidism all code is in a separate "nativetypes.py" file now. I honestly don't know what makes sense in terms of naming the classes/files. I chose "native" because it espoused how I think about it, but I realize that's not how everyone thinks. If you or the rest of the team want to decide and pick a name, I'll do the renames.

For the docs, how much do you want in the docstrings versus in the "docs" directory? Is there a make script for the "docs" dir ... I didn't see anything obvious. How does the dev team build and examine docs prior to push?

@davidism

This comment has been minimized.

Show comment
Hide comment
@davidism

davidism May 2, 2017

Member

Install sphinx, cd to docs, make html, open _build/html/index.html. Don't worry if the theme doesn't look the same. See docs/sandbox.rst as a possible template for your docs.

Member

davidism commented May 2, 2017

Install sphinx, cd to docs, make html, open _build/html/index.html. Don't worry if the theme doesn't look the same. See docs/sandbox.rst as a possible template for your docs.

@jctanner

This comment has been minimized.

Show comment
Hide comment
@jctanner

jctanner May 4, 2017

Contributor

@davidism first pass on webdocs is done. I wasn't sure where to put it in the TOC, so I placed last in the list, beneath tips n tricks.

Here's a rendered example http://tannerjc.net/tmp/jinja/html/nativetypes.html

Contributor

jctanner commented May 4, 2017

@davidism first pass on webdocs is done. I wasn't sure where to put it in the TOC, so I placed last in the list, beneath tips n tricks.

Here's a rendered example http://tannerjc.net/tmp/jinja/html/nativetypes.html

Show outdated Hide outdated docs/nativetypes.rst Outdated
Show outdated Hide outdated jinja2/nativetypes.py Outdated
Show outdated Hide outdated jinja2/nativetypes.py Outdated
Show outdated Hide outdated jinja2/nativetypes.py Outdated
Show outdated Hide outdated jinja2/nativetypes.py Outdated
Show outdated Hide outdated tests/test_nativetypes.py Outdated
Show outdated Hide outdated tests/test_nativetypes.py Outdated
Show outdated Hide outdated tests/test_nativetypes.py Outdated
Show outdated Hide outdated tests/test_nativetypes.py Outdated
Show outdated Hide outdated tests/test_nativetypes.py Outdated
Show outdated Hide outdated docs/contents.rst.inc Outdated
@davidism

This comment has been minimized.

Show comment
Hide comment
@davidism

davidism May 9, 2017

Member

I'm still hesitant to add this to Jinja. It's such a standalone thing that it would make sense as a Jinja-NativeEnv package.

Our philosophy is to avoid adding more features unless they're absolutely necessary, since we have limited resources and want to keep the packages focused on one obvious use case. "Render to Python types" is not an obvious use case, given the features in the rest of the library.

Will you be willing to monitor this repository and keep this feature in sync with the rest of the code? Is there a plan within Ansible to provide maintenance for this?

@untitaker @ThiefMaster can you help make a decision about this?

Member

davidism commented May 9, 2017

I'm still hesitant to add this to Jinja. It's such a standalone thing that it would make sense as a Jinja-NativeEnv package.

Our philosophy is to avoid adding more features unless they're absolutely necessary, since we have limited resources and want to keep the packages focused on one obvious use case. "Render to Python types" is not an obvious use case, given the features in the rest of the library.

Will you be willing to monitor this repository and keep this feature in sync with the rest of the code? Is there a plan within Ansible to provide maintenance for this?

@untitaker @ThiefMaster can you help make a decision about this?

@sivel

This comment has been minimized.

Show comment
Hide comment
@sivel

sivel May 9, 2017

Speaking from my POV, and not on behalf of @jctanner

I had initially voiced concern about this living outside of jinja, as the likelihood for a change to crop up in jinja, that impacts this code could be pretty high.

My initial recommendation was to get buy in from the authors of jinja, to keep this in mind, and perform external validation to ensure they were not breaking the code in Ansible to perform this, and potentially keeping us informed of such changes. Obviously, this route has some problems.

Including in jinja helps us ensure that they code is functional. However at some level I do imagine we would need to vendor this code as well, to support older versions of jinja, such as those an OS packaging system would provide.

We already have some of this problem currently. An example was the 2.9 release, where changes broke some things for our users, that we had to deal with it, because we aren't tightly integrated from a community perspective.

I'm not necessarily recommending a particular solution, just voicing some concerns.

sivel commented May 9, 2017

Speaking from my POV, and not on behalf of @jctanner

I had initially voiced concern about this living outside of jinja, as the likelihood for a change to crop up in jinja, that impacts this code could be pretty high.

My initial recommendation was to get buy in from the authors of jinja, to keep this in mind, and perform external validation to ensure they were not breaking the code in Ansible to perform this, and potentially keeping us informed of such changes. Obviously, this route has some problems.

Including in jinja helps us ensure that they code is functional. However at some level I do imagine we would need to vendor this code as well, to support older versions of jinja, such as those an OS packaging system would provide.

We already have some of this problem currently. An example was the 2.9 release, where changes broke some things for our users, that we had to deal with it, because we aren't tightly integrated from a community perspective.

I'm not necessarily recommending a particular solution, just voicing some concerns.

@jctanner

This comment has been minimized.

Show comment
Hide comment
@jctanner

jctanner May 9, 2017

Contributor

Will you be willing to monitor this repository and keep this feature in sync with the rest of the code? Is there a plan within Ansible to provide maintenance for this?

We are going to have to "vendor" a copy of this code for older versions of jinja that our consumers might have, so yeah we have to keep ourselves in sync with upstream and do everything we can to keep the upstream functional.

My eventual goal is to see if the jinja dev team would consider allowing me to break up CodeGenerator.visitOutput into a few smaller functions. If so, the nativetypes code can be drastically smaller and maintenance will be much simpler. I did not want to muddy the waters and confuse that refactor with this feature, so I didn't bring it up yet.

Our philosophy is to avoid adding more features unless they're absolutely necessary, since we have limited resources and want to keep the packages focused on one obvious use case. "Render to Python types" is not an obvious use case, given the features in the rest of the library.

I fully understand that mentality. I'm still trying to build a list of useful examples of this feature that make sense outside ansible, but I also think that the community might have some input too.

Contributor

jctanner commented May 9, 2017

Will you be willing to monitor this repository and keep this feature in sync with the rest of the code? Is there a plan within Ansible to provide maintenance for this?

We are going to have to "vendor" a copy of this code for older versions of jinja that our consumers might have, so yeah we have to keep ourselves in sync with upstream and do everything we can to keep the upstream functional.

My eventual goal is to see if the jinja dev team would consider allowing me to break up CodeGenerator.visitOutput into a few smaller functions. If so, the nativetypes code can be drastically smaller and maintenance will be much simpler. I did not want to muddy the waters and confuse that refactor with this feature, so I didn't bring it up yet.

Our philosophy is to avoid adding more features unless they're absolutely necessary, since we have limited resources and want to keep the packages focused on one obvious use case. "Render to Python types" is not an obvious use case, given the features in the rest of the library.

I fully understand that mentality. I'm still trying to build a list of useful examples of this feature that make sense outside ansible, but I also think that the community might have some input too.

@mitsuhiko

This comment has been minimized.

Show comment
Hide comment
@mitsuhiko

mitsuhiko May 30, 2017

Member

I think I'm generally in favour of adding that. In the past I used similar things. I will have a look at this.

Member

mitsuhiko commented May 30, 2017

I think I'm generally in favour of adding that. In the past I used similar things. I will have a look at this.

@mitsuhiko mitsuhiko self-assigned this May 30, 2017

Show outdated Hide outdated jinja2/nativetypes.py Outdated
Show outdated Hide outdated jinja2/nativetypes.py Outdated
Show outdated Hide outdated jinja2/nativetypes.py Outdated
Show outdated Hide outdated jinja2/nativetypes.py Outdated
Show outdated Hide outdated jinja2/nativetypes.py Outdated
@jctanner

This comment has been minimized.

Show comment
Hide comment
@jctanner

jctanner Jun 8, 2017

Contributor

@davidism I removed all of the text_type refs and set the finalize noop per your suggestions. Tests still seem to pass for me.

Contributor

jctanner commented Jun 8, 2017

@davidism I removed all of the text_type refs and set the finalize noop per your suggestions. Tests still seem to pass for me.

@jctanner

This comment has been minimized.

Show comment
Hide comment
@jctanner

jctanner Jun 20, 2017

Contributor

anything else I can do to help out here?

Contributor

jctanner commented Jun 20, 2017

anything else I can do to help out here?

@davidism

This comment has been minimized.

Show comment
Hide comment
@davidism

davidism Oct 31, 2017

Member

It's been a while (sorry!) and there was some confusion over what we were waiting on here. https://botbot.me/freenode/pocoo/2017-10-31/?msg=92959650&page=3 Our conclusion was that adding has_safe_repr there was appropriate and fixed the issue I came up with.

Member

davidism commented Oct 31, 2017

It's been a while (sorry!) and there was some confusion over what we were waiting on here. https://botbot.me/freenode/pocoo/2017-10-31/?msg=92959650&page=3 Our conclusion was that adding has_safe_repr there was appropriate and fixed the issue I came up with.

@davidism davidism merged commit d17c7db into pallets:master Oct 31, 2017

1 check passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details
@jctanner

This comment has been minimized.

Show comment
Hide comment
@jctanner

jctanner Oct 31, 2017

Contributor

@davidism thank you so much! You've made many ansible devs happy today!

Contributor

jctanner commented Oct 31, 2017

@davidism thank you so much! You've made many ansible devs happy today!

@davidism

This comment has been minimized.

Show comment
Hide comment
@davidism

davidism Nov 8, 2017

Member

@jctanner I just released Jinja 2.10 with this included. I tried to ping you on Twitter but couldn't find your username. 😄

Member

davidism commented Nov 8, 2017

@jctanner I just released Jinja 2.10 with this included. I tried to ping you on Twitter but couldn't find your username. 😄

@jctanner

This comment has been minimized.

Show comment
Hide comment
@jctanner

jctanner Nov 9, 2017

Contributor

@davidism I do all my microblogging in github comments =)

New downstream patch for ansible ansible/ansible#32738

Contributor

jctanner commented Nov 9, 2017

@davidism I do all my microblogging in github comments =)

New downstream patch for ansible ansible/ansible#32738

@arodier

This comment has been minimized.

Show comment
Hide comment
@arodier

arodier Apr 2, 2018

Hello,
Thank for your hard work. In which version of Ansible this has been fixed?

arodier commented Apr 2, 2018

Hello,
Thank for your hard work. In which version of Ansible this has been fixed?

@jctanner

This comment has been minimized.

Show comment
Hide comment
@jctanner

jctanner Apr 2, 2018

Contributor

@arodier we're going to merge ansible/ansible#32738 at the very beginning of the ansible 2.7 development cycle.

Contributor

jctanner commented Apr 2, 2018

@arodier we're going to merge ansible/ansible#32738 at the very beginning of the ansible 2.7 development cycle.

@arodier

This comment has been minimized.

Show comment
Hide comment
@arodier

arodier Apr 2, 2018

Thank you!

arodier commented Apr 2, 2018

Thank you!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment