Very weird bug: jinja2-cli undefined variables when using python in stdin #72

pawamoy · 2019-01-24T20:39:21Z

I'm not even sure this is the real cause, but here it is. I simplified the commands to keep it clear.

I want to run this:

$ ./output-json.py | jinja2 --format=json template.md

Easy right? But it leaves blanks were I put variables in my template. Indeed:

$ ./output-json.py | jinja2 --strict --format=json template.md
[truncated]
jinja2.exceptions.UndefinedError: 'main_usage' is undefined

I check that output-json.py actually outputs JSON:

$ ./output-json.py  # output truncated for readability
{"main_usage: "...", "commands": [...]}

So it seems jinja2-cli doesn't get my JSON contents.

But if I do

json_text="$(./output-json.py)"
echo "${json_text}" | jinja2 --format=json template.md

It works!!!! 😮

And

$ diff <(./output-json.py) <(echo "${json_text}")

...gives absolutely no difference!

Could it be because I'm running a Python script as input, and another Python script as output (jinja2), which messes with something in jinja2-cli's code? It seems crazy 😕

I tried variants like cat <(./output-json.py) | jinja2 ..., python output-json.py | jinja2 ..., echo "$(python output-json.py)" | jinja2 ..., but they all failed the same way.

I will try to get a reproducible example.

Maybe worth noting: I work with poetry in a virtualenv with Python 3.6. My jinja2-cli version is v0.6.0 and Jinja2 is v2.10.

The text was updated successfully, but these errors were encountered:

mattrobenolt · 2019-01-24T20:44:06Z

So. Funny that you mention this.

bf2120d

I had to fix this personally and had to bypass the auto detection of stdin. I was hitting the same issue and couldn't figure out how to make it work.

Can you try against latest master? It should be fine. I need to get around to tagging a new release. I've just been hacking on stuff to keep it working for me. :)

pawamoy · 2019-01-24T20:51:35Z

$ git clone https://github.com/mattrobenolt/jinja2-cli
Cloning into 'jinja2-cli'...
remote: Enumerating objects: 374, done.
remote: Total 374 (delta 0), reused 0 (delta 0), pack-reused 374
Receiving objects: 100% (374/374), 59.58 KiB | 0 bytes/s, done.
Resolving deltas: 100% (187/187), done.
$ workon aria2p-py3.6 
$ cd jinja2-cli/
$ python setup.py install
running install
running bdist_egg
running egg_info
creating jinja2_cli.egg-info
writing jinja2_cli.egg-info/PKG-INFO
writing dependency_links to jinja2_cli.egg-info/dependency_links.txt
writing entry points to jinja2_cli.egg-info/entry_points.txt
writing requirements to jinja2_cli.egg-info/requires.txt
writing top-level names to jinja2_cli.egg-info/top_level.txt
writing manifest file 'jinja2_cli.egg-info/SOURCES.txt'
reading manifest file 'jinja2_cli.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'jinja2_cli.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
running build_py
creating build
creating build/lib
creating build/lib/jinja2cli
copying jinja2cli/__init__.py -> build/lib/jinja2cli
copying jinja2cli/cli.py -> build/lib/jinja2cli
creating build/bdist.linux-x86_64
creating build/bdist.linux-x86_64/egg
creating build/bdist.linux-x86_64/egg/jinja2cli
copying build/lib/jinja2cli/__init__.py -> build/bdist.linux-x86_64/egg/jinja2cli
copying build/lib/jinja2cli/cli.py -> build/bdist.linux-x86_64/egg/jinja2cli
byte-compiling build/bdist.linux-x86_64/egg/jinja2cli/__init__.py to __init__.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/jinja2cli/cli.py to cli.cpython-36.pyc
creating build/bdist.linux-x86_64/egg/EGG-INFO
copying jinja2_cli.egg-info/PKG-INFO -> build/bdist.linux-x86_64/egg/EGG-INFO
copying jinja2_cli.egg-info/SOURCES.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying jinja2_cli.egg-info/dependency_links.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying jinja2_cli.egg-info/entry_points.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying jinja2_cli.egg-info/not-zip-safe -> build/bdist.linux-x86_64/egg/EGG-INFO
copying jinja2_cli.egg-info/requires.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying jinja2_cli.egg-info/top_level.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
creating dist
creating 'dist/jinja2_cli-0.7.0.dev0-py3.6.egg' and adding 'build/bdist.linux-x86_64/egg' to it
removing 'build/bdist.linux-x86_64/egg' (and everything under it)
Processing jinja2_cli-0.7.0.dev0-py3.6.egg
creating /media/Data/isolated/virtualenvs/aria2p-py3.6/lib/python3.6/site-packages/jinja2_cli-0.7.0.dev0-py3.6.egg
Extracting jinja2_cli-0.7.0.dev0-py3.6.egg to /media/Data/isolated/virtualenvs/aria2p-py3.6/lib/python3.6/site-packages
Adding jinja2-cli 0.7.0.dev0 to easy-install.pth file
Installing jinja2 script to /media/Data/isolated/virtualenvs/aria2p-py3.6/bin

Installed /media/Data/isolated/virtualenvs/aria2p-py3.6/lib/python3.6/site-packages/jinja2_cli-0.7.0.dev0-py3.6.egg
Processing dependencies for jinja2-cli==0.7.0.dev0
Searching for Jinja2==2.10
Best match: Jinja2 2.10
Adding Jinja2 2.10 to easy-install.pth file

Using /media/Data/isolated/virtualenvs/aria2p-py3.6/lib/python3.6/site-packages
Searching for MarkupSafe==1.1.0
Best match: MarkupSafe 1.1.0
Adding MarkupSafe 1.1.0 to easy-install.pth file

Using /media/Data/isolated/virtualenvs/aria2p-py3.6/lib/python3.6/site-packages
Finished processing dependencies for jinja2-cli==0.7.0.dev0
$ jinja2 --version
jinja2-cli v0.7.0.dev0
 - Jinja2 v2.10
$ cdrepo pawamoy/aria2p/
$ ./scripts/gen-readme-data.py | jinja2 --format=json scripts/templates/README.md 
Traceback (most recent call last):
  File "/media/Data/isolated/virtualenvs/aria2p-py3.6/bin/jinja2", line 11, in <module>
    load_entry_point('jinja2-cli==0.7.0.dev0', 'console_scripts', 'jinja2')()
  File "/media/Data/isolated/virtualenvs/aria2p-py3.6/lib/python3.6/site-packages/jinja2_cli-0.7.0.dev0-py3.6.egg/jinja2cli/cli.py", line 381, in main
    sys.exit(cli(opts, args))
  File "/media/Data/isolated/virtualenvs/aria2p-py3.6/lib/python3.6/site-packages/jinja2_cli-0.7.0.dev0-py3.6.egg/jinja2cli/cli.py", line 294, in cli
    sys.stdout.write(output)
TypeError: write() argument must be str, not bytes
$ ./scripts/gen-readme-data.py  # check output of my script
{"main_usage": "usage: aria2p [GLOBAL_OPTS...] COMMAND [COMMAND_OPTS...]\n\nCommand-line tool and Python library to interact with an `aria2c` daemon\nprocess through JSON-RPC.\n\nGlobal options:\n  -h, --help            Show this help message and exit. Commands also accept\n                        the -h/--help option.\n  -p PORT, --port PORT  Port to use to connect to the remote server.\n  -H HOST, --host HOST  Host address for the remote server.\n  -s SECRET, --secret SECRET\n                        Secret token to use to connect to the remote server.\n\nCommands:\n  \n    add-magnet          Add a download with a Magnet URI.\n    add-metalink        Add a download with a Metalink file.\n    add-torrent         Add a download with a Torrent file.\n    autopurge (autoclear)\n                        Automatically purge completed/removed/failed\n                        downloads.\n    call                Call a remote method through the JSON-RPC client.\n    pause               Pause downloads.\n    purge (clear)       Purge downloads.\n    pause-all           Pause all downloads.\n    remove (rm)         Remove downloads.\n    remove-all          Remove all downloads.\n    resume              Resume downloads.\n    resume-all          Resume all downloads.\n    show                Show the download progression.\n", "commands": [{"name": "add-magnet", "usage": "usage: aria2p add-magnet [-h] uri\n\nAdd a download with a Magnet URI.\n\npositional arguments:\n  uri         The magnet URI to use.\n\noptional arguments:\n  -h, --help  Show this help message and exit.\n"}, {"name": "add-metalink", "usage": "usage: aria2p add-metalink [-h] metalink_file\n\nAdd a download with a Metalink file.\n\npositional arguments:\n  metalink_file  The path to the metalink file.\n\noptional arguments:\n  -h, --help     Show this help message and exit.\n"}, {"name": "add-torrent", "usage": "usage: aria2p add-torrent [-h] torrent_file\n\nAdd a download with a Torrent file.\n\npositional arguments:\n  torrent_file  The path to the torrent file.\n\noptional arguments:\n  -h, --help    Show this help message and exit.\n"}, {"name": "autopurge", "usage": "usage: aria2p autopurge [-h]\n\nAutomatically purge completed/removed/failed downloads.\n\noptional arguments:\n  -h, --help  Show this help message and exit.\n"}, {"name": "call", "usage": "usage: aria2p call [-h] [-P PARAMS [PARAMS ...] | -J PARAMS] method\n\nCall a remote method through the JSON-RPC client.\n\npositional arguments:\n  method                The method to call (case insensitive). Dashes and\n                        underscores will be removed so you can use as many as\n                        you want, or none. Prefixes like 'aria2.' or 'system.'\n                        are also optional.\n\noptional arguments:\n  -h, --help            Show this help message and exit.\n  -P PARAMS [PARAMS ...], --params-list PARAMS [PARAMS ...]\n                        Parameters as a list of strings.\n  -J PARAMS, --json-params PARAMS\n                        Parameters as a JSON string. You should always wrap it\n                        at least once in an array '[]'.\n"}, {"name": "pause", "usage": "usage: aria2p pause [-h] [-f] gids [gids ...]\n\nPause downloads.\n\npositional arguments:\n  gids         The GIDs of the downloads to pause.\n\noptional arguments:\n  -h, --help   Show this help message and exit.\n  -f, --force  Pause without contacting servers first.\n"}, {"name": "purge", "usage": "usage: aria2p purge [-h] [gids [gids ...]]\n\nPurge downloads.\n\npositional arguments:\n  gids        The GIDs of the downloads to purge.\n\noptional arguments:\n  -h, --help  Show this help message and exit.\n"}, {"name": "pause-all", "usage": "usage: aria2p pause-all [-h] [-f]\n\nPause all downloads.\n\noptional arguments:\n  -h, --help   Show this help message and exit.\n  -f, --force  Pause without contacting servers first.\n"}, {"name": "remove", "usage": "usage: aria2p remove [-h] [-f] gids [gids ...]\n\nRemove downloads.\n\npositional arguments:\n  gids         The GIDs of the downloads to remove.\n\noptional arguments:\n  -h, --help   Show this help message and exit.\n  -f, --force  Remove without contacting servers first.\n"}, {"name": "remove-all", "usage": "usage: aria2p remove-all [-h] [-f]\n\nRemove all downloads.\n\noptional arguments:\n  -h, --help   Show this help message and exit.\n  -f, --force  Remove without contacting servers first.\n"}, {"name": "resume", "usage": "usage: aria2p resume [-h] gids [gids ...]\n\nResume downloads.\n\npositional arguments:\n  gids        The GIDs of the downloads to resume.\n\noptional arguments:\n  -h, --help  Show this help message and exit.\n"}, {"name": "resume-all", "usage": "usage: aria2p resume-all [-h]\n\nResume all downloads.\n\noptional arguments:\n  -h, --help  Show this help message and exit.\n"}, {"name": "show", "usage": "usage: aria2p show [-h]\n\nShow the download progression.\n\noptional arguments:\n  -h, --help  Show this help message and exit.\n"}]}

mattrobenolt · 2019-01-24T20:54:16Z

Ughh, why do I keep breaking this. Gimem a few minutes. I only use this with python 2.7, so I never hit these issues. :(

pawamoy · 2019-01-24T20:55:38Z

Haha no problem, take your time 🙂

mattrobenolt · 2019-01-24T21:02:05Z

Wanna try latest master again? 207d910

I confirmed manually with python3.7 -m jinja2cli.cli

pawamoy · 2019-01-24T21:05:53Z

It works!!!

pawamoy · 2019-01-24T21:06:08Z

Thank you for this very quick fix 😉

mattrobenolt · 2019-01-24T21:06:39Z

We did it. 👏

pawamoy · 2019-03-08T19:37:50Z

It's doing it again with master branch version... I don't get it...

pawamoy · 2019-03-08T19:56:42Z

Reproduce:

git clone https://github.com/pawamoy/aria2p
cd aria2p
poetry install
make readme   # OK
make credits  # not OK

Error output:

$ make credits
poetry run ./scripts/gen-credits-data.py | \
	poetry run jinja2 --strict --format=json scripts/templates/CREDITS.md > CREDITS.md
Traceback (most recent call last):
  File "/media/Data/isolated/virtualenvs/aria2p-py3.6/bin/jinja2", line 11, in <module>
    load_entry_point('jinja2-cli', 'console_scripts', 'jinja2')()
  File "/media/Data/isolated/virtualenvs/aria2p-py3.6/src/jinja2-cli/jinja2cli/cli.py", line 382, in main
    sys.exit(cli(opts, args))
  File "/media/Data/isolated/virtualenvs/aria2p-py3.6/src/jinja2-cli/jinja2cli/cli.py", line 291, in cli
    output = render(template_path, data, extensions, opts.strict)
  File "/media/Data/isolated/virtualenvs/aria2p-py3.6/src/jinja2-cli/jinja2cli/cli.py", line 215, in render
    output = env.get_template(os.path.basename(template_path)).render(data)
  File "/media/Data/isolated/virtualenvs/aria2p-py3.6/lib/python3.6/site-packages/jinja2/asyncsupport.py", line 76, in render
    return original_render(self, *args, **kwargs)
  File "/media/Data/isolated/virtualenvs/aria2p-py3.6/lib/python3.6/site-packages/jinja2/environment.py", line 1008, in render
    return self.environment.handle_exception(exc_info, True)
  File "/media/Data/isolated/virtualenvs/aria2p-py3.6/lib/python3.6/site-packages/jinja2/environment.py", line 780, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/media/Data/isolated/virtualenvs/aria2p-py3.6/lib/python3.6/site-packages/jinja2/_compat.py", line 37, in reraise
    raise value.with_traceback(tb)
  File "/media/Data/dev/pawamoy/aria2p/scripts/templates/CREDITS.md", line 7, in top-level template code
    ### Direct dependencies{% for dep in direct_dependencies %}
jinja2.exceptions.UndefinedError: 'direct_dependencies' is undefined
Makefile:14: recipe for target 'credits' failed
make: *** [credits] Error 1

Makefile rules:

.PHONY: readme
readme:  ## Regenerate README.md.
	poetry run ./scripts/gen-readme-data.py | \
		poetry run jinja2 --strict --format=json scripts/templates/README.md > README.md

.PHONY: credits
credits:  ## Regenerate CREDITS.md.
	poetry run ./scripts/gen-credits-data.py | \
		poetry run jinja2 --strict --format=json scripts/templates/CREDITS.md > CREDITS.md

Script gen-readme-data.py:

#!/usr/bin/env python

import argparse
import json

from aria2p.cli import get_parser

parser = get_parser()

aliases = ["autoclear", "clear", "rm"]

output = {"main_usage": parser.format_help(), "commands": []}

subparser_actions = [action for action in parser._actions if isinstance(action, argparse._SubParsersAction)]

for subparser_action in subparser_actions:
    for choice, subparser in subparser_action.choices.items():
        if choice in aliases:
            continue
        output["commands"].append({"name": choice, "usage": subparser.format_help()})

json_output = json.dumps(output)
print(json_output)

Script gen-credits-data.py:

#!/usr/bin/env python

import json
import os

import toml
from pip._internal.commands.show import search_packages_info


def clean_info(p):
    return {k: v for k, v in p.items() if k not in ("location", "files", "entry_points")}


metadata = toml.load(os.path.join(os.path.dirname(os.path.dirname(__file__)), "pyproject.toml"))["tool"]["poetry"]
direct_dependencies = sorted(list(metadata["dependencies"].keys()) + list(metadata["dev-dependencies"].keys()))
direct_dependencies.remove("python")

lock_data = toml.load(os.path.join(os.path.dirname(os.path.dirname(__file__)), "poetry.lock"))
indirect_dependencies = sorted([p["name"] for p in lock_data["package"] if p["name"] not in direct_dependencies])

package_info = {p["name"]: clean_info(p) for p in search_packages_info(direct_dependencies + indirect_dependencies)}

lower_package_info = {}
for package_name, package in package_info.items():
    lower = package_name.lower()
    if lower != package_name:
        lower_package_info[lower] = package

package_info.update(lower_package_info)

del lower_package_info
del lock_data
del metadata


json_output = json.dumps(
    {
        "direct_dependencies": direct_dependencies,
        "indirect_dependencies": indirect_dependencies,
        "package_info": package_info,
    }
)
print(json_output)

One difference is that gen-credits-data.py takes a bit more time to start writing to stdout, like almost 1 second.

pawamoy · 2019-03-08T23:58:21Z

Ah! It works correctly if I add - in the arguments passed to jinja2:

credits:
	poetry run ./scripts/gen-credits-data.py | \
		poetry run jinja2 --format=json scripts/templates/CREDITS.md - -o CREDITS.md

Still it's confusing that it works in one case and not the other ^^

dvershinin · 2020-03-22T02:38:51Z

I'm getting this also with the latest release and Python 3.6.
Putting - works, but is frustrating nonetheless.

mattrobenolt closed this as completed Jan 24, 2019

pawamoy mentioned this issue Jan 31, 2019

New release #73

Closed

chfw mentioned this issue Mar 8, 2019

Would you be interested in moban project? #74

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Very weird bug: jinja2-cli undefined variables when using python in stdin #72

Very weird bug: jinja2-cli undefined variables when using python in stdin #72

pawamoy commented Jan 24, 2019 •

edited

mattrobenolt commented Jan 24, 2019

pawamoy commented Jan 24, 2019

mattrobenolt commented Jan 24, 2019

pawamoy commented Jan 24, 2019

mattrobenolt commented Jan 24, 2019

pawamoy commented Jan 24, 2019

pawamoy commented Jan 24, 2019

mattrobenolt commented Jan 24, 2019

pawamoy commented Mar 8, 2019

pawamoy commented Mar 8, 2019 •

edited

pawamoy commented Mar 8, 2019

dvershinin commented Mar 22, 2020

Very weird bug: jinja2-cli undefined variables when using python in stdin #72

Very weird bug: jinja2-cli undefined variables when using python in stdin #72

Comments

pawamoy commented Jan 24, 2019 • edited

mattrobenolt commented Jan 24, 2019

pawamoy commented Jan 24, 2019

mattrobenolt commented Jan 24, 2019

pawamoy commented Jan 24, 2019

mattrobenolt commented Jan 24, 2019

pawamoy commented Jan 24, 2019

pawamoy commented Jan 24, 2019

mattrobenolt commented Jan 24, 2019

pawamoy commented Mar 8, 2019

pawamoy commented Mar 8, 2019 • edited

pawamoy commented Mar 8, 2019

dvershinin commented Mar 22, 2020

pawamoy commented Jan 24, 2019 •

edited

pawamoy commented Mar 8, 2019 •

edited