Add support for unicode in ansible-inventory CLI

[mkrizek]

SUMMARY

Fixes #57378

ISSUE TYPE

• Bugfix Pull Request

COMPONENT NAME

lib/ansible/cli/inventory.py

[bcoca]

code LGTM, would be nice to have test to avoid regressions

[abadger]

Ugh. I've looked at this file wholistically and it's unicode handling looks like it is very, very wrong. I think that it needs a lot of changes to work right. ping me on irc and I can walk through it with you.

[mkrizek]

So as a summary of what this PR does:

• fixes ability to print unicode chracters for ansible-inventory --list and ansible-inventory --list --yaml commands and also --output (output to a file) for both formats on Python 2 and 3,
• adds integration tests for all of above,
• does NOT touch any --toml code - unicode works for --toml in devel branch on Python 3 correctly. On Python 2 --toml prints unicode characters incorrectly (in a form of \u1234) but that is an existing issue in devel branch so this PR does NOT break that - the reason I found out about that is because I added a test for --toml for completeness. In the end, the test is restricted for Python 3 only.

As per above, this PR is an improvement to the unicode support in ansible-inventory --list over what is currently in devel without breaking or removing any functionality and as such could be backported.

Certainly there is more to improve but I don't think we should hold off on merging this (unless someone thinks the PR as is is incorrect of course). We can follow up in another PR. We can also leave #57378 open.

[abadger]

I discussed this with mkrizek on IRC and he fixed the major problem that I saw (writing the file as text which can traceback and could write the file in non-utf8 which ansible will not read correctly) and we also agreed on the other things that should be fixed in a followup(s) (fix use of to_native when calling display.*(), moving the toml routines to a [possibly private] utility file, moving the transformation of yaml output from bytes to text into AnsibleDumper).

We also discussed the toml output a bit. On Python2, the toml library both removes and adds backslashes that it shouldn't, which badly mangles the escaped strings. it would be possible to fix the problem in our toml_dumps function https://github.com/ansible/ansible/blob/devel/lib/ansible/plugins/inventory/toml.py#L118 by overriding the str, unicode (and AnsibleUnicode, AnsibleUnsafeBytes, and AnsibleUnsafeText) handlers in dump_funcs. We could also deprecate use of the toml library and promote use of tomlkit because tomlkit does the right thing here. However, the problem only occurs on Python2. The python3 repr functionality which the toml library relies on handles output of non-ascii characters in a non-supportive locale in a different way such that the toml library does not mangle those.