Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make binary data output possible #51387

Merged
merged 1 commit into from Feb 22, 2019

Conversation

@chrillux
Copy link
Contributor

commented Jan 29, 2019

If for example a pillar contains binary data it's possible that it won't
be possible to print it when doing e.g pillar.items:

File "/usr/lib/python3/dist-packages/salt/output/__init__.py", line 232, in strip_esc_sequence return txt.replace('\033', '?') TypeError: a bytes-like object is required, not 'str'

Replace non-printable data during output with a message instead.

What does this PR do?

We noticed that when we had binary data in our pillars, certain functions could not print the output. Also it wouldn't look very good printing big binary files that you have in your pillars. We haven't found any other way of doing this differently so here is a PR that solves this problem. Instead of trying to print the binary data, it will print a message instead.

When looking at the code you would think that in this passage it would be a string, because we are checking against "six.string_types". But this variable is set from type "Str" to a Tuple of the form: "(<class 'str'>, <class 'bytes'>)". I was not able to follow the runtime code fully, but it looks like the six.string_types is changed somewhere in the salt cli code. I'm very new to Saltstack so maybe all this is known things and intended behaviour.

What issues does this PR fix or reference?

None that I've seen.

Previous Behavior

Pillars containing binary data would make the pillar.items or other Salt output call to fail.

New Behavior

Output with binary data would be replaced with a message.

Tests written?

No

Commits signed with GPG?

No

If for example a pillar contains binary data it's possible that it won't
be possible to print it when doing e.g pillar.items:

  File "/usr/lib/python3/dist-packages/salt/output/__init__.py", line 232, in strip_esc_sequence
    return txt.replace('\033', '?')
TypeError: a bytes-like object is required, not 'str'

Replace non-printable data during output with a message instead.
@chrillux

This comment has been minimized.

Copy link
Contributor Author

commented Jan 30, 2019

From my research it looks like Saltstack is not supposed to support printing binary data, correct? And if you need to distribute binary files via pillars, e.g file_tree or custom ext_pillar, you need to base64 encode the data before putting it in the pillar. Then you can use file.decode, set encoding_type: base64, and your pillar will be decoded and saved as a binary file again.

This commit to modules/hashutil.py is needed for this to work though: ed25416

This is totally fine in our case. We have a custom Vault ext_pillar and when I set it to encode the file in base64 encoding, file.decode could correctly save the file as binary again.

@dwoz

This comment has been minimized.

Copy link
Contributor

commented Feb 13, 2019

@thatch45 Thoughts on this one?

@dwoz dwoz requested review from terminalmage and thatch45 Feb 13, 2019
@thatch45 thatch45 merged commit b1f7e85 into saltstack:2018.3 Feb 22, 2019
9 of 10 checks passed
9 of 10 checks passed
jenkins/pr/py3-windows-2016 The py3-windows-2016 job has failed
Details
WIP Ready for review
Details
continuous-integration/jenkins/pr-merge This commit looks good
Details
jenkins/pr/docs The docs job has passed
Details
jenkins/pr/lint Python lint test has passed
Details
jenkins/pr/py2-centos-7 The py2-centos-7 job has passed
Details
jenkins/pr/py2-ubuntu-1604 The py2-ubuntu-1604 job has passed
Details
jenkins/pr/py2-windows-2016 The py2-windows-2016 job has passed
Details
jenkins/pr/py3-centos-7 The py3-centos-7 job has passed
Details
jenkins/pr/py3-ubuntu-1604 The py3-ubuntu-1604 job has passed
Details
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.