Join GitHub today
GitHub is home to over 40 million developers working together to host and review code, manage projects, and build software together.Sign up
2019.2.0 binary pillar unicode error returns. #51879
Description of Issue/Question
It looks like #46934 has come back in
Same setup as #46934
Steps to Reproduce Issue
binary gpg render setup.
I'm still trying to wrap my head around whether or not this truly is a bug, or it's just a feature that we don't yet support.
My understanding is that we only support base64 encoded binary data in pillars. It appears that what's happening here is that when binary data is being decrypted (and contains some non-ascii values), it breaks the pillar - which currently is expected - if I create this pillar:
(where those ~K are literally 0x8b chars. In vim edit your pillar that's got say,
and try to run
It's actually super interesting to me that this used to be a thing that maybe worked - from what I can tell that's actually more of an accident due python2 treating bytes and text the same. I've been able to change the gpg renderer to return the decrypted pillar, but the failure ends out percolating up because it expects pillar data to be ascii/unicode, and not binary data.
Oh. This is interesting. Apparently if you stick a null byte in your binary data, it will happily dump the contents of the pillar, at least with the few changes I've made. I still have to gpg the text, but it will actually call it a binary file. It may make sense that if we fail decoding pillar contents that we should assume that it's binary data...
I've run into another bit of a hurdle.
It's clear that we have some challenges to get to a point where we have cohesive bytes/text story. The original behavior only existed in the first place because Python2 was terribly lazy at text vs. binary data, so it was pretty implicit about just writing strings including binary data.
When we started to support (now) legacy Python and Python3 it looks like there was an implicit understanding that pillars would be textual data (that is, ASCII or Unicode), along with basically everything else. It's unfortunate, because technically speaking there's no real reason that we can't be sticking binary data into pillars - in fact, YAML has a binary datatype.
From what it looks, there is a possibility that serializing binary data via JSON could be problematic, though - which may only come into play for salt-ssh but is something that we should definitely be aware of.