(MODULES-7143) add UTF-8 encoding to ERB templates.

[ThoughtCrhyme]

This PR adds UTF-8 support for fields inside of custom dsc templates.

It does not support the template custom name itself being in UTF-8, but parameters passed to fields in the template can now be UTF-8.

Also added an integration test for this functionality.

[Iristyle]

I believe all of these characters are the same byte width... can we instead use characters whose width vary?

The standard string we use in most places is

# different UTF-8 widths
  # 1-byte A
  # 2-byte ۿ - http://www.fileformat.info/info/unicode/char/06ff/index.htm - 0xDB 0xBF / 219 191
  # 3-byte ᚠ - http://www.fileformat.info/info/unicode/char/16A0/index.htm - 0xE1 0x9A 0xA0 / 225 154 160
  # 4-byte 𠜎 - http://www.fileformat.info/info/unicode/char/2070E/index.htm - 0xF0 0xA0 0x9C 0x8E / 240 160 156 142
  MIXED_UTF8 = "A\u06FF\u16A0\u{2070E}" # Aۿᚠ𠜎

See https://github.com/puppetlabs/puppet/blob/7e210c334f0231e961b699c663c3efd4b32373bf/spec/integration/faces/config_spec.rb#L13 for an example

[ThoughtCrhyme]

Yup, we can use variable bit width here.

[Iristyle]

Should we use a Powershell call with EncodedCommand here to avoid sending UTF-8 over SSH?

[Iristyle]

Should we use a Powershell call with EncodedCommand here to avoid sending UTF-8 over SSH?

[Iristyle]

I'm pretty sure cat in PowerShell doesn't set an exit code since its an alias to a cmdlet ... you'll want to make sure the contents are empty and exit the command yourself I think.

[Iristyle]

Could also do a file count in the other example above, and verify there are 0 files by the given name.

[Iristyle]

This might be giving you a (false?) positive result, but I'm not sure it's quite right. You want to pass the UTF8 over the wire to the host using EncodedCommand => true. Something like this (going from memory, verify that I didn't screw anything up):

powershell_cmd = "dir c:\\ | ? { $_.BaseName -eq '#{fake_name}' } | Measure-Object | Select Count"

This guarantees that PowerShell is taking Unicode input and using that for a comparison. Because we can't trust what's coming back from the remote host, then the assertion becomes that you got 1 matching result. Something like:

assert_equal('1', result.stdout, 'File with correct UTF-8 characters was not present.')

[Iristyle]

We might want to do this a little differently to avoid the UTF-8 problem.

One solutions is to write these files to a randomized directory (ASCII characters only) and recursively delete it.

[Iristyle]

^ this is fine, but SecureRandom.uuid does roughly the same thing (uuids are limited to [0-9A-F]) ;0

[Iristyle]

A little pedantic, but if we want to make sure we're matching the file by name we intended to create, we'd want:

(Get-ChildItem C:\\#{file_path} -Filter '#{file_name}' | Measure-Object).Count

You may have run into errors doing something like Get-ChildItem "C:\\#{file_path}\\#{file_name}" but this will avoid that. Examples of executing code:

C:\source\> (Get-ChildItem .\locales\ -Filter 'config.foo' | Measure-Object).Count
0

C:\source\> (Get-ChildItem .\locales\ -Filter 'config.yaml' | Measure-Object).Count
1

With a randomized parent directory, there shouldn't be other files in there (unless there was a subtle test bug or a future test maintainer made a subtly incorrect change). I'm pointing this out for the specificity reason, but also because we don't need to use EncodedCommand with how you have the assert setup anymore. I think the -Filter is preferable, but if you feel strongly about the other, we should yank the EncodedCommand bit since file_path is only alpha-numeric.

[Iristyle]

Call the zookeeper... we've got a wild pry on the loose

[ThoughtCrhyme]

lulz

[Iristyle]

What happens with multi-line output (in a failing test case)? Do we need to use anchors on the string (or maybe just assert_equal and skip the regex)?

[Iristyle]

Same here regex vs assert_equal