Add support for utf16 character encoding

[avihay-av]

Pull Request Checklist

Is this in reference to an existing issue?
Yes - #25

General

• Update Changelog following the conventions laid out on Keep A Changelog

• Update README with any necessary configuration snippets

• Binstubs are created if needed

• RuboCop passes

• Existing tests pass

New Plugins

• Tests

• Add the plugin to the README

• Does it have a complete header as outlined here

Purpose

Known Compatablity Issues

[majormoses]

@avoosh thanks for submitting will take a look in a second.

[majormoses]

Let's make it an option with a default matching old behavior to keep backwards compatibility.

[majormoses]

Let's make this an option, while UTF-16 does have some advantages it does also have some disadvantages as outlined here (quick summary): https://stackoverflow.com/questions/4655250/difference-between-utf-8-and-utf-16 to avoid breaking change we could have it default to utf8 and then override where neccessry.

[avihay-av]

@majormoses thx for the reply, I've added an option to encode utf16 before line matching.

[majormoses]

Based on what I saw that would not work as you would encode utf-16 and then re-encode utf-8. Anyways there is a better path forward. Let's just expose an option to use any encoding they want and just pass it in. Let's keep the default of UTF-8 to avoid breaking changes.

[majormoses]

why not make the option just encoding? for example:

option :encoding
        description: 'Encode line with the following encoding before matching',
        long: '--encoding $ENCODING'
        default: 'UTF-8'

This makes it easier when someone decides hey I want utf-32 encoding when that exists.

[majormoses]

I see your point that being said its trivial to set the encoding on the regex itself, if you still feel strongly about it I am willing to conceded as long as you add some comments in the code around it so a year from now we understand why it was done that way.

[avihay-av]

@majormoses

"A regexp can be matched against a string when they either share an encoding, or the regexp’s encoding is US-ASCII and the string’s encoding is ASCII-compatible."
[ruby-doc]

The issue that we have is invalid byte sequence when matching strings, and the prefered encoding in such cases is UTF-8 (ASCII is a compatible subset).
Since we want to remove invalid byte sequence, we are using the encode utf16 to do that for us.

"it is impossible to fix an invalid UTF-8 filename using a UTF-16 .... The opposite is not true: it is trivial to translate invalid UTF-16 to a unique (though technically invalid) UTF-8 string, so a UTF-8 API can control both UTF-8 and UTF-16 files and names"
[wiki]

Hence we do want to re-encode with utf-8 before calling the match function.
And the solution does keep the default of UTF-8 to avoid breaking changes. 👍

[majormoses]

just add a space between the # and the link and this looks good.

[majormoses]

trailing whitespace.

[majormoses]

released: https://rubygems.org/gems/sensu-plugins-logs/versions/1.2.0