Assume symbols are either ASCII or UTF-8

[casperisfine]

This is a fresh take on #211

Right now only US-ASCII are properly preserved. Any other encoding comes back with ASCII-8BIT (binary) encoding.

After this change UTF-8 symbols are properly preserved as well. Other encoding cause an EncodingError. Since UTF-8 is the only encoding handled by msgpack strings, I think this change makes sense.

[tagomoris]

Now I'm afraid that someone will accidentally be surprised by sudden EncodingError after upgrading msgpack. It may break users' applications that are running now (with hidden encoding mismatch currently).
Can't we avoid raising errors at deserialization?

[casperisfine]

I'll look into it.

[casperisfine]

Here's the solution. I first try what should be the happy path, and deal with the error in a rescue as this should basically never happen ever, so no reason the happy path should pay an extra cost for it.

[tagomoris]

Thanks for the fix and awesome code comment!

[tagomoris]

Oh, CI is failing with JRuby.

[casperisfine]

Hum, seems like a JRuby bug IMO. Somehow it seems to picks up the existing symbol even though the encoding doesn't match.

I'll likely submit a PR to https://github.com/ruby/spec for this.

I'll check how it currently behaves on master and try to keep that same behavior.

[casperisfine]

Done.

[casperisfine]

RUby Spec for this ruby/spec#919, so it's likely that future versions of JRuby will have this fixed.

[casperisfine]

I just noticed there is a small bug with symbols used as hash keys:

        it 'works with hash keys' do
          expect(roundtrip(symbol: 1)).to be == { symbol: 1 }
        end

  1) MessagePack::Factory the special treatment of symbols with ext type if an ext type is registered for symbols if using the default serializer works with hash keys
     Failure/Error: data.force_encoding(Encoding::UTF_8).to_sym
     
     FrozenError:
       can't modify frozen String: "symbol"
     # ./lib/msgpack/symbol.rb:20:in `force_encoding'
     # ./lib/msgpack/symbol.rb:20:in `from_msgpack_ext'
     # ./lib/msgpack/factory.rb:70:in `call'
     # ./lib/msgpack/factory.rb:70:in `full_unpack'
     # ./lib/msgpack/factory.rb:70:in `load'
     # ./spec/factory_spec.rb:290:in `roundtrip'
     # ./spec/factory_spec.rb:337:in `block (5 levels) in <top (required)>'

This is because Ruby freeze strings used as hash keys. I'll submit a fix as soon as possible.

[Laykou]

@casperisfine I see this was already merged prior the last fixes.

For me this fixed an issue with special characters in YAML files, mentioned here: ruby-i18n/i18n#606

@tagomoris They are waiting for new version release for mgspack. Do you think new version could be released soon?

[tagomoris]

@Laykou My original plan is to release a new version including many changes (not merged yet), but yes, I can make a release with enough fixes for symbols.
I think I'll be able to find time for it today or in 2-3 days.

[tagomoris]

Shipped v1.4.5 🎉