Problems with UTF8 encoding in gemspecs

[mckern]

I don't think this is a Geminabox problem per se, but I'm working my way through the chain. Whenever I upload a gem, say will_paginate-3.0.2.gem, any Syck encoded UTF8 characters get mangled:

Poor @mislav (whose last name should in fact render as Marohnić, not Marohnić). I've tried using Ruby 1.9.2-p320 & 1.9.3-p448, and Rubygems 1.8.25 & 2.1.5, and I've tried forcing YAML to use syck in config.ru (and incoming_gem.rb and geminabox.rb), to no effect.

I'm relatively confident that this is an underlying issue with Rubygems, not with Geminabox but I thought I'd start the chain here as I work my way through this and see if anyone else has run into this or might have better insight on where to focus my efforts.

[mckern]

I dug into the source for Rubygems 2.1.15 tonight, and found that to get the expected behavior I had to add these lines into config.ru:

require 'yaml'
YAML::ENGINE.yamler = 'syck'
ENV['TEST_SYCK'] = 'true'

There's more testing to do before I close my own issue (different versions of Ruby, different versions of Gem, and combinations there-of), but this worked in my quick-and-dirty testing tonight.

(Also, ick. I'm sad that it's 2013 and I still have to drop into Syck to correctly unmarshall these gemspecs.)

[tomlea]

Encoding is hard. I'm not confident around the various YAML flavours, and am too scared of YAMLgate like security problems to make this a system wide change. Do you know if this was a your machine problem, or is going to be a wider problem? The reason I ask is that I thought we'd solved the encoding issues earlier (unless some of the windows fixes have caused a regression).

[mckern]

I had this problem in disparate but similar environments: OS X 10.8.5 w/ Ruby 1.9.3p448 & Rubygems 2.1.5 and CentOS 6.4 w/ Ruby 1.9.3p448 & Rubygems 2.1.5. I'm loading a set of 685 gems into geminabox for a local mirror here at ${DAYJOB}; of those, 112 are in-house, leaving 573 external "real-world" gems.

I agree that encoding is hard and I definitely haven't done broad enough testing to validate whether what I did is the "real" way to solve the problem I had or not. I'm hoping to find time to wire up a harness for testing geminabox across numerous versions of Ruby (1.9.3 & 2.0.0) and Rubygems (1.8.23.2 through 2.1.5) and see what I turn up.