Problems with UTF8 encoding in gemspecs #134

mckern opened this Issue Oct 1, 2013 · 3 comments


None yet

2 participants

mckern commented Oct 1, 2013

I don't think this is a Geminabox problem per se, but I'm working my way through the chain. Whenever I upload a gem, say will_paginate-3.0.2.gem, any Syck encoded UTF8 characters get mangled:


Poor @Mislav (whose last name should in fact render as Marohnić, not Marohnić). I've tried using Ruby 1.9.2-p320 & 1.9.3-p448, and Rubygems 1.8.25 & 2.1.5, and I've tried forcing YAML to use syck in (and incoming_gem.rb and geminabox.rb), to no effect.

I'm relatively confident that this is an underlying issue with Rubygems, not with Geminabox but I thought I'd start the chain here as I work my way through this and see if anyone else has run into this or might have better insight on where to focus my efforts.

mckern commented Oct 2, 2013

I dug into the source for Rubygems 2.1.15 tonight, and found that to get the expected behavior I had to add these lines into

require 'yaml'
YAML::ENGINE.yamler = 'syck'
ENV['TEST_SYCK'] = 'true'

There's more testing to do before I close my own issue (different versions of Ruby, different versions of Gem, and combinations there-of), but this worked in my quick-and-dirty testing tonight.

(Also, ick. I'm sad that it's 2013 and I still have to drop into Syck to correctly unmarshall these gemspecs.)

tomlea commented Oct 8, 2013

Encoding is hard. I'm not confident around the various YAML flavours, and am too scared of YAMLgate like security problems to make this a system wide change. Do you know if this was a your machine problem, or is going to be a wider problem? The reason I ask is that I thought we'd solved the encoding issues earlier (unless some of the windows fixes have caused a regression).

mckern commented Oct 8, 2013

I had this problem in disparate but similar environments: OS X 10.8.5 w/ Ruby 1.9.3p448 & Rubygems 2.1.5 and CentOS 6.4 w/ Ruby 1.9.3p448 & Rubygems 2.1.5. I'm loading a set of 685 gems into geminabox for a local mirror here at ${DAYJOB}; of those, 112 are in-house, leaving 573 external "real-world" gems.

I agree that encoding is hard and I definitely haven't done broad enough testing to validate whether what I did is the "real" way to solve the problem I had or not. I'm hoping to find time to wire up a harness for testing geminabox across numerous versions of Ruby (1.9.3 & 2.0.0) and Rubygems ( through 2.1.5) and see what I turn up.

@tomlea tomlea added the Stale label Feb 15, 2016
@tomlea tomlea closed this Feb 15, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment