When opening files in 1.9.x the following mode specification is valid: "r:bom|utf-8"
This seems to be a fallback syntax "if you can't open it in bom mode try utf-8".
I found the entry for the same bug for jruby http://jira.codehaus.org/browse/JRUBY-6320 but not the documented source yet.
The following code reproduces the problem
open path, "r:bom|utf-8" do |f|
Workaround for rubinius/rubinius#1766
Fixed in 5be1b2e
FWIW, the code in JRuby is here: https://github.com/jruby/jruby/blob/master/src/org/jruby/RubyIO.java#L4407