Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

Update psych to edf05c9f40e0c566e391f7e04b4a30327c31a828 #578

Closed
wants to merge 1 commit into from

3 participants

Alex Tambellini Hiro Asari Charles Oliver Nutter
Alex Tambellini
Collaborator

There's tons of string quotation fixes that I would :heart: to see in jruby 1.7.4. In order to generate this diff I ran the following commands:

cp -R <psych checkout>/psych/lib/psych <jruby checkout>/lib/ruby/1.9
cp <psych checkout>/psych/lib/psych.rb <jruby checkout>/lib/ruby/1.9/psych.rb

Hiro Asari
Owner

I'd personally very much prefer that there is a Psych release which includes this.

Alex Tambellini
Collaborator

@tenderlove, could you make a 2.0.0 release of the psych gem?

If a 2.0.0 release of the psych gem is made will I be able to put that in my Gemfile and it will take precedence over the stdlib version in jruby? I know the json gem does that but there is a java platform version of the json gem...while the psych gem has a bunch of native stuff in it. I'm guessing the only way to do the psych in Gemfile thing for jruby would be to move to the way the json gem is handled by moving all of the jruby java bits into the psych gem itself?

Charles Oliver Nutter
Owner

We do not currently support loading psych from the gem rather than from stdlib. We would like to be able to do that.

Until then, we'll have to update stdlib to pick up updates. For that, we will tend to prefer released versions.

Alex Tambellini
Collaborator

Sounds good, it looks like the 2.0 stdlib already has the fixes I need so I'll just wait for the next release and use --2.0 mode.

Alex Tambellini atambo closed this
Charles Oliver Nutter
Owner

I talked with @tenderlove and he said psych 2.0 is basically done but he's been holding off doing a gem release. We could pull from his 2.0.0 branch and apply updated psych to JRuby's 1.9 mode.

So, we could probably update any time. Any concerns about pulling psych 2.0 into our 1.9 mode?

Reopening to keep this task fresh.

Charles Oliver Nutter headius reopened this
Charles Oliver Nutter
Owner

@tenderlove also told me he does plan to backport some fixes into 1.9.3, which would mean we'd pick them up after that. So...we could move straight to 2.0 psych right now, or just wait until fixes trickle back.

I'd just as soon go to psych 2.0 stuff right now, so I'm going to explore that quickly.

Charles Oliver Nutter
Owner

Sole failure in 1.9 psych tests when updating psych to the 2.0 versions:

  1) Failure:
test_tagged_binary_should_be_dumped_as_binary(Psych::TestString) [/Users/headius/projects/jruby/test/externals/ruby1.9/psych/test_string.rb:47]:
Expected /binary/ to match "--- hello world!\n".
Alex Tambellini
Collaborator

@headius, will 5120e5f update psych for 1.9 and 2.0 mode or only 2.0 mode? If so, then this can be closed?

Charles Oliver Nutter
Owner

Ahh yes, 5120e5f does indeed use psych 2.0 in both 1.9 and 2.0 modes, so this can be closed.

Charles Oliver Nutter headius closed this
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
This page is out of date. Refresh to see the latest.
195 lib/ruby/1.9/psych.rb
View
@@ -18,8 +18,8 @@
###
# = Overview
#
-# Psych is a YAML parser and emitter.
-# Psych leverages libyaml [Home page: http://pyyaml.org/wiki/LibYAML]
+# Psych is a YAML parser and emitter.
+# Psych leverages libyaml [Home page: http://pyyaml.org/wiki/LibYAML]
# or [Git repo: https://github.com/zerotao/libyaml] for its YAML parsing
# and emitting capabilities. In addition to wrapping libyaml, Psych also
# knows how to serialize and de-serialize most Ruby objects to and from
@@ -43,14 +43,72 @@
# level, is an event based parser. Mid level is access to the raw YAML AST,
# and at the highest level is the ability to unmarshal YAML to ruby objects.
#
-# === Low level parsing
+# == YAML Emitting
#
-# The lowest level parser should be used when the YAML input is already known,
-# and the developer does not want to pay the price of building an AST or
-# automatic detection and conversion to ruby objects. See Psych::Parser for
-# more information on using the event based parser.
+# Psych provides a range of interfaces ranging from low to high level for
+# producing YAML documents. Very similar to the YAML parsing interfaces, Psych
+# provides at the lowest level, an event based system, mid-level is building
+# a YAML AST, and the highest level is converting a Ruby object straight to
+# a YAML document.
+#
+# == High-level API
+#
+# === Parsing
+#
+# The high level YAML parser provided by Psych simply takes YAML as input and
+# returns a Ruby data structure. For information on using the high level parser
+# see Psych.load
+#
+# ==== Reading from a string
+#
+# Psych.load("--- a") # => 'a'
+# Psych.load("---\n - a\n - b") # => ['a', 'b']
+#
+# ==== Reading from a file
+#
+# Psych.load_file("database.yml")
+#
+# ==== Exception handling
#
-# === Mid level parsing
+# begin
+# # The second argument chnages only the exception contents
+# Psych.parse("--- `", "file.txt")
+# rescue Psych::SyntaxError => ex
+# ex.file # => 'file.txt'
+# ex.message # => "(file.txt): found character that cannot start any token"
+# end
+#
+# === Emitting
+#
+# The high level emitter has the easiest interface. Psych simply takes a Ruby
+# data structure and converts it to a YAML document. See Psych.dump for more
+# information on dumping a Ruby data structure.
+#
+# ==== Writing to a string
+#
+# # Dump an array, get back a YAML string
+# Psych.dump(['a', 'b']) # => "---\n- a\n- b\n"
+#
+# # Dump an array to an IO object
+# Psych.dump(['a', 'b'], StringIO.new) # => #<StringIO:0x000001009d0890>
+#
+# # Dump an array with indentation set
+# Psych.dump(['a', ['b']], :indentation => 3) # => "---\n- a\n- - b\n"
+#
+# # Dump an array to an IO with indentation set
+# Psych.dump(['a', ['b']], StringIO.new, :indentation => 3)
+#
+# ==== Writing to a file
+#
+# Currently there is no direct API for dumping Ruby structure to file:
+#
+# File.open('database.yml', 'w') do |file|
+# file.write(Psych.dump(['a', 'b']))
+# end
+#
+# == Mid-level API
+#
+# === Parsing
#
# Psych provides access to an AST produced from parsing a YAML document. This
# tree is built using the Psych::Parser and Psych::TreeBuilder. The AST can
@@ -58,28 +116,33 @@
# Psych::Nodes, and Psych::Nodes::Node for more information on dealing with
# YAML syntax trees.
#
-# === High level parsing
+# ==== Reading from a string
#
-# The high level YAML parser provided by Psych simply takes YAML as input and
-# returns a Ruby data structure. For information on using the high level parser
-# see Psych.load
+# # Returns Psych::Nodes::Stream
+# Psych.parse_stream("---\n - a\n - b")
#
-# == YAML Emitting
+# # Returns Psych::Nodes::Document
+# Psych.parse("---\n - a\n - b")
#
-# Psych provides a range of interfaces ranging from low to high level for
-# producing YAML documents. Very similar to the YAML parsing interfaces, Psych
-# provides at the lowest level, an event based system, mid-level is building
-# a YAML AST, and the highest level is converting a Ruby object straight to
-# a YAML document.
+# ==== Reading from a file
#
-# === Low level emitting
+# # Returns Psych::Nodes::Stream
+# Psych.parse_stream(File.read('database.yml'))
#
-# The lowest level emitter is an event based system. Events are sent to a
-# Psych::Emitter object. That object knows how to convert the events to a YAML
-# document. This interface should be used when document format is known in
-# advance or speed is a concern. See Psych::Emitter for more information.
+# # Returns Psych::Nodes::Document
+# Psych.parse_file('database.yml')
+#
+# ==== Exception handling
#
-# === Mid level emitting
+# begin
+# # The second argument chnages only the exception contents
+# Psych.parse("--- `", "file.txt")
+# rescue Psych::SyntaxError => ex
+# ex.file # => 'file.txt'
+# ex.message # => "(file.txt): found character that cannot start any token"
+# end
+#
+# === Emitting
#
# At the mid level is building an AST. This AST is exactly the same as the AST
# used when parsing a YAML document. Users can build an AST by hand and the
@@ -87,18 +150,76 @@
# Psych::Nodes::Node, and Psych::TreeBuilder for more information on building
# a YAML AST.
#
-# === High level emitting
+# ==== Writing to a string
#
-# The high level emitter has the easiest interface. Psych simply takes a Ruby
-# data structure and converts it to a YAML document. See Psych.dump for more
-# information on dumping a Ruby data structure.
+# # We need Psych::Nodes::Stream (not Psych::Nodes::Document)
+# stream = Psych.parse_stream("---\n - a\n - b")
+#
+# stream.to_yaml # => "---\n- a\n- b\n"
+#
+# ==== Writing to a file
+#
+# # We need Psych::Nodes::Stream (not Psych::Nodes::Document)
+# stream = Psych.parse_stream(File.read('database.yml'))
+#
+# File.open('database.yml', 'w') do |file|
+# file.write(stream.to_yaml)
+# end
+#
+# == Low-level API
+#
+# === Parsing
+#
+# The lowest level parser should be used when the YAML input is already known,
+# and the developer does not want to pay the price of building an AST or
+# automatic detection and conversion to ruby objects. See Psych::Parser for
+# more information on using the event based parser.
+#
+# ==== Reading to Psych::Nodes::Stream structure
+#
+# parser = Psych::Parser.new(TreeBuilder.new) # => #<Psych::Parser>
+# parser = Psych.parser # it's an alias for the above
+#
+# parser.parse("---\n - a\n - b") # => #<Psych::Parser>
+# parser.handler # => #<Psych::TreeBuilder>
+# parser.handler.root # => #<Psych::Nodes::Stream>
+#
+# ==== Receiving an events stream
+#
+# parser = Psych::Parser.new(Psych::Handlers::Recorder.new)
+#
+# parser.parse("---\n - a\n - b")
+# parser.events # => [list of [event, args] lists]
+# # event is one of: Psych::Handler::EVENTS
+# # args are the arguments passed to the event
+#
+# === Emitting
+#
+# The lowest level emitter is an event based system. Events are sent to a
+# Psych::Emitter object. That object knows how to convert the events to a YAML
+# document. This interface should be used when document format is known in
+# advance or speed is a concern. See Psych::Emitter for more information.
+#
+# ==== Writing to a ruby structure
+#
+# Psych.parser.parse("--- a") # => #<Psych::Parser>
+#
+# parser.handler.first # => #<Psych::Nodes::Stream>
+# parser.handler.first.to_ruby # => ["a"]
+#
+# parser.handler.root.first # => #<Psych::Nodes::Document>
+# parser.handler.root.first.to_ruby # => "a"
+#
+# # You can instantiate an Emitter manually
+# Psych::Visitors::ToRuby.new.accept(parser.handler.root.first)
+# # => "a"
module Psych
# The version is Psych you're using
- VERSION = '1.3.4'
+ VERSION = '2.0.0'
# The version of libyaml Psych is using
- LIBYAML_VERSION = Psych.libyaml_version.join '.' unless RUBY_ENGINE == 'jruby'
+ LIBYAML_VERSION = Psych.libyaml_version.join '.'
class Exception < RuntimeError
end
@@ -123,7 +244,7 @@ class BadAlias < Exception
# Psych.load("--- `", "file.txt")
# rescue Psych::SyntaxError => ex
# ex.file # => 'file.txt'
- # ex.message # => "(foo.txt): found character that cannot start any token"
+ # ex.message # => "(file.txt): found character that cannot start any token"
# end
def self.load yaml, filename = nil
result = parse(yaml, filename)
@@ -131,7 +252,7 @@ def self.load yaml, filename = nil
end
###
- # Parse a YAML string in +yaml+. Returns the first object of a YAML AST.
+ # Parse a YAML string in +yaml+. Returns the Psych::Nodes::Document.
# +filename+ is used in the exception message if a Psych::SyntaxError is
# raised.
#
@@ -139,13 +260,13 @@ def self.load yaml, filename = nil
#
# Example:
#
- # Psych.parse("---\n - a\n - b") # => #<Psych::Nodes::Sequence:0x00>
+ # Psych.parse("---\n - a\n - b") # => #<Psych::Nodes::Document:0x00>
#
# begin
# Psych.parse("--- `", "file.txt")
# rescue Psych::SyntaxError => ex
# ex.file # => 'file.txt'
- # ex.message # => "(foo.txt): found character that cannot start any token"
+ # ex.message # => "(file.txt): found character that cannot start any token"
# end
#
# See Psych::Nodes for more information about YAML AST.
@@ -157,7 +278,7 @@ def self.parse yaml, filename = nil
end
###
- # Parse a file at +filename+. Returns the YAML AST.
+ # Parse a file at +filename+. Returns the Psych::Nodes::Document.
#
# Raises a Psych::SyntaxError when a YAML syntax error is detected.
def self.parse_file filename
@@ -173,7 +294,7 @@ def self.parser
end
###
- # Parse a YAML string in +yaml+. Returns the full AST for the YAML document.
+ # Parse a YAML string in +yaml+. Returns the Psych::Nodes::Stream.
# This method can handle multiple YAML documents contained in +yaml+.
# +filename+ is used in the exception message if a Psych::SyntaxError is
# raised.
@@ -195,7 +316,7 @@ def self.parser
# Psych.parse_stream("--- `", "file.txt")
# rescue Psych::SyntaxError => ex
# ex.file # => 'file.txt'
- # ex.message # => "(foo.txt): found character that cannot start any token"
+ # ex.message # => "(file.txt): found character that cannot start any token"
# end
#
# See Psych::Nodes for more information about YAML AST.
8 lib/ruby/1.9/psych/scalar_scanner.rb
View
@@ -68,11 +68,11 @@ def tokenize string
string
end
when /^\.inf$/i
- 1 / 0.0
+ Float::INFINITY
when /^-\.inf$/i
- -1 / 0.0
+ -Float::INFINITY
when /^\.nan$/i
- 0.0 / 0.0
+ Float::NAN
when /^:./
if string =~ /^:(["'])(.*)\1/
@symbol_cache[string] = $2.sub(/^:/, '').to_sym
@@ -96,7 +96,7 @@ def tokenize string
@string_cache[string] = true
string
else
- Float(string.gsub(/[,_]/, ''))
+ Float(string.gsub(/[,_]|\.$/, ''))
end
else
int = parse_int string.gsub(/[,_]/, '')
98 lib/ruby/1.9/psych/visitors/to_ruby.rb
View
@@ -141,28 +141,6 @@ def visit_Psych_Nodes_Mapping o
return revive_hash({}, o) unless o.tag
case o.tag
- when /^!(?:str|ruby\/string)(?::(.*))?/, 'tag:yaml.org,2002:str'
- klass = resolve_class($1)
- members = Hash[*o.children.map { |c| accept c }]
- string = members.delete 'str'
-
- if klass
- string = klass.allocate.replace string
- register(o, string)
- end
-
- init_with(string, members.map { |k,v| [k.to_s.sub(/^@/, ''),v] }, o)
- when /^!ruby\/array:(.*)$/
- klass = resolve_class($1)
- list = register(o, klass.allocate)
-
- members = Hash[o.children.map { |c| accept c }.each_slice(2).to_a]
- list.replace members['internal']
-
- members['ivars'].each do |ivar, v|
- list.instance_variable_set ivar, v
- end
- list
when /^!ruby\/struct:?(.*)?$/
klass = resolve_class($1)
@@ -187,6 +165,43 @@ def visit_Psych_Nodes_Mapping o
Struct.new(*h.map { |k,v| k.to_sym }).new(*h.map { |k,v| v })
end
+ when /^!ruby\/object:?(.*)?$/
+ name = $1 || 'Object'
+
+ if name == 'Complex'
+ h = Hash[*o.children.map { |c| accept c }]
+ register o, Complex(h['real'], h['image'])
+ elsif name == 'Rational'
+ h = Hash[*o.children.map { |c| accept c }]
+ register o, Rational(h['numerator'], h['denominator'])
+ else
+ obj = revive((resolve_class(name) || Object), o)
+ obj
+ end
+
+ when /^!(?:str|ruby\/string)(?::(.*))?/, 'tag:yaml.org,2002:str'
+ klass = resolve_class($1)
+ members = Hash[*o.children.map { |c| accept c }]
+ string = members.delete 'str'
+
+ if klass
+ string = klass.allocate.replace string
+ register(o, string)
+ end
+
+ init_with(string, members.map { |k,v| [k.to_s.sub(/^@/, ''),v] }, o)
+ when /^!ruby\/array:(.*)$/
+ klass = resolve_class($1)
+ list = register(o, klass.allocate)
+
+ members = Hash[o.children.map { |c| accept c }.each_slice(2).to_a]
+ list.replace members['internal']
+
+ members['ivars'].each do |ivar, v|
+ list.instance_variable_set ivar, v
+ end
+ list
+
when '!ruby/range'
h = Hash[*o.children.map { |c| accept c }]
register o, Range.new(h['begin'], h['end'], h['excl'])
@@ -206,19 +221,6 @@ def visit_Psych_Nodes_Mapping o
end
set
- when '!ruby/object:Complex'
- h = Hash[*o.children.map { |c| accept c }]
- register o, Complex(h['real'], h['image'])
-
- when '!ruby/object:Rational'
- h = Hash[*o.children.map { |c| accept c }]
- register o, Rational(h['numerator'], h['denominator'])
-
- when /^!ruby\/object:?(.*)?$/
- name = $1 || 'Object'
- obj = revive((resolve_class(name) || Object), o)
- obj
-
when /^!map:(.*)$/, /^!ruby\/hash:(.*)$/
revive_hash resolve_class($1).new, o
@@ -261,28 +263,42 @@ def register_empty object
def revive_hash hash, o
@st[o.anchor] = hash if o.anchor
- o.children.each_slice(2) { |k,v|
+ o.children.each_slice(2) { |k,v|
key = accept(k)
+ val = accept(v)
if key == '<<'
case v
when Nodes::Alias
- hash.merge! accept(v)
+ begin
+ hash.merge! val
+ rescue TypeError
+ hash[key] = val
+ end
when Nodes::Sequence
- accept(v).reverse_each do |value|
- hash.merge! value
+ begin
+ h = {}
+ val.reverse_each do |value|
+ h.merge! value
+ end
+ hash.merge! h
+ rescue TypeError
+ hash[key] = val
end
else
- hash[key] = accept(v)
+ hash[key] = val
end
else
- hash[key] = accept(v)
+ hash[key] = val
end
}
hash
end
+ def merge_key hash, key, val
+ end
+
def revive klass, node
s = klass.allocate
@st[node.anchor] = s if node.anchor
60 lib/ruby/1.9/psych/visitors/yaml_tree.rb
View
@@ -8,6 +8,30 @@ module Visitors
# builder.tree # => #<Psych::Nodes::Stream .. }
#
class YAMLTree < Psych::Visitors::Visitor
+ class Registrar # :nodoc:
+ def initialize
+ @obj_to_id = {}
+ @obj_to_node = {}
+ @counter = 0
+ end
+
+ def register target, node
+ @obj_to_node[target.object_id] = node
+ end
+
+ def key? target
+ @obj_to_node.key? target.object_id
+ end
+
+ def id_for target
+ @obj_to_id[target.object_id] ||= (@counter += 1)
+ end
+
+ def node_for target
+ @obj_to_node[target.object_id]
+ end
+ end
+
attr_reader :started, :finished
alias :finished? :finished
alias :started? :started
@@ -17,7 +41,7 @@ def initialize options = {}, emitter = TreeBuilder.new, ss = ScalarScanner.new
@started = false
@finished = false
@emitter = emitter
- @st = {}
+ @st = Registrar.new
@ss = ss
@options = options
@coders = []
@@ -72,9 +96,9 @@ def push object
def accept target
# return any aliases we find
- if @st.key? target.object_id
- oid = target.object_id
- node = @st[oid]
+ if @st.key? target
+ oid = @st.id_for target
+ node = @st.node_for target
anchor = oid.to_s
node.anchor = anchor
return @emitter.alias anchor
@@ -147,8 +171,8 @@ def visit_Exception o
@emitter.start_mapping nil, tag, false, Nodes::Mapping::BLOCK
{
- 'message' => private_iv_get(o, 'mesg') || o.message,
- 'backtrace' => private_iv_get(o, 'backtrace') || o.backtrace,
+ 'message' => private_iv_get(o, 'mesg'),
+ 'backtrace' => private_iv_get(o, 'backtrace'),
}.each do |k,v|
next unless v
@emitter.scalar k, nil, nil, true, false, Nodes::Scalar::ANY
@@ -221,16 +245,17 @@ def visit_BigDecimal o
end
def binary? string
- string.encoding == Encoding::ASCII_8BIT ||
+ (string.encoding == Encoding::ASCII_8BIT && !string.ascii_only?) ||
string.index("\x00") ||
- string.count("\x00-\x7F", "^ -~\t\r\n").fdiv(string.length) > 0.3
+ string.count("\x00-\x7F", "^ -~\t\r\n").fdiv(string.length) > 0.3 ||
+ string.class != String
end
private :binary?
def visit_String o
- plain = false
- quote = false
- style = Nodes::Scalar::ANY
+ plain = true
+ quote = true
+ style = Nodes::Scalar::PLAIN
tag = nil
str = o
@@ -239,15 +264,14 @@ def visit_String o
tag = '!binary' # FIXME: change to below when syck is removed
#tag = 'tag:yaml.org,2002:binary'
style = Nodes::Scalar::LITERAL
+ plain = false
+ quote = false
elsif o =~ /\n/
- quote = true
style = Nodes::Scalar::LITERAL
- elsif o =~ /^\W/
- quote = true
- style = Nodes::Scalar::DOUBLE_QUOTED
else
- quote = !(String === @ss.tokenize(o))
- plain = !quote
+ unless String === @ss.tokenize(o)
+ style = Nodes::Scalar::SINGLE_QUOTED
+ end
end
ivars = find_ivars o
@@ -409,7 +433,7 @@ def find_ivars target
end
def register target, yaml_obj
- @st[target.object_id] = yaml_obj
+ @st.register target, yaml_obj
yaml_obj
end
2  lib/ruby/1.9/psych/y.rb
View
@@ -1,4 +1,6 @@
module Kernel
+ ###
+ # An alias for Psych.dump_stream meant to be used with IRB.
def y *objects
puts Psych.dump_stream(*objects)
end
Something went wrong with that request. Please try again.