Recursive hashing corrupts shared hash buffer

[notEthan]

I started memoizing the #hash method of a class (Ptr, below), which seems to have broken comparison of a Set containing instances of a class (Node) that has an ivar containing a Ptr, created in Node's initializer.

require 'set'

class Ptr
  def ==(other)
    other.is_a?(Ptr)
  end

  alias_method :eql?, :==

  def hash
    h = @hash ||= {x: 0}.hash
    puts "Ptr#hash:  #{h}"
    h
  end
end

class Node
  def initialize
    @ptr = Ptr.new
    @node_set = Set[self]
  end

  attr_reader :node_set

  def ==(other)
    other.is_a?(Node)
  end

  alias_method :eql?, :==

  def hash
    h = {ptr: @ptr}.hash
    puts "Node#hash: #{h}"
    h
  end
end

node = Node.new
post_node_set = Set[node]

puts
puts "post_node_set == node.node_set:"
puts  post_node_set == node.node_set
puts
puts "post_node_set.map(&:hash) == node.node_set.map(&:hash):"
puts  post_node_set.map(&:hash) == node.node_set.map(&:hash)
puts
puts "node.node_set == post_node_set:"
puts  node.node_set == post_node_set

Expected: all == comparisons result in true
Actual: first comparison is false (and the rest are true)

Some confluence of factors is at play here, I've simplified this down a lot, but simplifying further seems to no longer trigger the issue. Doing less in Node#initialize, or if Ptr#hash or Node#hash are hashing something simpler than Hash instances, result in expected behavior.

Output of the above, with notes:

Ptr#hash:  6413925280117835575
Node#hash: 8466601984141238220 ⬅️ from inside Node#initialize
Ptr#hash:  6413925280117835575
Node#hash: 3645463223149969145 ⬅️ after Node#initialize, should be same as from inside Node#initialize

post_node_set == node.node_set:
Ptr#hash:  6413925280117835575
Node#hash: 3645463223149969145
false                          ⬅️ expected: true

post_node_set.map(&:hash) == node.node_set.map(&:hash):
Ptr#hash:  6413925280117835575
Node#hash: 3645463223149969145
Ptr#hash:  6413925280117835575
Node#hash: 3645463223149969145
true

node.node_set == post_node_set:
Ptr#hash:  6413925280117835575
Node#hash: 3645463223149969145
true

Environment Information

• JRuby version 9.4.3.0
• Operating system: reproduces on all of ubuntu-latest, macos-latest, windows-latest on github actions
• Test runs at https://github.com/notEthan/ghtest/actions/runs/5627207157

[headius]

post_node_set == node.node_set

So this is the only one that fails, yes? I have not found a cause yet but it seems like the problem must revolve around Node not hashing the same as you expect. I believe Set is working properly but some interaction with your overridden methods might be confusing the underlying Hash.

[headius]

Removing the memoization or calling @ptr.hash in Node#initialize both make it work as expected. Problem seems to be something to do with the memoization.

[headius]

Here's the smallest repro I have that still shows the issue:

Edit: I pasted the wrong source at first.

require 'set'

class Ptr
  def hash
    @hash ||= {x:1}.hash
  end
end

class Node
  def initialize
    @ptr = Ptr.new
  end

  def hash
    {ptr: @ptr}.hash
  end
end

node = Node.new
set1 = Set[node]
set2 = Set[node]
puts set1 == set2
puts set2 == set1

There's some interaction between Set's logic for standing up the internal hash and the memoized Ptr#hash value.

[headius]

I have tested whether something is wrong with our Set by forcing it to use the gem, which is the same pure-Ruby set.rb that CRuby uses. It did not change the result, so it seems the issue is in JRuby's Hash or in how we are compiling this code.

[headius]

New smaller reproduction that eliminates Set and Node. It's clearly not a problem with Set.

class Ptr
  def hash
    @hash ||= { x: 1 }.hash
  end
end

ptr = Ptr.new
set1 = { { ptr: ptr } => true}
set2 = { { ptr: ptr } => true}
puts set1 == set2
puts set2 == set1

[headius]

Even smaller!

class Ptr
  def hash
    @hash ||= { x: 1 }.hash
  end
end

ptr = Ptr.new
puts({ ptr: ptr }.hash, { ptr: ptr }.hash)

[headius]

OMG I found it.

In JRuby 9.4.0.0, 60ae8ba introduced a cache for the byte[16] used to calculate an aggregate hash based on the key and value, but none of us noticed that the buffer was in-use during recursive hash calls. That led to those recursive calls wiping out the partial results already present.

I've fixed it in #7901.