Running DomainName multi-threaded leads to Stack Errors

[chainum]

I'm running Mechanize (https://github.com/tenderlove/mechanize) multi-threaded using Sidekiq (https://github.com/mperham/sidekiq).

Mechanize pulls in the DomainName-gem to handle (for example) parsing of cookie domain names. Apparently due to the large size (~4000 items) of the hash in lib/domain_name/etld_data.rb stack errors occur. This does not happen when in single-threaded mode though.

Changing the hash to be first instantiated and then adding the respective hash items seems to solve it.

The tests pass on 1.9.3-p125 and 1.9.2-p290. If necessary I can see if I can set up a sample Rails-app with Mechanize/Sidekiq so you can reproduce the issue in multi-threaded mode yourself.

[knu]

The patch is far from acceptable because eliminating caching in a class variable means a large hash table is created every time the library tries to look etld_data up.

I don't really get the problem but will guarding the initialization with a mutex do?

require 'thread'

# ...
def self.etld_data
  Thread.exclusive {
    @@etld_data ||= {
      # ...
    }
  }
end

[chainum]

Hey!

I tried to keep the caching/memoization but that didn't fix the StackError.

The problem is basically the following:

• I run mechanize in multi-threaded mode using Sidekiq.
• I post to a form using mechanize and it's cookie handling-logic uses DomainName to parse the domain of the cookie.
• When it parses the cookie DomainName.etld_data is loaded which crashes the Sidekiq-processors due to Stack Errors - they completely die.

An interesting thing is that I removed ~3k entries or so from etld_data.rb (leaving only 1k left) and somehow the Stack Errors stopped.

Using my solution (thus without the class-variable/memoization) worked for all the 4k entries though.

I'll do some more investigation later tonight and see if I can get it to work while preserving the class-variable and the cache/memoization.

[knu]

I think I'll just constify the table so the initialization won't occur in multiple threads simultaneously.