Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

AWS SDK segmentation fault in Encoding#find_non_ascii_index #3461

Closed
yorickpeterse opened this issue Jul 16, 2015 · 5 comments
Closed

AWS SDK segmentation fault in Encoding#find_non_ascii_index #3461

yorickpeterse opened this issue Jul 16, 2015 · 5 comments

Comments

@yorickpeterse
Copy link
Contributor

Logs: https://gist.github.com/YorickPeterse/0d5cfef61583eccbaf7b

Code that triggered it:

require 'thread'
require 'sequel'
require 'progress_bar'
require 'aws-sdk'
require 'json'
require 'oga'
require 'securerandom'

Thread.abort_on_exception = true

puts 'Connecting...'

DB = Sequel.connect(
  :adapter         => 'postgres',
  :encoding        => 'UTF8',
  :test            => true,
  :host            => ENV['DB_HOST'],
  :database        => ENV['DB_NAME'],
  :user            => ENV['DB_USER'],
  :password        => ENV['DB_PASSWORD'],
  :max_connections => 60
)

queue_url = 'https://this-is-kinda-private'

reviews = DB[:reviews]
  .where('reviewer_origin IS NOT NULL OR travel_type IS NOT NULL or travel_composition IS NOT NULL')
  .count

last_id  = 0
iterated = 0
queue    = Queue.new
output   = Queue.new
control  = Queue.new
progress = ProgressBar.new(reviews)
sqs_lock = Mutex.new

puts 'Starting threads...'

threads = 10.times.map do
  Thread.new do
    thread_sqs = Aws::SQS::Client.new

    loop do
      entries = queue.pop.map do |review|
        {:id => SecureRandom.hex, :message_body => JSON.dump(review)}
      end

      #sqs_lock.synchronize do
        thread_sqs.send_message_batch(:queue_url => queue_url, :entries => entries)
      #end

      output << entries.length
    end
  end
end

filler = Thread.new do
  while iterated < reviews
    reviews_page = DB[:reviews]
      .select(:id, :reviewer_origin, :travel_type, :travel_composition)
      .where('reviewer_origin IS NOT NULL OR travel_type IS NOT NULL or travel_composition IS NOT NULL')
      .where { id > last_id }
      .order(:id)
      .limit(5_000)
      .to_a

    reviews_page.each_slice(10) do |slice|
      queue << slice
    end

    last_id = reviews_page.last[:id]
  end

  control << :terminate
end

loop do
  terminate = !control.empty? && control.pop == :terminate

  break if terminate && queue.empty? && output.empty?

  progress.increment!(output.pop)
end

filler.join

Not sure how to actually reproduce this outside of using the above script.

@yorickpeterse
Copy link
Contributor Author

Managed to get my hands on a GDB trace for this error: https://gist.github.com/YorickPeterse/df7386bdd4c07d3fb979

@yorickpeterse
Copy link
Contributor Author

Standalone script to reproduce this (and the other crashes I reported earlier today):

EDIT: the code is in the following gist, which aids readability of the issue and collaboration on the code. -- @brixen

https://gist.github.com/brixen/bc9e2a88338439bee855

@headius For JRuby 9k you could take a look at this script (as we discussed yesterday). I haven't had a chance yet to run it on 9k but it's pretty much a 1:1 port of the original script but using dummy data instead of a production database.

@headius
Copy link
Contributor

headius commented Jul 19, 2015

@yorickpeterse Thanks! I'll run it through its paces and add an issue to JRuby's tracker.

@headius
Copy link
Contributor

headius commented Jul 19, 2015

@yorickpeterse Filed: jruby/jruby#3153

@brixen
Copy link
Member

brixen commented Jan 4, 2020

Much of the internals of Rubinius have been completely or mostly rewritten in the past couple years. This includes the garbage collector, concurrency facilities, Fibers, much of the instruction set, and a migration away from "primitive" functions that implement Ruby features.

Since a number of segfaults or process hangs have occurred in these features over time, this issue may be fixed.

The focus for Rubinius in the near term is on the following capabilities:

  1. Instruction set
  2. Debugger
  3. Profiler
  4. Just-in-time compiler
  5. Concurrency
  6. Garbage collector

Contributions in the form of PRs for any of the areas of focus above are appreciated. Once these capabilities are more robust, it will be possible to more efficiently debug and fix any process crashes.

Other than these core capabilities, PRs to fix any specific issue are always welcome.

@brixen brixen closed this as completed Jan 4, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants