feature: Add GoodJob adapter (#116)

* Copy delayed_job and rename

* rename files and folders

* Add GoodJob sample app and implement collector logic

* Update README.md

* Rename variables

* Find all known queues

* Fix CI

* Fix CI

* Query good_jobs table directly instead of using JobsFilter

* Rename variable

* Add finished_at for index preference

* Fix syntax for Ruby 2.6

* Add busy jobs reporting for GoodJob

* Silence query logs for GoodJob collector

* Move query for "known queue names" to initializer

* Change mock API endpoint for sample apps

.github/workflows/judoscale-good_job-test.yml

@@ -0,0 +1,46 @@
+name: judoscale-good_job tests
+defaults:
+  run:
+    working-directory: judoscale-good_job
+on:
+  push:
+    branches:
+      - main
+  pull_request:
+jobs:
+  test:
+    strategy:
+      fail-fast: false
+      matrix:
+        gemfile:
+          - Gemfile
+          - Gemfile-activerecord-6-1
+        ruby:
+          - "2.6"
+          - "2.7"
+          - "3.0"
+          - "3.1"
+        exclude:
+          - gemfile: Gemfile
+            ruby: "2.6"
+    runs-on: ubuntu-latest
+    env: # $BUNDLE_GEMFILE must be set at the job level, so it is set for all steps
+      BUNDLE_GEMFILE: ${{ github.workspace }}/judoscale-good_job/${{ matrix.gemfile }}
+    services:
+      db:
+        image: postgres:latest
+        env:
+          POSTGRES_HOST_AUTH_METHOD: trust
+        ports: ["5432:5432"]
+        options: >-
+          --health-cmd pg_isready
+          --health-interval 10s
+          --health-timeout 5s
+          --health-retries 5
+    steps:
+      - uses: actions/checkout@v2
+      - uses: ruby/setup-ruby@v1
+        with:
+          ruby-version: ${{ matrix.ruby }}
+          bundler-cache: true # runs bundle install and caches installed gems automatically
+      - run: bundle exec rake

README.md

@@ -3,6 +3,7 @@
 [![Build Status: judoscale-ruby](https://github.com/judoscale/judoscale-ruby/actions/workflows/judoscale-ruby-test.yml/badge.svg)](https://github.com/judoscale/judoscale-ruby/actions)
 [![Build Status: judoscale-rails](https://github.com/judoscale/judoscale-ruby/actions/workflows/judoscale-rails-test.yml/badge.svg)](https://github.com/judoscale/judoscale-ruby/actions)
 [![Build Status: judoscale-delayed_job](https://github.com/judoscale/judoscale-ruby/actions/workflows/judoscale-delayed_job-test.yml/badge.svg)](https://github.com/judoscale/judoscale-ruby/actions)
+[![Build Status: judoscale-good_job](https://github.com/judoscale/judoscale-ruby/actions/workflows/judoscale-good_job-test.yml/badge.svg)](https://github.com/judoscale/judoscale-ruby/actions)
 [![Build Status: judoscale-que](https://github.com/judoscale/judoscale-ruby/actions/workflows/judoscale-que-test.yml/badge.svg)](https://github.com/judoscale/judoscale-ruby/actions)
 [![Build Status: judoscale-sidekiq](https://github.com/judoscale/judoscale-ruby/actions/workflows/judoscale-sidekiq-test.yml/badge.svg)](https://github.com/judoscale/judoscale-ruby/actions)
 
@@ -24,10 +25,11 @@ gem "judoscale-rails"
 # gem "judoscale-sidekiq"
 # gem "judoscale-resque"
 # gem "judoscale-delayed_job"
+# gem "judoscale-good_job"
 # gem "judoscale-que"
 ```
 
-_If you're using a background job queue like Sidekiq or Delayed Job, make sure you include the corresponding judoscale-\* gem as well._
+_If you're using a background job queue, make sure you include the corresponding judoscale-\* gem as well._
 
 The adapters report queue metrics to Judoscale every 10 seconds. The reporter will not run in development, or any other environment missing the `JUDOSCALE_URL` environment variable. (This environment variable is set for you automatically when provisioning the add-on.)
 
@@ -50,12 +52,13 @@ The middleware will start the async reporter when it processes the first request
 
 ## Worker adapters
 
-Judoscale will autoscale your worker dynos! Four job backends are supported: Sidekiq, Delayed Job, and Que. Be sure to install the gem specific to your job backend:
+Judoscale will autoscale your worker dynos! Four job backends are supported: Sidekiq, Delayed Job, Good Job, and Que. Be sure to install the gem specific to your job backend:
 
 ```ruby
 gem "judoscale-sidekiq"
 gem "judoscale-resque"
 gem "judoscale-delayed_job"
+gem "judoscale-good_job"
 gem "judoscale-que"
 ```
 
@@ -86,7 +89,7 @@ Judoscale aggregates and stores this information to power the autoscaler algorit
 
 ## Migrating from `rails_autoscale_agent`
 
-The migration from `rails_autoscale_agent` to `judoscale-rails` (+ your job framework gem) is typically a single step: replace the `gem "rails_autoscale_agent"` in your Gemfile with `gem "judoscale-rails"` _and_ the appropriate `judoscale-*` package for your back-end job framework (`sidekiq`, `resque`, `delayed_job`, or `que`) or see the [Installation](#installation) section above for further specifics. If you previously had any custom configuration for the `rails_autoscale_agent`, please note that we now use a `configure` block as shown below.
+The migration from `rails_autoscale_agent` to `judoscale-rails` (+ your job framework gem) is typically a single step: replace the `gem "rails_autoscale_agent"` in your Gemfile with `gem "judoscale-rails"` _and_ the appropriate `judoscale-*` package for your back-end job framework (`sidekiq`, `resque`, `delayed_job`, `good_job`, or `que`) or see the [Installation](#installation) section above for further specifics. If you previously had any custom configuration for the `rails_autoscale_agent`, please note that we now use a `configure` block as shown below.
 
 Looking for the old `rails_autoscale_agent` docs? They're available on [this branch](https://github.com/judoscale/judoscale-ruby/tree/rails_autoscale_agent).
 

judoscale-good_job/Gemfile

@@ -0,0 +1,10 @@
+source "https://rubygems.org"
+
+gemspec name: "judoscale-good_job"
+
+gem "judoscale-ruby", path: "../judoscale-ruby"
+gem "activerecord"
+gem "pg"
+gem "minitest"
+gem "rake"
+gem "rake-release"

judoscale-good_job/Gemfile-activerecord-6-1

@@ -0,0 +1,10 @@
+source "https://rubygems.org"
+
+gemspec name: "judoscale-good_job"
+
+gem "judoscale-ruby", path: "../judoscale-ruby"
+gem "activerecord", "~> 6.1"
+gem "pg"
+gem "minitest"
+gem "rake"
+gem "rake-release"

judoscale-good_job/Gemfile.lock

@@ -0,0 +1,113 @@
+PATH
+  remote: ../judoscale-ruby
+  specs:
+    judoscale-ruby (1.2.3)
+
+PATH
+  remote: .
+  specs:
+    judoscale-good_job (1.2.3)
+      good_job (>= 3.0)
+      judoscale-ruby (= 1.2.3)
+
+GEM
+  remote: https://rubygems.org/
+  specs:
+    actionpack (7.0.4.1)
+      actionview (= 7.0.4.1)
+      activesupport (= 7.0.4.1)
+      rack (~> 2.0, >= 2.2.0)
+      rack-test (>= 0.6.3)
+      rails-dom-testing (~> 2.0)
+      rails-html-sanitizer (~> 1.0, >= 1.2.0)
+    actionview (7.0.4.1)
+      activesupport (= 7.0.4.1)
+      builder (~> 3.1)
+      erubi (~> 1.4)
+      rails-dom-testing (~> 2.0)
+      rails-html-sanitizer (~> 1.1, >= 1.2.0)
+    activejob (7.0.4.1)
+      activesupport (= 7.0.4.1)
+      globalid (>= 0.3.6)
+    activemodel (7.0.4.1)
+      activesupport (= 7.0.4.1)
+    activerecord (7.0.4.1)
+      activemodel (= 7.0.4.1)
+      activesupport (= 7.0.4.1)
+    activesupport (7.0.4.1)
+      concurrent-ruby (~> 1.0, >= 1.0.2)
+      i18n (>= 1.6, < 2)
+      minitest (>= 5.1)
+      tzinfo (~> 2.0)
+    builder (3.2.4)
+    concurrent-ruby (1.1.10)
+    crass (1.0.6)
+    erubi (1.12.0)
+    et-orbi (1.2.7)
+      tzinfo
+    fugit (1.8.0)
+      et-orbi (~> 1, >= 1.2.7)
+      raabro (~> 1.4)
+    globalid (1.0.0)
+      activesupport (>= 5.0)
+    good_job (3.7.4)
+      activejob (>= 6.0.0)
+      activerecord (>= 6.0.0)
+      concurrent-ruby (>= 1.0.2)
+      fugit (>= 1.1)
+      railties (>= 6.0.0)
+      thor (>= 0.14.1)
+      webrick (>= 1.3)
+    i18n (1.12.0)
+      concurrent-ruby (~> 1.0)
+    loofah (2.19.1)
+      crass (~> 1.0.2)
+      nokogiri (>= 1.5.9)
+    method_source (1.0.0)
+    minitest (5.17.0)
+    nokogiri (1.14.0-arm64-darwin)
+      racc (~> 1.4)
+    nokogiri (1.14.0-x86_64-linux)
+      racc (~> 1.4)
+    pg (1.4.5)
+    raabro (1.4.0)
+    racc (1.6.2)
+    rack (2.2.6)
+    rack-test (2.0.2)
+      rack (>= 1.3)
+    rails-dom-testing (2.0.3)
+      activesupport (>= 4.2.0)
+      nokogiri (>= 1.6)
+    rails-html-sanitizer (1.4.4)
+      loofah (~> 2.19, >= 2.19.1)
+    railties (7.0.4.1)
+      actionpack (= 7.0.4.1)
+      activesupport (= 7.0.4.1)
+      method_source
+      rake (>= 12.2)
+      thor (~> 1.0)
+      zeitwerk (~> 2.5)
+    rake (13.0.6)
+    rake-release (1.3.0)
+      bundler (>= 1.11, < 3)
+    thor (1.2.1)
+    tzinfo (2.0.5)
+      concurrent-ruby (~> 1.0)
+    webrick (1.7.0)
+    zeitwerk (2.6.6)
+
+PLATFORMS
+  arm64-darwin-21
+  x86_64-linux
+
+DEPENDENCIES
+  activerecord
+  judoscale-good_job!
+  judoscale-ruby!
+  minitest
+  pg
+  rake
+  rake-release
+
+BUNDLED WITH
+   2.3.9

judoscale-good_job/Rakefile

@@ -0,0 +1,12 @@
+# frozen_string_literal: true
+
+require "rake/release"
+require "rake/testtask"
+
+Rake::TestTask.new(:test) do |t|
+  t.libs << "lib"
+  t.libs << "test"
+  t.test_files = FileList["test/**/*_test.rb"]
+end
+
+task default: :test

judoscale-good_job/judoscale-good_job.gemspec

@@ -0,0 +1,30 @@
+lib = File.expand_path("../lib", __FILE__)
+$LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
+require "judoscale/good_job/version"
+
+Gem::Specification.new do |spec|
+  spec.name = "judoscale-good_job"
+  spec.version = Judoscale::GoodJob::VERSION
+  spec.authors = ["Adam McCrea", "Carlos Antonio da Silva"]
+  spec.email = ["adam@adamlogic.com"]
+
+  spec.summary = "This gem provides GoodJob integration with the Judoscale autoscaling add-on for Heroku."
+  spec.homepage = "https://judoscale.com"
+  spec.license = "MIT"
+
+  spec.metadata = {
+    "homepage_uri" => "https://judoscale.com",
+    "bug_tracker_uri" => "https://github.com/judoscale/judoscale-ruby/issues",
+    "documentation_uri" => "https://judoscale.com/docs",
+    "changelog_uri" => "https://github.com/judoscale/judoscale-ruby/blob/main/CHANGELOG.md",
+    "source_code_uri" => "https://github.com/judoscale/judoscale-ruby"
+  }
+
+  spec.files = `git ls-files -z`.split("\x0").reject { |f| f.match(%r{^(test|spec|features)/}) }
+  spec.require_paths = ["lib"]
+
+  spec.required_ruby_version = ">= 2.6.0"
+
+  spec.add_dependency "judoscale-ruby", Judoscale::GoodJob::VERSION
+  spec.add_dependency "good_job", ">= 3.0"
+end

judoscale-good_job/lib/judoscale-good_job.rb

@@ -0,0 +1,3 @@
+# frozen_string_literal: true
+
+require "judoscale/good_job"

judoscale-good_job/lib/judoscale/good_job.rb

@@ -0,0 +1,19 @@
+# frozen_string_literal: true
+
+# require "good_job" fails unless we require these first
+require "rails"
+require "active_support/core_ext/numeric/time"
+require "active_job/railtie"
+require "good_job"
+require "judoscale-ruby"
+require "judoscale/config"
+require "judoscale/good_job/version"
+require "judoscale/good_job/metrics_collector"
+
+Judoscale.add_adapter :"judoscale-good_job",
+  {
+    adapter_version: Judoscale::GoodJob::VERSION,
+    framework_version: ::GoodJob::VERSION
+  },
+  metrics_collector: Judoscale::GoodJob::MetricsCollector,
+  expose_config: Judoscale::Config::JobAdapterConfig.new(:good_job)

judoscale-good_job/lib/judoscale/good_job/metrics_collector.rb

@@ -0,0 +1,66 @@
+# frozen_string_literal: true
+
+require "judoscale/job_metrics_collector"
+require "judoscale/job_metrics_collector/active_record_helper"
+require "judoscale/metric"
+
+module Judoscale
+  module GoodJob
+    class MetricsCollector < Judoscale::JobMetricsCollector
+      include ActiveRecordHelper
+
+      def self.adapter_config
+        Judoscale::Config.instance.good_job
+      end
+
+      def initialize
+        super
+
+        queue_names = run_silently do
+          ::GoodJob::Execution.select("distinct queue_name").map(&:queue_name)
+        end
+        self.queues |= queue_names
+      end
+
+      def collect
+        metrics = []
+        time = Time.now.utc
+
+        # logically we don't need the finished_at condition, but it lets postgres use the indexes
+        oldest_execution_time_by_queue = run_silently do
+          ::GoodJob::Execution
+            .where(performed_at: nil, finished_at: nil)
+            .group(:queue_name)
+            .pluck(:queue_name, Arel.sql("min(coalesce(scheduled_at, created_at))"))
+            .to_h
+        end
+        self.queues |= oldest_execution_time_by_queue.keys
+
+        if track_busy_jobs?
+          busy_count_by_queue = run_silently do
+            ::GoodJob::Execution.running.group(:queue_name).count
+          end
+          self.queues |= busy_count_by_queue.keys
+        end
+
+        queues.each do |queue|
+          run_at = oldest_execution_time_by_queue[queue]
+          # DateTime.parse assumes a UTC string
+          run_at = DateTime.parse(run_at) if run_at.is_a?(String)
+          latency_ms = run_at ? ((time - run_at) * 1000).ceil : 0
+          latency_ms = 0 if latency_ms < 0
+
+          metrics.push Metric.new(:qt, latency_ms, time, queue)
+
+          if track_busy_jobs?
+            busy_count = busy_count_by_queue[queue] || 0
+            metrics.push Metric.new(:busy, busy_count, Time.now, queue)
+          end
+        end
+
+        log_collection(metrics)
+        metrics
+      end
+    end
+  end
+end

judoscale-good_job/lib/judoscale/good_job/version.rb

@@ -0,0 +1,7 @@
+# frozen_string_literal: true
+
+module Judoscale
+  module GoodJob
+    VERSION = "1.2.3"
+  end
+end

judoscale-good_job/lib/rails-autoscale-good_job.rb

@@ -0,0 +1,3 @@
+# frozen_string_literal: true
+
+require "judoscale/good_job"