Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Future enhancements to the guide #1

37 tasks
pirj opened this issue Sep 14, 2019 · 1 comment
37 tasks

Future enhancements to the guide #1

pirj opened this issue Sep 14, 2019 · 1 comment
enhancement New feature or request good first issue Good for newcomers help wanted Extra attention is needed


Copy link

pirj commented Sep 14, 2019

This is a bag of thoughts issue, I encourage everyone to file more specific detailed issues and send pull requests.

  • do not use keyword arguments in plain Sidekiq workers Sidekiq does not support keyword arguments sidekiq/sidekiq#2372 (check if it works or not with ActiveJob)

  • do not schedule jobs inside a transaction if a record created/updated inside this transaction will be directly or indirectly accessed by a job: this comes with a risk of not finding a record, or getting a record with non-updated data in case job is performed before the transaction is committed. Use Sidekiq::Postpone.wrap around each transaction do

  • can be used to detect jobs scheduled from inside a transaction

  • avoid setting the queue through sidekiq_options

  • explain the difference between bucket and window
    [ ] warn on the use
    [ ] make sure examples are right

  • extend default limiter backoff.
    it's in minutes, this is not so effective if you want to run jobs ASAP (seconds), as it re-schedules,
    e.g. puts jobs after all other jobs. Do not set it to too low (e.g. 1s), otherwise overhead will start kicking in

    [ ] It'd be nice if there was more concrete argumentation for when you need the commercial features

  • limiter policy: :skip not to raise

  • write about lack of concurrency on cpu-intensive jobs

  • think of a better way of dealing with limiter backoff, keeping all of them in a single file is weird (#default-limiter-backoff guideline)
    mention that in Sidekiq-ent 2.0 it can be per-limiter

  • race condition in Business Logic in Jobs (stale data, the job should not be performed) - same problem without transactions, and also with, without locking)

  • avoid using unique jobs?
    it's mentioned sidekiq/sidekiq@40cc629#diff-0469ba29524ee01a49be37ff05e1f499R12

  • precaution - don't use queue settings in sidekiq_options

  • serialization: only simple objects or what ActiveJob supports. custom serializers

  • what happens if the record is removed and can't be deserialized. use discard_on ActiveRecord::RecordNotFounddiscard_on ActiveJob::DeserializationError in ApplicationJob?
    [ ] how does discard_on work with batches

  • adopt the style suggested by Dan Allen - have separate good and bad code examples with explanations in their captions.

  • ? ALLOW_CONCURRENCY=true vs RAILS_ENV=production in local testing, nested concurrent limiters misbehave, sometimes neither works. prefer to use EAGER_LOAD=true?

  • emphasize on the distinction between jobs that don't need automatic retries and ephemeral jobs
    The retry property can be set on a worker or specific job to disable retries completely (job goes straight to Dead) or disable death (the failed job is simply discarded). If your Failed count is increasing but you don't see anything in the Retry or Dead tabs, it's likely you've disabled one or both of those:
    class SomeWorker
    # will be completely ephemeral, not in Retry or Dead
    sidekiq_options retry: false
    # will go immediately to the Dead tab upon the first failure
    sidekiq_options retry: 0

  • mention limiters for a long-running one-off in addition to batches

  • improve limiter backoff guideline and provide an example of how:

  • a limiter can be shared across jobs

  • a limiter can be locally specified (injected?)

  • a limiter is something-specific, e.g. user_id

  • mention leaky bucket rate limiter

  • ? set lock_timeout to longer than average job execution time to prevent premature lock release
    lock_timeout option ensures a crashed Ruby process does not hold a lock forever. You must ensure that your operations take less than this number of seconds
    The same Limiter (based on the name) may be used with different lock_timeout values which allow for different blocks of code to lock on the same resource with a different lock_timeout.

  • track your errors in error tracking software

  • infra: use persistent, reliable, dedicated redis (sentinels, ...)

  • don't use Process.clock_gettime(Process::CLOCK_MONOTONIC) to share data across processes/machines
    Monotonic times are not comparable sidekiq/sidekiq#4105

  • timeout (includes Net::HTTP and libraries using it, provide a reference to its source)
    Don't ever use Ruby's Timeout module. You will get mysterious stuck or hung processes randomly.
    net:http based: httparty, restclient, open-uri

  • exception to pass by id for external ids - harmonize with cop (salesforce_external_id)

  • uncached
    +# In Rails 5, ActiveJob execution is wrapped with code reloader, which
    +# as one of its callbacks enables query cache. There are some background
    +# jobs (e.g. {MidasReportJob}) that traverse many records and do not create
    +# any. In such cases, the query cache is being filled with data that is not
    +# being used ever again and it is huge. In Rails 4.2, jobs were running
    +# without query cache and we didn't notice any slowdowns or database
    +# overloads. The disabling cache will prevent from creating such cache
    +# and will make job server use much less memory.
    class ApplicationJob < ActiveJob::Base
    extend Memoist

    • around_perform do |_job, block|
    • ActiveRecord::Base.uncached do
    • end
    • end
  • recommend MALLOC_ARENA_MAX=2

  • Draper: Active Job Integration. Active Job allows you to pass ActiveRecord objects to background tasks directly and performs the necessary serialization and deserialization.
    In order to do this, arguments to a background job must implement Global ID.
    Decorated objects implement Global ID by delegating to the object they are decorating.
    This means you can pass decorated objects to background jobs, however, the object won't be decorated when it is deserialized.

  • extend renaming jobs guideline regarding the changes to perform signatures, there can be scheduled jobs with old signature serialized

  • avoid changing mailers' signatures just like regular job's signatures

  • avoid using Redis namespaces if you keep data for several different Sidekiq instances in one Redis instance. (check if advanced Sidekiq features using Lua can interfere and even respect namespaces when used with e.g. redis-namespace)

  • when using Reliable Push, keep Redis timeouts (connection pool's network_timeout, connection pool's number of connections, redis's timeout, connect_timeout, read_timeout, write_timeout, reconnect_attempts, reconnect_delay, reconnect_delay_max) to the minimum. Doing otherwise would result in 30s+ pushes when Redis is down/degraded

  • don't rely on queue weights for job prioritization

  • make sure to add mailers to Sidekiq's configuration when using it with Active Job's deliver_later

  • use custom DeliveryJob to separate mailers between queues with different priorities

  • config: how to set up different pool sizes for Rails and Sidekiq processes (per-process RAILS_MAX_THREADS, and pool in config/database.yml for Rails and Sidekiq, ...)

@pirj pirj added enhancement New feature or request help wanted Extra attention is needed good first issue Good for newcomers labels Sep 14, 2019
@pirj pirj self-assigned this Sep 14, 2019
@pirj pirj pinned this issue Oct 15, 2019
@pirj pirj self-assigned this Jul 16, 2020
@pirj pirj mentioned this issue Nov 4, 2021
Copy link

fwolfst commented Apr 12, 2024

Misses mention of (if you enqueue in transactions, you likely want to use this)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
enhancement New feature or request good first issue Good for newcomers help wanted Extra attention is needed
None yet

No branches or pull requests

2 participants