Starting a good job worker pre-selecting only certain labels

[francois]

Hey all!

In dev, I have multiple workspaces:

Projects/
  dayjob00/
  dayjob01/
  dayjob08/

Each dayjobNN is a different workspace (git worktree, jj workspace) where I will run both a web server process and a GoodJob worker instance.

If I'm developing a job in dayjob01, I wouldn't want the GoodJob worker in dayjob08 or 00 to pick up that job. I'm thinking of tagging all jobs with the workspace's basename to add that filtering capability, but the docs does not mention if that filter exists.

My ideal workflow would be:

good_job --queues '...' --label $( basename $PWD ) # only picks up jobs tagged with `basename $PWD`

I see there is already an "index_good_jobs_on_labels" gin (labels) WHERE labels IS NOT NULL. That would help with performance.

@bensheldon, would you be open to a PR that adds such a feature?

Related, can I add a label in a before_enqueue?

class ApplicationJob < ActiveJob::Base
  # ...
  before_enqueue :tag_with_workspace_name if Rails.env.development?

  private

  def tag_with_workspace_name
    self.good_job_labels ||= []
    self.good_job_labels << Rails.root.basename
  end
end

I haven't tried it, but I guess this should work?

Thanks a bunch!
François

[bensheldon]

It's an interesting problem.

Specifically for worktrees, I think you should be completely isolating your databases. Here's how I add a worktree to my database name: https://gist.github.com/bensheldon/bda0d1d6afc0dba3d389b0c1aee8e324

Is that possible given your setup?

[francois]

Yeah, not really. In order to test against real data, our database dump expands to 60 GB. I can’t copy that around for each new worktree. I understand the reasoning and I would love to do that, but it’s just not realistic to copy 60 GB of data on start. We have fully isolated test databases, and that works great. I even have parallel databases per workspace: dayjob00_00, dayjob00_01, dayjob01_00, … I will most probably fork and add that feature for ourselves. If you want to merge that later, fine. If you don’t, we can keep our fork up-to-date. Thanks, François

[bensheldon]

Active Job has a queue_name_prefix configuration that I think is maybe more 1:1 with your problem.

I still mentally categorize this as "oh no, this is totally gonna be abused" but... what about allowing passing a wildcard to ---queues=your_prefix_* ?

[francois]

I didn’t know about queue_name_prefix! I’ll go ahead and investigate that. The docs aren’t explicit on whether `—queues "dayjob01_*:0"` is allowed or not. Is that the feature you’re recommending be added instead?

[bensheldon]

Is that the feature you’re recommending be added instead?

yes, that would be a new feature to expand the queue syntax to add a wildcard and then passing that down to the job query (I imagine changing the query to a Postgres regex match or something if there's a wildcard...though I wouldn't want the GoodJob query syntax to allow a full regex.

I imagine it will be terrible for performance, but work for development/test challenges like yours.

[francois]

Would you want to support both *_slo60m and dayjob00_* ? So, both prefix and suffix?

[bensheldon]

I think so 🤷

if it's using a Postgres regex, as long as we differentiate between "*" and anything else I even could see "*foo_*bar*" being valid because (as I'm imagining) it would be replacing any * with ".*" in the constructed regex for Postgres.