Extended djhuey to support multiple huey instances

[SeverinAlexB]

Heey guys

We use huey with django in our production system and we need independent huey instances to process different tasks. Unfortunately the huey django plugin only supported one huey instance so I reworked the django plugin (djhuey).

What changed

• HUEY configuration supports multiple huey instances.
• 'python manage.py run_huey' has a additional parameter to indicate on which huey instance it will consume.
• This change is backward compatible. Old configurations can still be used. No code has to be changed to use these new features.
• Updated the docs.

Configuration example

HUEY = {
    'my-app': {
        'default': True,  # Indicates the default huey instance
        'backend': 'huey.backends.redis_backend',
        'connection': {'host': 'localhost', 'port': 6379},
            'consumer': {
                'workers': 4,
                'worker_type': 'process',
        }
    },
    'my-app2': {
        'backend': 'huey.backends.sqlite_backend',
        'connection': {'location': 'sqlite filename'},
            'consumer': {
                'workers': 4,
                'worker_type': 'process',
        }
    },
}

The old configuration style is still valid. It only doesn't support multiple huey instances.

run_huey example

python manage.py run_huey --queue my-app2 
python manage.py run_huey // Runs a consumer on the default huey instance

Define task example

from huey.contrib.djhuey import task

@task('my-app2')
def example_task(param):
        print(param)

@task()  # Uses the default huey instance.
def example_task2(param):
        print(param)

---

We currently use my huey branch in our production system and it is thus already quiet mature.
Don't hesitate to contact me for small changes.

[coleifer]

Wow, this is quite clean. Thanks. I will review this as soon as I can with an eye towards getting it merged into master.

[winkidney]

Great work!
Since it is a breaking change, would you like to provide a method to keep compatible of the old version?

Best Wishes

[SeverinAlexB]

@winkidney All the old configurations are still working with this pull request. It only adds a new way to use several huey instances. Thus it is not a breaking change.

[winkidney]

@Sebubu Sorry I missed it. It helps a lot : )

[SeverinAlexB]

This commit above fixes a bug in djhuey which always started the consumer with one thread.
I really don't know why this bug fix lets one test fail. The changes have nothing to do with the main huey code.

[camilonova]

Looks awesome @Sebubu 🍺

[coleifer]

Yes! It does! I'm sorry I have been so flakey about going over this. Thanks for your patience.

[Gasoid]

@Sebubu Is it stable? We are looking forward this PR))

[ilyinon]

It looks amazing!

@coleifer Could you please answer, are you planning to merge it into master?

[coleifer]

Apologies for the slowness... Could you rebase into a single commit, remove all "stylistic" (i.e. newlines/whitespace) changes, and resolve conflicts? When that's done I'm happy to review.

[SeverinAlexB]

I can make that. I'm currently busy in another project but I try to do that till the 24. february.

Maybe I need your help briefly @coleifer in order to fix one test which didn't run the last time and I had no clue why.

[coleifer]

Thanks @Sebubu - I'm sorry for letting this sit idle and accumulate conflicts. I'm happy to help with failing tests once you submit a clean/rebased patch.

[MarcoGlauser]

As @Sebubu is on holiday, I took over this PR.
@coleifer I did a rebase patch and merged your master.
But the tests are failing and i cannot figure out why because the PR only changes files in djhuey. can you point me into the right direction?

[MarcoGlauser]

after some more digging I found the culprit. The problem was that the django test runner also tried running all non django tests and failed. now that i configured the django runner only to discover djhuey test it works.

[coleifer]

Very cool, will give it a review so we can get this merged.

[coleifer]

Looks good. A few small things here and there, but I can clean them up after merging. Thanks!

[coleifer]

Clean-ups: cfdc87c

[coleifer]

@MarcoGlauser / @Sebubu -- are you all running this in production with the new queue option? There have been a couple brain-dead bugs that I think would've cropped up if you all were using this:

• "Support multiple huey instances" breaking changes  #297
• run_huey fails to run with --queue parameter #300

I don't want to revert this, but am concerned about the stability of the changes that were introduced...especially given the types of issues people are encountering.

[coleifer]

I've decided to roll this patch back and have pushed a new release, 1.9.0.

I apologize for the inconvenience this may cause you all, but I can't maintain a patch that has bugs like the two that were reported. If you want to re-submit a clean patch that addresses the reported issues and includes a more robust test-suite, I'll be happy to bring this functionality back.