BUG: Doorman runner extensibility

[mfendeksilverstripe]

Doorman runner extensibility

This is a follow up to #279

Details

• DefaultRules now uses named keys so project level configuration override is possible (typically we want to replace or remove the default rule)
• new configuration max_ticks - this limits the number of loops (disabled by default, backwards compatible)
• new configuration tick_interval - this makes the hardcoded interval of sleep(1) configurable (defaults to 1, backwards compatible)
• new configuration child_runner - this makes the hardcoded name of the dev task which is used to run the child processes configurable (defaults to ProcessJobQueueChildTask , backwards compatible)
• queue runner is now checking for maintenance lock
• exposed list of all job statuses as a new public function

Related issues

#288
#297
#296

[emteknetnz]

@mfendeksilverstripe removing the protected function getNextJobDescriptorWithoutMutex() would technically make this a breaking change, meaning we need to release this as a major version which may not happen for a long time

Just a quick recap:

• add/modify/remove privates = patch (best)
• add public/protected = minor
• delete/modify-signature public/protected = major

are you able to refactor this PR so that we could release this as either a minor or even patch?

[mfendeksilverstripe]

@emteknetnz Thanks for the heads up, I pushed the following fixes:

• restored getNextJobDescriptorWithoutMutex method for backwards compatibility
• exposed the list of all job statuses as a new public function
• added initialising_state_ttl setting
• minor code cleanup

Please review ;-)

[mfendeksilverstripe]

Pushed up a fix for job locking in the case of the job processing hitting memory or execution limits.

[mfendeksilverstripe]

@Cheddam Could you please have a look at this one?

[mfendeksilverstripe]

@maxime-rainville Could you please find some time to review this?

[nicole-ashley]

@mfendeksilverstripe Would it be possible to split out the release job lock fix into a separate PR? It's an important fix that's completely separate to Doorman and would be great to have in mainline.

[mfendeksilverstripe]

@nikrolls Sure, I can do that ;-)

[nicole-ashley]

Thanks!

[mfendeksilverstripe]

@nikrolls Job lock release changes split into a separate PR

#320

[Cheddam]

Have tested locally in a limited fashion. Setting max_ticks: 1 successfully bails on long-running tasks, but I'm not sure whether they're intended to remain at STATUS_INIT?

[mfendeksilverstripe]

@Cheddam Thanks for feedback, I've pushed up the changes you suggested. Also, the branch is now rebased.

I tested this locally and it's working for me. This is how it looks like:

You may need to run it with this this Doorman fix in place though.

[mfendeksilverstripe]

@Cheddam It's rebased now, I tested this locally and it's running well for me. All test suites are passing except one, not sure why.

What's the difference between the nightly test suite compared to others?

[Cheddam]

Nightly in this context refers to PHP 8. We've just merged #334, so this failure has been resolved - one more rebase should get your build green 😅

[mfendeksilverstripe]

Thanks for update @Cheddam . It looks like #334 needs to be merged into 4 before I can rebase. Please let me know when this is done. :)

[Cheddam]

@mfendeksilverstripe The merge up has been completed now :)

[mfendeksilverstripe]

@Cheddam I've rebased the branch and the tests are now green. Tested this again on my local setup and it's all looking good. Just for reference this is my local test setup:

{
    "name": "silverstripe/installer",
    "type": "silverstripe-recipe",
    "description": "The SilverStripe Framework Installer",
    "require": {
        "php": ">=7.1.0",
        "silverstripe/recipe-plugin": "^1.2",
        "silverstripe/recipe-cms": "^4.7",
        "silverstripe-themes/simple": "~3.2.0",
        "silverstripe/login-forms": "4.1.2@stable",
        "symbiote/silverstripe-queuedjobs": "dev-bugfix/doorman-runner as 4.6.0",
        "silverstripe-terraformers/gridfield-rich-filter-header": "^2.1"
    },
    "require-dev": {
        "phpunit/phpunit": "^5.7",
        "sminnee/phpunit-mock-objects": "^3.4.5"
    },
    "extra": {
        "resources-dir": "_resources",
        "project-files-installed": [
            "app/.htaccess",
            "app/_config.php",
            "app/_config/database.yml",
            "app/_config/mimevalidator.yml",
            "app/_config/mysite.yml",
            "app/src/Page.php",
            "app/src/PageController.php"
        ],
        "public-files-installed": [
            ".htaccess",
            "index.php",
            "web.config"
        ]
    },
    "config": {
        "process-timeout": 600
    },
    "prefer-stable": true,
    "minimum-stability": "dev",
    "repositories": {
        "symbiote/silverstripe-queuedjobs": {
            "type": "git",
            "url": "https://github.com/silverstripe-terraformers/silverstripe-queuedjobs.git"
        }
    }
}

---
Name: app-queue
After: '#queuedjobsettings'
---
Symbiote\QueuedJobs\Services\QueuedJobService:
  lock_file_enabled: true
  lock_file_path: '/public/queue-lock'
  max_init_jobs: 100
  memory_limit: 128m
  use_shutdown_function: false

SilverStripe\Core\Injector\Injector:
  Symbiote\QueuedJobs\Services\QueuedJobService:
    properties:
      queueRunner: '%$Symbiote\QueuedJobs\Tasks\Engines\DoormanRunner'

[Cheddam]

Thanks for all your hard work on this one @mfendeksilverstripe!