Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cron proposal - Draft 1 #171

Open
wants to merge 1 commit into
base: master
from
Open
Changes from all commits
Commits
File filter...
Filter file types
Jump to…
Jump to file or symbol
Failed to load files and symbols.

Always

Just for now

@@ -0,0 +1,63 @@
# Overview
The cron is one of the most business critical components of Magento, offloading resource
intensive tasks from the user request into the backend and allowing scheduling long running
processes.

The business critical element of the cron needs better supporting in the foundation of the module and
supporting documentation giving better control to developers, technology providers and eCommerce managers.

# Problem
The cron implementation in it's current existence doesn't provide the business critical foundation
or visibility required for a fully functioning eCommerce system.

# Solution - Business Critical Foundation
The cron should be built with a *business critical* foundation. It's responsible for some of the
most critical processes on a Magento implementation; from transactional emails to integrations.

1. Separation
2. Visibility
3. Education

## 1. Separation
Separation of business critical and none business critical processes as part of the core architecture
of the cron. This should be multi threaded "out the box" running business critical processes
in their own threads (Emails, Indexes).

At the foundation no rogue process should ever prevent a business critical operation.

- Multi thread core cron processes out the box
- Process timeouts (Default X minutes and configurable by job)
- Retry limits
- Updated documentation to encourage multi thread setup as standard


## 2. Visibility
On Magento 1 we had the AOE Scheduler which provided us a considerably enhanced user experience. This
module isn't available for Magento 2. Allowing webmasters and developers to clearly see what processes
are running, scheduled and complete gives them greater control and reduces support requests and increases
confidence in the platform.

- Admin visible timeline of scheduled cron tasks (Scheduled, in progress, complete, errored)
- Ability to schedule, delete, remove, pause specific cron tasks.

This comment has been minimized.

Copy link
@buskamuza

buskamuza Jun 17, 2019

Contributor

How "pause" should work? Is it pausing before the task started?

This comment has been minimized.

Copy link
@lbajsarowicz

lbajsarowicz Jun 24, 2019

Contributor

I think it's more "Suspend" than "Pause", which means that you can stop executing specific task until it's unsuspended.

This comment has been minimized.

Copy link
@knowj

knowj Jun 25, 2019

Author

^ Exactly what I'm proposing here. A safe way to kill a process once it's within a queue/worker might be something to consider

- Cleanup of redundant processes to reduce noise
- Logging and Alerting

Alternative approach would be a cron dashboard displaying processes in progress, queued and error conditions.

## 3. Education
- Technical documentation for effectively implementing crons into Magento modules
- Setup documentation for correctly implementing multi threaded cron (Standard setup)

# Additional discussion
1. Should RabbitMQ be standard/recommended for certain tasks such as transactional emails and indexes?

This comment has been minimized.

Copy link
@buskamuza

buskamuza Jun 14, 2019

Contributor

Yes, it would be good to document. What rule would be a good description? As I see it:

  1. Message queue should be used for async tasks. Transactional emails and indexes go into this category.
  2. Cron should be used for tasks scheduled for specific time or delay.

This comment has been minimized.

Copy link
@nuzil

nuzil Jun 14, 2019

Contributor

Here I would add, that we should not stick to RabbitMQ directly, but more use Magento queueing system and use DB queue if RabbitMQ is not available. But in any case transactional emails have to be queued and decoupled with cron.

2. New Relic currently logging as *unknown* we have an implementation but the Cron module requires work to allow it

# Examples

## Example 1 - Rogue process
A process that connects to an API such as a payment provider could end up getting stuck in a loop
and blocking any other process from progressing currently there is no mechanic in place to
a: Allow other business critical processes to continue aside from defining multiple crontab processes in groups
b: Report or stop the rogue process from continuing until manually spotted
c: No visibility of the processes aside from checking the database
d: No reporting or alerting of the above condition
ProTip! Use n and p to navigate between commits in a pull request.
You can’t perform that action at this time.