Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature][Draft][WIP] Scheduler #35315

Open
wants to merge 6 commits into
base: master
from

Conversation

@Guikingone
Copy link
Contributor

Guikingone commented Jan 12, 2020

Q A
Branch? master
Bug fix? no
New feature? yes
Deprecations? no
Tickets Fix #
License MIT
Doc PR To define

Hi everyone πŸ‘‹

New year, new idea, first, let me explain why I open this PR as a draft, this new "experimentation" is mostly a draft as the framework integration is not completely done (I've some work pushed on a private repository).

So, what is it?

First, it's a new component, inspired by what Laravel can do with Illuminate/Scheduling but more important, it's inspired by modern platform that allows us to deploy our apps, in most platform, we can define tasks (cron and so on) and if we deploy our apps to "old" platform, we need to edit a crontab file where we need to declare all of our tasks before executing them (and waiting to fail?), the Scheduler component is here to solve most of the problems that can comes with scheduled tasks:

  • What if my platform does not support cron?
  • What if I need to plan tasks using Http?
  • What if I need to store tasks in a queue?
  • Etc...

So, what's inside?

First, we've a Scheduler, that's the entrypoint when it comes to scheduling a Task (this one can be a Http one, Cron, command, queue and so on), once scheduled, a Worker handle a task and execute it.

For now, this experimentation is not complete (the Worker need to be improved, the Messenger "bridge" also and so on) and this PR lack what I'm working on when it comes to the framework integration, this PR is opened in order to discuss and see if it could be a good idea to work on or if it's preferable to stop it now πŸ™‚

Thanks for the feedback πŸ‘‹

PS: I know that SymfonyCloud use YAML to declare tasks but we don't always use it and the Scheduler idea is here to give a "standard" solution.

Here's a small example:

Standalone

<?php 

use ...; 

$scheduler = new Scheduler('Europe/Paris');

$tasks = new ShellTask('app.test', 'echo Symfony', '*/5 * * * *');
$scheduler->schedule($task);

Framework

scheduler:
    timezone: 'Europe/Paris'
    directory: '/etc/cron.d'
    filename: 'app'
    tasks:
        shell:
            - { name: 'app.foo', command: 'echo Symfony!', 'expression' => '*/30 * * * *', tags: ['dev', 'benchmark'] }
        command:
            - { name: 'app.foo', command: 'cache:clear', 'expression' => '*/30 * * * *', options: { '--env': 'dev' } }

Roadmap

Features State
Being able to schedule Symfony commands using the Command FQCN βœ…
Adding shortcuts to create "default" cron expressions βœ…
Add methods that can trigger some logic before/after the execution of the task βœ…
Being able to export the scheduled tasks using txt and json formats βœ…
Being able to create cron files ➑️
Being able to create a pool of tasks 🚫 (Abandoned)
Schedule tasks for specific duration βœ…
Schedule tasks for specific execution date ➑️
Adding task (and pool) priority (Pools removed) βœ…
Track and optimise tasks execution ➑️
Allow to trigger tasks using a specific path (like fragments) ➑️
Scheduler factory TODO
Cloud platform bridges ➑️
FrameworkBundle integration ➑️
WebProfilerBundle integration ➑️
Notifier integration To discuss
Documentation TODO
@Guikingone Guikingone changed the title [Feature] Scheduler [Feature][Draft] Scheduler Jan 12, 2020
@Guikingone Guikingone force-pushed the Guikingone:component/scheduler branch 2 times, most recently from 38d2b08 to 680941e Jan 12, 2020
@nicolas-grekas nicolas-grekas added this to the next milestone Jan 12, 2020
@Guikingone Guikingone force-pushed the Guikingone:component/scheduler branch 4 times, most recently from 0ae5e20 to fcbdf90 Jan 12, 2020
@stephane-lou

This comment has been minimized.

Copy link

stephane-lou commented Jan 13, 2020

I actually made a private bundle that does this. It doesn't use Messenger (for now) but allows me to schedule symfony commands using a graphical interface.
Currently I made a main command (taskamanager:execute) that checks every minute if there are scheduler commands to execute.

I also think this would be a great new component !

@Guikingone

This comment has been minimized.

Copy link
Contributor Author

Guikingone commented Jan 13, 2020

Hi @stephane-lou πŸ‘‹

For now, the Messenger implementation is mainly here in order to allow to send tasks to queue and build something similar to what Notifier does, lately, I've planned to build something that allows us to do the following process:

  • Send a task to RabbitMQ queue
  • Consume the message thanks to a handler when the task expression is due
  • If the message task is successful, send a notification (think Slack and so on)

For the graphical interface, I think it could be a great idea (using Security of course πŸ˜‰ ), this way, we can easily gave the access to this interface to a specific team department that allows them to create task without specific knowledge of the internal logic πŸ™‚

@Guikingone Guikingone force-pushed the Guikingone:component/scheduler branch 2 times, most recently from 988048a to 6e12fc5 Jan 13, 2020
@kbond

This comment has been minimized.

Copy link
Contributor

kbond commented Jan 13, 2020

Wow, what timing, I just open sourced a bundle for this: https://github.com/kbond/schedule-bundle

Preparing for a 1.0 release this week.

I'd be happy to help contribute if this is something that is decided should be in core.

@Guikingone Guikingone force-pushed the Guikingone:component/scheduler branch 2 times, most recently from f39b2f5 to 2617be7 Jan 14, 2020
@Guikingone Guikingone requested a review from fabpot Jan 18, 2020
@Guikingone

This comment has been minimized.

Copy link
Contributor Author

Guikingone commented Jan 18, 2020

Oups, wrong click, sorry for the ping FabPot :(

@Guikingone Guikingone removed the request for review from fabpot Jan 18, 2020
@fabpot

This comment has been minimized.

Copy link
Member

fabpot commented Jan 18, 2020

The ping is actually interesting... as I do have a "Cron" component locally. It was supposed to be announced at a Symfony conference last year, but I decided to release the Notifier one first. Might come soon though.

@monteiro

This comment has been minimized.

Copy link
Contributor

monteiro commented Jan 19, 2020

Thanks a lot for contributing @Guikingone.

Besides handling crons, I think this could also be interesting for one time tasks. For example, after 3 hours on a specific action, send an e-mail to try to convert a the customer instead of using a CRON.

@Guikingone

This comment has been minimized.

Copy link
Contributor Author

Guikingone commented Jan 19, 2020

@monteiro Actually, that's a planned feature, I've also planned to being able to create a pool of tasks (think about user specific tasks) that can be executed anytime πŸ™‚

@Guikingone Guikingone force-pushed the Guikingone:component/scheduler branch 2 times, most recently from 8696fc6 to 8a2f6e4 Jan 23, 2020
@Guikingone

This comment has been minimized.

Copy link
Contributor Author

Guikingone commented Jan 23, 2020

Hi everyone πŸ‘‹

Here's a small list of recently added features (that need to be completely tested):

  • Task pools
  • Task pools & tasks priority
  • Failed tasks list (the Worker will retrieve it when a runner throw an Error).
@Guikingone Guikingone force-pushed the Guikingone:component/scheduler branch 2 times, most recently from 2b6179e to 9e7b24b Jan 23, 2020
@Guikingone

This comment has been minimized.

Copy link
Contributor Author

Guikingone commented Jan 23, 2020

@monteiro Regarding your feedback, I've pushed a refactor that allow to pass a valid expression handled by strtotime or even a \DatetimeInterface, the actual CRON lib allow to calculate the due date depending on a valid expression or a \DatetimeInterface

@Guikingone Guikingone force-pushed the Guikingone:component/scheduler branch from 9e7b24b to c18e139 Jan 23, 2020
@Guikingone Guikingone force-pushed the Guikingone:component/scheduler branch from c18e139 to b3416fe Feb 2, 2020
@Guikingone

This comment has been minimized.

Copy link
Contributor Author

Guikingone commented Feb 2, 2020

Hi everyone πŸ‘‹

First, I want to thanks everyone for the first feedbacks and @nicolas-grekas for the mention on the Sf meetup slides 😊

I've pushed a refactoring which remove the notion of TaskPool, the TaskList already play this role, second, I've started an integration of the Lock component (as mentioned by @bigfoot90 and @jderusse), It's not fully in place but I'm gonna improve the tests and continue to test the integration.

Third point, I've push a first functional implementation of the Exporter using the Serializer (for formats like XML and JSON), I've planned to take a moment to look at a CLI "formatter" but I'm not sure if it worth the works.

Fourth, what's the plan now?

As I've a first functional version of the component, I'm gonna polish the tests, finish every case that I can see by far, continue on the framework integration and improve the DX and so on πŸ™‚

@Guikingone Guikingone force-pushed the Guikingone:component/scheduler branch from 8e7587c to 3e721c6 Feb 2, 2020
@Guikingone

This comment has been minimized.

Copy link
Contributor Author

Guikingone commented Feb 3, 2020

Hi everyone,

I've just added a roadmap in the first post, the idea is to list the features that I've planned to add, which are done and so on.

@Guikingone Guikingone force-pushed the Guikingone:component/scheduler branch 2 times, most recently from 04cdfb7 to 294c53b Feb 3, 2020
@Guikingone

This comment has been minimized.

Copy link
Contributor Author

Guikingone commented Feb 3, 2020

Ok, new day, new feature, I've just pushed an ExpressionFactory which is capable of building CRON expressions using shortcuts, as I don't know if the FabPot "CRON component" will support both at and batch syntax, I don't have shortcuts for it.

@Guikingone Guikingone force-pushed the Guikingone:component/scheduler branch 4 times, most recently from 2025b0a to 2f956e9 Feb 4, 2020
@Guikingone

This comment has been minimized.

Copy link
Contributor Author

Guikingone commented Feb 4, 2020

Hi everyone πŸ‘‹

New day, new feature (again, I know πŸ˜„ ), I've focused on bringing a DataCollector, for now, it collect the data during the request using a TraceableScheduler (I've planned to do the same thing for Worker), I'm not sure about what to collect by far so I've built a simple array which contains the core data linked to a task.

I've also worked on a core feature of the component, being able to execute tasks using a specific route (like fragments), I'm not sure about the implementation but definitively a fun moment playing with this.

Last but not least, I've improved the output handling when a runner finish to run a task, by default, the runner can return a string or null, the output is also send to an event.

Don't hesitate to give feedbacks πŸ™‚

@Guikingone Guikingone force-pushed the Guikingone:component/scheduler branch from 76484f9 to ae5e3f1 Feb 6, 2020
@Guikingone

This comment has been minimized.

Copy link
Contributor Author

Guikingone commented Feb 6, 2020

Hi everyone πŸ‘‹

New day, new feature, I've focused the work on the DX, the Scheduler has now an interface which define core methods (resume, pause, etc).

Most of the logic is tested and in place, I've started 3 bridges for the following platforms:

  • Google Cloud Platform
  • Amazon Web Services
  • Microsoft Azure

I've also planned to build a bridge for SymfonyCloud but I don't know if there's a public API/SDK which allows us to plan cron tasks, for now and if it's not possible, this bridge will stay as a "planned one", for the bridges, the Google one is prettymuch in place, I've used the HttpClient component to call the REST API, the Google approach is pretty simple when it comes to scheduling tasks, Microsoft is a bit harder (and I don't talk about Amazon for now ... πŸ˜„ ).

That's all for today, don't hesitate to give feedbacks πŸ˜‰

@Guikingone Guikingone force-pushed the Guikingone:component/scheduler branch 3 times, most recently from d022805 to 2b2173f Feb 6, 2020

$exportDirectory = sprintf('%s/%s', $exportDirectory, $filename);

$this->exporter->export($tasks, $exportDirectory, $format);

This comment has been minimized.

Copy link
@94noni

94noni Feb 12, 2020

Contributor

IMO this should be in the execute method

interface FormatterInterface
{
public const PRIORITIES = [
'ultra_high' => 'ultra_high',

This comment has been minimized.

Copy link
@94noni

94noni Feb 12, 2020

Contributor

Shouldnt value be integers?

This comment has been minimized.

Copy link
@Guikingone

Guikingone Feb 13, 2020

Author Contributor

In fact, values will be integers, I've started with string but it's not as stable/easy to use as integers πŸ˜‰


public function __construct(TaskInterface $task)
{
$this->task = $task;

This comment has been minimized.

Copy link
@94noni

94noni Feb 12, 2020

Contributor

Why passing the whole task and not a guid
And then the worker « restore » the task from its storage via this identifier?

src/Symfony/Component/Scheduler/README.md Outdated Show resolved Hide resolved
*/
interface RunnerInterface
{
public function run(TaskInterface $task): string;

This comment has been minimized.

Copy link
@94noni

94noni Feb 12, 2020

Contributor

Maybe describe the string that must be returned

For the support: bool it is obvious, but not the run: string

*/
private $metadata = [];

public function __construct(string $name, array $metadata = [], array $additionalMetadata = [])

This comment has been minimized.

Copy link
@94noni

94noni Feb 12, 2020

Contributor

Why not options instead of metadata?

This comment has been minimized.

Copy link
@Guikingone

Guikingone Feb 16, 2020

Author Contributor

I've started the refactoring on this part πŸ™‚

@Guikingone Guikingone force-pushed the Guikingone:component/scheduler branch 2 times, most recently from 0cf331d to b48f69e Feb 13, 2020
@Guikingone Guikingone changed the title [Feature][Draft] Scheduler [Feature][Draft][WIP] Scheduler Feb 16, 2020
@Guikingone Guikingone force-pushed the Guikingone:component/scheduler branch from b48f69e to 1274e31 Feb 16, 2020
@Guikingone Guikingone force-pushed the Guikingone:component/scheduler branch from 1274e31 to b8033a3 Feb 16, 2020
Guikingone added 5 commits Feb 16, 2020
rft
rft
@Guikingone

This comment has been minimized.

Copy link
Contributor Author

Guikingone commented Feb 16, 2020

Hi everyone πŸ‘‹

Small update about this PR, I've pushed a "first valid implementation" of the framework configuration (not handled via the container for now) in order to demonstrate how to configure it, some parts are missing (but pushed on secondary repo).

I've also improved the syntax, some classes and the internal logic of the component, feel free to give feedbacks πŸ™‚

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

10 participants
You can’t perform that action at this time.