Proposing state management capabilities to NCronJob

[falvarez1]

I'd like to discuss the possibility of enhancing NCronJob with state management for jobs, similar to other job orchestration frameworks. This feature would track states like Scheduled, Running, Retrying, Completed, Failed, and Cancelled. Here are the potential benefits and features:

• Enhanced Observability: Track and visualize job states in real-time, providing better insights and monitoring capabilities.
• Actionable States: Implement state change hooks that allow actions to be triggered based on state transitions. This can be used for notifications, cleanup tasks or conditional execution of other jobs (job chaining).
• Job History and Persistence: With state tracking, we can store historical data on job executions, which can be useful for auditing, debugging, and optimizing job schedules.

This enhancement would not only improve the robustness and usability of NCronJob but also bring it closer to feature parity with more advanced workflow orchestration tools.

I see state management as the core just the beginning of what we can do. Starting with basic state management, we have the opportunity to lay a solid foundation that future enhancements can build upon. We can kick things off with the essentials, and then, based on our experiences and feedback, we can expand and refine the features over time.

Would love to hear your thoughts on this idea.

[linkdotnet]

Thanks for the rich ideas!

• Enhanced Observability: I am totally on board! Currently, it is more of a black box that isn't created in somewhat enterprise-grade applications.
• Actionable States: In general, I like the idea. My gut tells me that it is a slippery slope. On the one hand, I like to keep the scheduler rather simple than complex (otherwise, it might end up as yet another Quartz/Hangfire clone). On the other hand, I do understand the need for that. To make it right there has to be a lot of flexibility in the system as well as a nice API that keeps it nice and easy for you
• Job History and Persistence: Also one of the things I ruled out, with the same explanation as above. I wanted to have a very low barrier to use the library with almost 0 config.

But all in all these are very good features. My current way of thinking is to make the NCronJob library Plug&Play ready and offer hooks for extensions packages.

For example, persistence: We would probably need multiple providers (like Postgres, SQL Server, Mongo, Cosmos, ...), so the system has to be flexible either way.

[falvarez1]

For actionable states, we certainly could hash out the design details here.
This wouldn't be another Quartznet/Hangfire. TBH every time I've tried using them I've always found something missing and would end up writing something custom or finding other solutions.

I hear you on very low barrier and easy solution and I'm on the same page. Persistence doesn't necessarily have to use external databases like you mentioned. In order to start simple, or even just keep things simple, we could use a file based write-ahead logging approach. Zero config to the end user. Maybe just a switch that says "persist to disk".
Microsoft has a project called FASTER that may be an option. Either way I think start simple, no need to set up docker-compose scripts with services that call to databases etc. just to set up the infrastructure.

[linkdotnet]

https://github.com/microsoft/FASTER sounds nice! I am very much in favor of just trying it out and in the worst scrap it again.
Alone the name of the library should be a reason to use it :D

Kidding aside - I do agree that is a good start. I had some previous exposure to job-servers in the .net landscape and being overly flexible comes with a high price. That is why I wanted to keep it as simple as possible. So yeah! You convinced me to try out this direciton.

[falvarez1]

@linkdotnet
I started a branch at experiment/state-events-system with some preliminary state management. It's still a WIP but I'd like your feedback if you get a chance.

[falvarez1]

Yes the idea was to broadcast or stream the state changes and progress to a dashboard. I don't think this would be too heavy so long as we buffer the events. Instead of sending one at a time you aggregate all events over a period, let's say one second, then flush everything in the buffer. I can think of a few other strategies to deal with this at various levels of complexity but keeping it simple is best.

[linkdotnet]

Another concept that somewhat fits "natural" here is RxJS / RxNET with a store approach.
Given our requirements for buffering and whatnot, it could be worth a shot.

The only issue is that folks normally don't like it :D

[falvarez1]

Yes it does fit well and is a great package. I've used Reactive extensions many years ago, on different platforms, however I tend to shy away from heavy packages unless absolutely necessary. I think basic buffering can be done with standard data structures.

[falvarez1]

I've used this approach in other projects successfully (simplified here to represent the concept):

public class BufferedNotifier<T> : IDisposable
{
    private readonly IMessageService<T> _messageService;
    private readonly List<T> _messageBuffer = new List<T>();
    private readonly Timer _timer;
    private readonly SemaphoreSlim _asyncLock = new SemaphoreSlim(1, 1);
    private readonly ILogger<BufferedNotifier<T>> _logger;

    public BufferedNotifier(IMessageService<T> messageService, ILogger<BufferedNotifier<T>> logger)
    {
        _messageService = messageService;
        _logger = logger;
        _timer = new Timer(async _ => await FlushMessagesAsync(), null, Timeout.Infinite, Timeout.Infinite);
    }

    public async void Notify(T message)
    {
        await HandleMessageAsync(message);
    }

    private async Task HandleMessageAsync(T message)
    {
        _logger.LogTrace("Handling message");

        if (!_messageService.IsConnected)
        {
            _logger.LogWarning("The message service connection is not open.");
            return;
        }

        await _asyncLock.WaitAsync().ConfigureAwait(false);
        try
        {
            _messageBuffer.Add(message);

            if (_messageBuffer.Count == 1)
            {
                _timer.Change(TimeSpan.FromSeconds(1), Timeout.InfiniteTimeSpan);
            }
        }
        finally
        {
            _asyncLock.Release();
        }
    }

    private async Task FlushMessagesAsync()
    {
        await _asyncLock.WaitAsync();
        List<T> messagesToSend;

        try
        {
            if (_messageBuffer.Count == 0)
                return;

            messagesToSend = [.._messageBuffer];
            _messageBuffer.Clear();
        }
        finally
        {
            _asyncLock.Release();
        }

        if (_messageService.IsConnected)
        {
            try
            {
                await _messageService.SendBatchMessageAsync("ReceiveMessagesBatch", messagesToSend);
            }
            catch (Exception ex)
            {
                _logger.LogError(ex, "An error occurred while sending batched messages");
            }
        }
    }

    public void Dispose()
    {
        _timer?.Dispose();
        _asyncLock?.Dispose();
    }
}

[linkdotnet]

Yes it does fit well and is a great package. I've used Reactive extensions many years ago, on different platforms, however I tend to shy away from heavy packages unless absolutely necessary. I think basic buffering can be done with standard data structures.

Fair point. Yes - it doesn't necessarily call for such heavy calibers here. Once you worked with such libraries the flow of data is somewhat nice to follow.

But for now - let's start with a unified mediator.

[falvarez1]

@linkdotnet I created this branch to start working on designing the job commands capabilities to support the actions that would be performed on the Dashboard. So far supports:

• Cancel in-progress JobRun
• Pause JobRun
• Resume JobRun
• Pause Job Queue
• Resume Job Queue
• Cancel all in-progress Job Runs for a given Job type

Job Commands Management WIP branch

Currently focusing on making sure the basic commands work well.
Looking for feedback

[linkdotnet]

It's a really nice start - I was wondering what "Pause JobRun" should do.

By the way that would be also a very nice enabler for the whole persistence story (if we do that).

[falvarez1]

I guess technically it's more of a "pause Job" or "pause job triggers", where a trigger can be a scheduled execution, manual trigger (instant job), and anything else that may start execution. The act of pausing a Job means you don't want any job of that Job Type to be executed. There's 3 options I see here for what it means:

• Option 1: Removing Paused Jobs from the Queue - When a job is paused, any jobs of that type currently in the queue are removed. This ensures that no paused jobs will be executed, but it also means that those jobs need to be re-queued when the job is resumed. State is persisted in the JobRegistry
• Option 2: Mark Jobs as Paused in the Queue - When a job is paused, any jobs of that type currently in the queue are marked as paused. When it comes time to execute, these jobs are skipped. State is persisted in the JobRunManager or the JobRun
• Option 3: Pausing the Job Queue - The entire queue stops processing jobs. Jobs remain in the queue, but they are not dequeued or executed until the queue is resumed. This may create a tighter coupling between a Job and a Queue. The state would be defined and persisted in the JobQueue so that the JobWorker can use that as the reference.

There's a few design decisions that need to be made. Depending on the option we may need a higher level JobType which itself contains the list of instances (JobDefinitions). Or similarly to how a JobRun references a JobDefinition, we can have a JobDefinition reference a JobType. This JobType would have the metadata that doesn't change like policy metadata, JobName, JobFullName, and could include a job type state with the values (Active, Disabled, Paused) and its own Id (for persistence). We may need this for persistence either way.

[linkdotnet]

Option 1 and 3 sounds exactly like the "Disable Job" function we introduced with the runtime feature some days ago.
And Option 2 is like a variation of that. So, if we introduce something along the lines of Option 1, we might reconsider (for v3) what our public surface will be.

Of course any of those options above are somewhat better as we currently "trick" the system with an invalid CRON expression.
The runtime registry can use the manager under the hood to fulfill the requirements.

[falvarez1]

Great point! So option 2 seems to be the best representation of a paused job. And seems to align with other libraries like Quartz. I'll draft up some assumptions we can review.