New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Background task schedule system #41

Closed
hikalkan opened this Issue Jul 17, 2014 · 15 comments

Comments

Projects
None yet
8 participants
@hikalkan
Member

hikalkan commented Jul 17, 2014

Develop a system that we can scnedule and queue background tasks.

A task consists of;

  • A class implements IBackgroundTask (or something like that)
  • A schedule (one time, hourly, daily... etc).

There should be a schedular class that can be worked in single or multiple threaded. Their working should be logged (and measured duration optionally) and so on...

This schedular should work in IIS process. Thus, application should be alive always. If it's not a web app, it should work in-process.

This may be a good feature of an application framework.

NOTE: We should implement these feature as extensible. Also, anyone may want to use Quartz.Net or other tools. So, it should be possible to create adapters to these libraries.

@EverPresent

This comment has been minimized.

Show comment
Hide comment
@EverPresent

EverPresent Jul 25, 2014

I've used the Quartz.NET job scheduler before under IIS (http://www.quartz-scheduler.net/). It is subject to IIS limitations (http://stackoverflow.com/questions/16755373/quartz-net-job-stops-running) but I don't think any in-process IIS scheduler will be able to work around those (but I'm far from an IIS expert)...

The scheduling API is a bit confusing but I believe your plan is to completely wrap the library behind standard interfaces so this might not be much of a drawdown.

EverPresent commented Jul 25, 2014

I've used the Quartz.NET job scheduler before under IIS (http://www.quartz-scheduler.net/). It is subject to IIS limitations (http://stackoverflow.com/questions/16755373/quartz-net-job-stops-running) but I don't think any in-process IIS scheduler will be able to work around those (but I'm far from an IIS expert)...

The scheduling API is a bit confusing but I believe your plan is to completely wrap the library behind standard interfaces so this might not be much of a drawdown.

@hikalkan

This comment has been minimized.

Show comment
Hide comment
@hikalkan

hikalkan Jul 26, 2014

Member

Hi @EverPresent

I did not use but I know Quartz.NET is one of the most popular libraries in this subject. I think to use this or another good library with a simplified wrapper API as you said.

I know IIS limitations. But there may be some options to solve these problems. I'll search and try more before implement this feature. Running background jobs in a windows service may be better but it's more complicated. Especially if you want to interact these jobs with web requests or uses some application level caching. There may be situations where we want to inform background jobs on a web request, and this requires an interprocess communication. So, at least for simple tasks, running in web application process would be good.

Thanks for your comments.

Member

hikalkan commented Jul 26, 2014

Hi @EverPresent

I did not use but I know Quartz.NET is one of the most popular libraries in this subject. I think to use this or another good library with a simplified wrapper API as you said.

I know IIS limitations. But there may be some options to solve these problems. I'll search and try more before implement this feature. Running background jobs in a windows service may be better but it's more complicated. Especially if you want to interact these jobs with web requests or uses some application level caching. There may be situations where we want to inform background jobs on a web request, and this requires an interprocess communication. So, at least for simple tasks, running in web application process would be good.

Thanks for your comments.

@natiki

This comment has been minimized.

Show comment
Hide comment
@natiki

natiki Sep 4, 2014

Contributor

I second the suggestion to use Quartz.NET. I am not really in favour of a Windows service implementation as often deployments are to shared hosting where access to service installation is limited. AFAIK Quartz also offers a service based option. The Quartz API is pretty well encapsulated, but I have found that adding a thin wrapper helper class around it is beneficial.

Contributor

natiki commented Sep 4, 2014

I second the suggestion to use Quartz.NET. I am not really in favour of a Windows service implementation as often deployments are to shared hosting where access to service installation is limited. AFAIK Quartz also offers a service based option. The Quartz API is pretty well encapsulated, but I have found that adding a thin wrapper helper class around it is beneficial.

@hikalkan

This comment has been minimized.

Show comment
Hide comment
@hikalkan

hikalkan Sep 5, 2014

Member

I also investigated Quartz.Net a bit and it's really wonderfull. Although there is no restriction to not use it with ABP now, it will be good to create a little abstraction/wrapper and conventions for it.
Thanks for suggestions.

Member

hikalkan commented Sep 5, 2014

I also investigated Quartz.Net a bit and it's really wonderfull. Although there is no restriction to not use it with ABP now, it will be good to create a little abstraction/wrapper and conventions for it.
Thanks for suggestions.

@hikalkan

This comment has been minimized.

Show comment
Hide comment
@hikalkan

hikalkan Sep 7, 2014

Member

Also, investigate http://hangfire.io/. It's more proper at first look since it's persistent job focused.

Member

hikalkan commented Sep 7, 2014

Also, investigate http://hangfire.io/. It's more proper at first look since it's persistent job focused.

@natiki

This comment has been minimized.

Show comment
Hide comment
@natiki

natiki Sep 8, 2014

Contributor

Quartz.net can persist to SQL Server as well as other DBMS's out of the box. If all you need is something basic it does memory persistence by default.

Contributor

natiki commented Sep 8, 2014

Quartz.net can persist to SQL Server as well as other DBMS's out of the box. If all you need is something basic it does memory persistence by default.

@hikalkan

This comment has been minimized.

Show comment
Hide comment
@hikalkan

hikalkan Sep 8, 2014

Member

Yes it can but it's a bit harder to work Quartz.Net with persistence. I could start with hangfire in a few minutes, it created db tables immediately.
But.. for both of them, I did not like that they create too much (8-10) tables :) Maybe we can store jobs to a dedicated embedded db instead of our main db.

Member

hikalkan commented Sep 8, 2014

Yes it can but it's a bit harder to work Quartz.Net with persistence. I could start with hangfire in a few minutes, it created db tables immediately.
But.. for both of them, I did not like that they create too much (8-10) tables :) Maybe we can store jobs to a dedicated embedded db instead of our main db.

@hikalkan hikalkan modified the milestone: ABP v0.7.0 Mar 20, 2015

@hikalkan hikalkan modified the milestones: ABP v1.0.0, ABP v0.7.0 Apr 15, 2015

@johnkattenhorn

This comment has been minimized.

Show comment
Hide comment
@johnkattenhorn

johnkattenhorn Jan 2, 2016

We've been using Hangfire with ABP for a while now and it took a while to figure out the Castle Windsor issues but it works fine now.

We use it as a WebJob in Azure as we didn't want to affect the performance of App whilst the jobs was running.

Let me know if you want us to share some code and I'll see what I can do.

johnkattenhorn commented Jan 2, 2016

We've been using Hangfire with ABP for a while now and it took a while to figure out the Castle Windsor issues but it works fine now.

We use it as a WebJob in Azure as we didn't want to affect the performance of App whilst the jobs was running.

Let me know if you want us to share some code and I'll see what I can do.

@wocar

This comment has been minimized.

Show comment
Hide comment
@wocar

wocar Jan 2, 2016

Contributor

I'm interested. Would you mind sharing? Thank you!!

El sáb., ene. 2, 2016 2:51 PM, John Kattenhorn notifications@github.com
escribió:

We've been using Hangfire with ABP for a while now and it took a while to
figure out the Castle Windsor issues but it works fine now.

We use it as a WebJob in Azure as we didn't want to affect the performance
of App whilst the jobs was running.

Let me know if you want us to share some code and I'll see what I can do.


Reply to this email directly or view it on GitHub
#41 (comment)
.

Contributor

wocar commented Jan 2, 2016

I'm interested. Would you mind sharing? Thank you!!

El sáb., ene. 2, 2016 2:51 PM, John Kattenhorn notifications@github.com
escribió:

We've been using Hangfire with ABP for a while now and it took a while to
figure out the Castle Windsor issues but it works fine now.

We use it as a WebJob in Azure as we didn't want to affect the performance
of App whilst the jobs was running.

Let me know if you want us to share some code and I'll see what I can do.


Reply to this email directly or view it on GitHub
#41 (comment)
.

@johnkattenhorn

This comment has been minimized.

Show comment
Hide comment
@johnkattenhorn

johnkattenhorn Jan 3, 2016

No problem, I've put it into my TODO list for next week, let you know soonest.

johnkattenhorn commented Jan 3, 2016

No problem, I've put it into my TODO list for next week, let you know soonest.

@hikalkan hikalkan modified the milestones: ABP v0.8.0, ABP v1.0.0 Jan 20, 2016

@iyhammad

This comment has been minimized.

Show comment
Hide comment
@iyhammad

iyhammad Jan 20, 2016

Contributor

We are using Hangfire with ABP for sometime now and it does the job so far with integration with ABP (Windsor) dependency injection.
Here is how we set it up.

In Global.asax we do the registration and add the SQL Support if required

var options = new SqlServerStorageOptions
{
       //InvisibilityTimeout = TimeSpan.FromMinutes(30) // default value
};
Hangfire.GlobalConfiguration.Configuration.UseWindsorJobActivator(IocManager.Instance)
.UseSqlServerStorage("Default", options); // Here you can put any Connection String
System.Web.Http.GlobalConfiguration.Configure(WebApiConfig.Register);

Here are the Windsor Configuration

using System;
using Hangfire;
using Hangfire.Annotations;
using Abp.Dependency;

namespace MySystem.Infrastructure
{
    public static class GlobalConfigurationExtensions
    {
        public static IGlobalConfiguration<WindsorJobActivator> UseWindsorJobActivator(
            [NotNull] this IGlobalConfiguration configuration,
            [NotNull]  IIocResolver iocResolver)
        {
            if (configuration == null) throw new ArgumentNullException("configuration");
            if (iocResolver == null) throw new ArgumentNullException("iocResolver");

            return configuration.UseActivator(new WindsorJobActivator(iocResolver));
        }
    }
}

Here is the Windsor Job Activator

using System;
using Hangfire;
using Abp.Dependency;

namespace MySystem.Infrastructure
{
    public class WindsorJobActivator : JobActivator
    {
        readonly IIocResolver _iocResolver;

        /// <summary>
        /// Initializes new instance of WindsorJobActivator with a Windsor Kernel
        /// </summary>
        /// <param name="kernel">Kernel that will be used to create instance
        /// of classes during job activation process.</param>
        public WindsorJobActivator(IIocResolver iocResolver)
        {
            if (iocResolver == null) throw new ArgumentNullException("iocResolver");

            _iocResolver = iocResolver;
        }

        /// <summary>
        /// Activates a job of a given type using the Windsor Kernel
        /// </summary>
        /// <param name="jobType">Type of job to activate</param>
        /// <returns></returns>
        public override object ActivateJob(Type jobType)
        {
            return _iocResolver.Resolve(jobType);
        }
    }
}
Contributor

iyhammad commented Jan 20, 2016

We are using Hangfire with ABP for sometime now and it does the job so far with integration with ABP (Windsor) dependency injection.
Here is how we set it up.

In Global.asax we do the registration and add the SQL Support if required

var options = new SqlServerStorageOptions
{
       //InvisibilityTimeout = TimeSpan.FromMinutes(30) // default value
};
Hangfire.GlobalConfiguration.Configuration.UseWindsorJobActivator(IocManager.Instance)
.UseSqlServerStorage("Default", options); // Here you can put any Connection String
System.Web.Http.GlobalConfiguration.Configure(WebApiConfig.Register);

Here are the Windsor Configuration

using System;
using Hangfire;
using Hangfire.Annotations;
using Abp.Dependency;

namespace MySystem.Infrastructure
{
    public static class GlobalConfigurationExtensions
    {
        public static IGlobalConfiguration<WindsorJobActivator> UseWindsorJobActivator(
            [NotNull] this IGlobalConfiguration configuration,
            [NotNull]  IIocResolver iocResolver)
        {
            if (configuration == null) throw new ArgumentNullException("configuration");
            if (iocResolver == null) throw new ArgumentNullException("iocResolver");

            return configuration.UseActivator(new WindsorJobActivator(iocResolver));
        }
    }
}

Here is the Windsor Job Activator

using System;
using Hangfire;
using Abp.Dependency;

namespace MySystem.Infrastructure
{
    public class WindsorJobActivator : JobActivator
    {
        readonly IIocResolver _iocResolver;

        /// <summary>
        /// Initializes new instance of WindsorJobActivator with a Windsor Kernel
        /// </summary>
        /// <param name="kernel">Kernel that will be used to create instance
        /// of classes during job activation process.</param>
        public WindsorJobActivator(IIocResolver iocResolver)
        {
            if (iocResolver == null) throw new ArgumentNullException("iocResolver");

            _iocResolver = iocResolver;
        }

        /// <summary>
        /// Activates a job of a given type using the Windsor Kernel
        /// </summary>
        /// <param name="jobType">Type of job to activate</param>
        /// <returns></returns>
        public override object ActivateJob(Type jobType)
        {
            return _iocResolver.Resolve(jobType);
        }
    }
}
@hikalkan

This comment has been minimized.

Show comment
Hide comment
@hikalkan

hikalkan Jan 20, 2016

Member

@iyhammad thank you for your sharing. But isn't this lead to a memory leak? See this: BredStik/HangFire.Windsor#3

Member

hikalkan commented Jan 20, 2016

@iyhammad thank you for your sharing. But isn't this lead to a memory leak? See this: BredStik/HangFire.Windsor#3

@IanYates

This comment has been minimized.

Show comment
Hide comment
@IanYates

IanYates Jan 20, 2016

See this gist I just posted for a Windsor-friendly Hangfire job activator
https://gist.github.com/IanYates/29180ce09c41e42b9a7e

Also refer to to the HangFire.Windsor issue that @hikalkan referenced for its explanation (although I think I'll copy that explanation over to the gist too)

IanYates commented Jan 20, 2016

See this gist I just posted for a Windsor-friendly Hangfire job activator
https://gist.github.com/IanYates/29180ce09c41e42b9a7e

Also refer to to the HangFire.Windsor issue that @hikalkan referenced for its explanation (although I think I'll copy that explanation over to the gist too)

@andmattia

This comment has been minimized.

Show comment
Hide comment
@andmattia

andmattia Jan 22, 2016

Fantastic!

andmattia commented Jan 22, 2016

Fantastic!

@hikalkan

This comment has been minimized.

Show comment
Hide comment
@hikalkan

hikalkan Jan 22, 2016

Member

Finished implementation.

A most simple background job:

public class TestJob : BackgroundJob<int>, ITransientDependency
{
    public override void Execute(int args)
    {
        Logger.Debug(args.ToString());
    }
}

How to add a job to background job queue:

public class MyService : ITransientDependency
{
    private readonly IBackgroundJobManager _backgroundJobManager;

    public MyService(IBackgroundJobManager backgroundJobManager)
    {
        _backgroundJobManager = backgroundJobManager;
    }

    public MyService()
    {
        _backgroundJobManager.EnqueueAsync<TestJob, int>(42);
    }
}

OK, I know you want a more REAL example:

public class SendPrivateEmailJob : BackgroundJob<SendPrivateEmailJobArgs>, ITransientDependency
{
    private readonly IRepository<User, long> _userRepository;

    public SendPrivateEmailJob(IRepository<User, long> userRepository)
    {
        _userRepository = userRepository;
    }

    [UnitOfWork]
    public override void Execute(SendPrivateEmailJobArgs args)
    {
        using (CurrentUnitOfWork.SetFilterParameter(AbpDataFilters.MayHaveTenant, AbpDataFilters.Parameters.TenantId, args.TargetTenantId))
        {
            var user = _userRepository.FirstOrDefault(args.TargetUserId);
            if (user == null)
            {
                Logger.WarnFormat("Unknown userId: {0}. Can not execute job!", args.TargetUserId);
                return;
            }

            //Here, we should actually send the email! We can inject and use IEmailSender for example.
            Logger.Info("Sending email to " + user.EmailAddress + " -> " + args.Subject);
            Logger.Info(args.Body);
        }
    }
}

And how to create a job:

[AbpAuthorize]
public class PrivateEmailAppService : BackgroundJobAndNotificationsDemoAppServiceBase, IPrivateEmailAppService
{
    private readonly IBackgroundJobManager _backgroundJobManager;

    public PrivateEmailAppService(IBackgroundJobManager backgroundJobManager)
    {
        _backgroundJobManager = backgroundJobManager;
    }

    public async Task Send(SendPrivateEmailInput input)
    {
        var targetUser = await UserManager.FindByNameAsync(input.UserName);
        if (targetUser == null)
        {
            throw new UserFriendlyException("There is no such a user: " + input.UserName);
        }

        await _backgroundJobManager.EnqueueAsync<SendPrivateEmailJob, SendPrivateEmailJobArgs>(
            new SendPrivateEmailJobArgs
            {
                Subject = input.Subject,
                Body = input.Body,
                SenderUserId = AbpSession.GetUserId(),
                TargetTenantId = AbpSession.TenantId,
                TargetUserId = targetUser.Id
            });
    }
}

How to switch HangFire:

  1. Add Abp.HangFire package to your project (for example, to the .Web project)
  2. Add AbpHangfireModule dependency to your module class.
  3. In your preinitialize method:
Configuration.BackgroundJobs.UseHangfire(configuration =>
{
    configuration.GlobalConfiguration.UseSqlServerStorage("Default");
});

Example project: https://github.com/aspnetboilerplate/aspnetboilerplate-samples/tree/master/BackgroundJobAndNotificationsDemo (uses local Abp references since nuget packages are not published yet)

Will document all soon.

Member

hikalkan commented Jan 22, 2016

Finished implementation.

A most simple background job:

public class TestJob : BackgroundJob<int>, ITransientDependency
{
    public override void Execute(int args)
    {
        Logger.Debug(args.ToString());
    }
}

How to add a job to background job queue:

public class MyService : ITransientDependency
{
    private readonly IBackgroundJobManager _backgroundJobManager;

    public MyService(IBackgroundJobManager backgroundJobManager)
    {
        _backgroundJobManager = backgroundJobManager;
    }

    public MyService()
    {
        _backgroundJobManager.EnqueueAsync<TestJob, int>(42);
    }
}

OK, I know you want a more REAL example:

public class SendPrivateEmailJob : BackgroundJob<SendPrivateEmailJobArgs>, ITransientDependency
{
    private readonly IRepository<User, long> _userRepository;

    public SendPrivateEmailJob(IRepository<User, long> userRepository)
    {
        _userRepository = userRepository;
    }

    [UnitOfWork]
    public override void Execute(SendPrivateEmailJobArgs args)
    {
        using (CurrentUnitOfWork.SetFilterParameter(AbpDataFilters.MayHaveTenant, AbpDataFilters.Parameters.TenantId, args.TargetTenantId))
        {
            var user = _userRepository.FirstOrDefault(args.TargetUserId);
            if (user == null)
            {
                Logger.WarnFormat("Unknown userId: {0}. Can not execute job!", args.TargetUserId);
                return;
            }

            //Here, we should actually send the email! We can inject and use IEmailSender for example.
            Logger.Info("Sending email to " + user.EmailAddress + " -> " + args.Subject);
            Logger.Info(args.Body);
        }
    }
}

And how to create a job:

[AbpAuthorize]
public class PrivateEmailAppService : BackgroundJobAndNotificationsDemoAppServiceBase, IPrivateEmailAppService
{
    private readonly IBackgroundJobManager _backgroundJobManager;

    public PrivateEmailAppService(IBackgroundJobManager backgroundJobManager)
    {
        _backgroundJobManager = backgroundJobManager;
    }

    public async Task Send(SendPrivateEmailInput input)
    {
        var targetUser = await UserManager.FindByNameAsync(input.UserName);
        if (targetUser == null)
        {
            throw new UserFriendlyException("There is no such a user: " + input.UserName);
        }

        await _backgroundJobManager.EnqueueAsync<SendPrivateEmailJob, SendPrivateEmailJobArgs>(
            new SendPrivateEmailJobArgs
            {
                Subject = input.Subject,
                Body = input.Body,
                SenderUserId = AbpSession.GetUserId(),
                TargetTenantId = AbpSession.TenantId,
                TargetUserId = targetUser.Id
            });
    }
}

How to switch HangFire:

  1. Add Abp.HangFire package to your project (for example, to the .Web project)
  2. Add AbpHangfireModule dependency to your module class.
  3. In your preinitialize method:
Configuration.BackgroundJobs.UseHangfire(configuration =>
{
    configuration.GlobalConfiguration.UseSqlServerStorage("Default");
});

Example project: https://github.com/aspnetboilerplate/aspnetboilerplate-samples/tree/master/BackgroundJobAndNotificationsDemo (uses local Abp references since nuget packages are not published yet)

Will document all soon.

hikalkan added a commit that referenced this issue Jan 22, 2016

KenKimura88 pushed a commit to KenKimura88/aspnetboilerplate that referenced this issue Jan 17, 2017

KenKimura88 pushed a commit to KenKimura88/aspnetboilerplate that referenced this issue Jan 17, 2017

KenKimura88 pushed a commit to KenKimura88/aspnetboilerplate that referenced this issue Jan 17, 2017

KenKimura88 pushed a commit to KenKimura88/aspnetboilerplate that referenced this issue Jan 17, 2017

KenKimura88 pushed a commit to KenKimura88/aspnetboilerplate that referenced this issue Jan 17, 2017

KenKimura88 pushed a commit to KenKimura88/aspnetboilerplate that referenced this issue Jan 17, 2017

KenKimura88 pushed a commit to KenKimura88/aspnetboilerplate that referenced this issue Jan 17, 2017

KenKimura88 pushed a commit to KenKimura88/aspnetboilerplate that referenced this issue Jan 17, 2017

KenKimura88 pushed a commit to KenKimura88/aspnetboilerplate that referenced this issue Jan 17, 2017

KenKimura88 pushed a commit to KenKimura88/aspnetboilerplate that referenced this issue Jan 17, 2017

KenKimura88 pushed a commit to KenKimura88/aspnetboilerplate that referenced this issue Jan 17, 2017

KenKimura88 pushed a commit to KenKimura88/aspnetboilerplate that referenced this issue Jan 17, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment