Double executing of the same job

[ApmeM]

I have a strange behavior for some (not all) jobs:

We have a few equally configured jobs (you can see "Bubbleroom" text on the image, we have a few more jobs with different names there. Some of them executed once, but this one executed twice without any notifications.
We have only 1 server and 1 DEFAULT queue executed as windows service.
Job each time executed from start to end without exceptions (last line is writing a log message to papertrail and this log message appears twice).
Recurring jobs page shows that it was started exactly when it should (it is true).
All jobs (24 jobs on a image) have tag to prevent concurrent executions, but

I cant understand why this happen and what should I check and how to prevent this issue?

[YukaAn]

Im experiencing the same issue. Did you find the cause and the solution for it?

[dlazzarino]

Hi all,
I have exactly the same issue. Let me describe the lifecycle of my application, useful
to understand if something is not well configured or something must be moved.

I'm using Hangfire server version 1.6.17 with SQL Server storage, the production environment is:

• 2 Windows server 2012 R2
• Sql Server 2016

My application is an asp.net MVC 5 website, .Net Framework 4.6.1.
The application is configured on "Always running" mode on both server, following the Hangfire documentation.

Below the application lifecycle (some non-significant parts will be omitted) :

1 - Application preload

...
public void Preload(string[] parameters)
{
	HangfireBootstrapper.Instance.Start();
}
...

HangfireBootstrapper class Start function boby

...
var options = new SqlServerStorageOptions
{
	PrepareSchemaIfNecessary = false,
	SchemaName = "dbo"                    
};

GlobalConfiguration.Configuration.UseSqlServerStorage("SRP_HANGFIRE", options);

// Disable automatic retry for all hangfire job
GlobalJobFilters.Filters.Add(new AutomaticRetryAttribute { Attempts = 0 });
GlobalJobFilters.Filters.Add(new DisableConcurrentExecutionAttribute(60 * 5));

BackgroundJobServerOptions srvOptions = new BackgroundJobServerOptions()
{
	ServerName = Environment.MachineName,
	Queues = new[] { "warehouse", "default" }                    
};

_backgroundJobServer = new BackgroundJobServer(srvOptions);
...

2 - Global.asax.cs

...
protected void Application_Start(object sender, EventArgs e)
{
	HangfireBootstrapper.Instance.Start();
}

protected void Application_End(object sender, EventArgs e)
{
	HangfireBootstrapper.Instance.Stop();
}
...

3 - OWIN startup class

...	
app.UseHangfireDashboard("/Home/hangfire", new DashboardOptions
{
	Authorization = new[] { new DashboardAuthorizationFilter() },
	AppPath = VirtualPathUtility.ToAbsolute("~")
});

JobsBootstrapper.SetupJobs();		
...

The class JobsBootstrapper contains all my recurring job (45 jobs), like this:

...
DummyJobParam djParam = new DummyJobParam()
{
	Number = 100,
	MultipleOf = 5
};
RecurringJob.AddOrUpdate("999", () => new DummyJob().DoWork("Test - Dummy_Job", djParam), "*/3 * * * *", TimeZoneInfo.Utc, "default");	
...

Returning to the problem of double execution, I think that is related to the application pool recycling, what I saw is that the concurrent
execution management doesn't work well when one or both app pools are restarted (not necessarily simultaneously).
For me is very strange because, using a centralized storage like SQL Server, it's simple to manage the concurrency on database and avoid this situation.

Another possible cause, but I'm not sure, can be the JobBootstrapper phase. In this phase, executed on startup, the scheduler engine is running and
we are adding/updating jobs to the job list. Is it safe do this after the BackgroundJobserver start? I didn't find a specific documentation about.
Is it better update the job list before starting the server?
Is it better update the job list only on one server?

I hope someone will give me suggestions and that this BUG will be fixed soon

Thanks in advance

[dlazzarino]

Hi all,
I want to confirm that the BUG is related to the application pool recycling. I traced on my servers an app pool recycling at 02:00 am and the job executed twice few days ago is scheduled exactly at the same time 02:00 am

Another question is: the PRO version works well or has the same issue?

For me, this is an urgent issue to resolve, the multi-instance feature with SQL Server storage doesn't work as expected

Br

[fcometto]

Same issue also for me, We observed it just after the application pool recycle.
Any forecast for the resolution?

Thanks

[IanYates]

Was your job executing at the time of the recycle?

It sounds like you want Hangfire to kick off the job at the time, but regardless of how the job proceeds, to never try again. So that if the recycle occurs mid-job then you don't want the job to be tried again. Is that right?

I guess one hacky way to reduce the chance of this occurring is to make your Hangfire job a single statement that starts a new task on the thread pool. That way the Hangfire job starts and finishes really quickly and, if the task gets killed, so be it.

[praveenlobo7]

Hi,

We have the same issue. We have a long running job which takes more than 30 mins. Right after 30 mins, same job kicks off.
Any solution for this?