New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Hangfire Job starts multiple times--Same issue is experienced as #590 #1025

Open
YukaAn opened this Issue Oct 17, 2017 · 19 comments

Comments

@YukaAn
Copy link

YukaAn commented Oct 17, 2017

I have gone through all the open issues here and found that the issue Im experiencing supposed to be solved with v1.5.8. But Im running v1.6.6 and still seeing the similar issue. So the same job will be processed multiple times randomly. I also saw issue #842 describing the same thing. Can someone help me to fix it?

I'm using Hangfire.SqlServer V1.6.6

@YukaAn

This comment has been minimized.

Copy link

YukaAn commented Oct 17, 2017

some instances of this issue:

multipleprocessing

@odinserj

This comment has been minimized.

Copy link
Member

odinserj commented Oct 18, 2017

@YukaAn, what storage are you using? Could you include the full details, e.g. the data column?

@YukaAn

This comment has been minimized.

Copy link

YukaAn commented Oct 18, 2017

@odinserj I'm using sql server 2008, Hangfire version is 1.6.6. Below is a screenshot of a job state info:
ss

@odinserj

This comment has been minimized.

Copy link
Member

odinserj commented Oct 18, 2017

Could you show me your configuration logic and recurring method's signature?

@YukaAn

This comment has been minimized.

Copy link

YukaAn commented Oct 18, 2017

@odinserj Thank you for the quick response!

The configuration logic looks like this:

public void Configuration(IAppBuilder app) {
            GlobalConfiguration.Configuration.UseSqlServerStorage("HangfireDb");
            app.UseHangfireDashboard();
            app.UseHangfireServer();
        }

the recurring methods like this:

public void CreateRecurringJob(int hour, int minute, int Id, string Name, string occurence)
        {
            try
            {
                if(!MinuteCheck(minute) || !HourCheck(hour) || !CronCheck(occurence))
                {
                    return;
                }
                string cron = BuildCron(hour, minute, occurence);
                if(IsExistingOrNewMethod(Id, Name))
                {
                    ScheduledJobHandler handler = new ScheduledJobHandler();
                    RecurringJob.AddOrUpdate(
                        JobNameBuilder(Id, Name),
                        () => handler.SendRequest(Id, Name),
                        cron,
                        TimeZoneInfo.FindSystemTimeZoneById("Eastern Standard Time")
                    );
                }
            }
            catch(Exception ex)
            {
                throw ApiException(ex);
            }
        }
@YukaAn

This comment has been minimized.

Copy link

YukaAn commented Oct 20, 2017

@odinserj I have around 70 recurring jobs scheduled each day and this issue keeps happening couple times every day (randomly on different jobs). I'm still waiting for your reply and I appreciate your help. Thanks!

@odinserj

This comment has been minimized.

Copy link
Member

odinserj commented Nov 7, 2017

@YukaAn, sorry for the delay. Try to upgrade to the latest version. At least Hangfire.Core 1.6.12 has a fix related to a problem like yours:

• Fixed – Buggy state filters may cause background job to be infinitely retried.

Looks like there's a transient exception that occur when your job is completed, and only logging could help to investigate the issue in detail. Please see this article to learn how to enable it, and feel free to post your log messages into this thread to conduct a further investigation.

@pinual

This comment has been minimized.

Copy link

pinual commented Aug 15, 2018

I am having this same issue on Hangfire 1.6.20 and using LiteDBStorage. I have seen several other reports of this issue but no resolution. Are you still using a work around?

@mattxo

This comment has been minimized.

Copy link

mattxo commented Aug 16, 2018

Same issue here. Hangfire 1.6.20 and Hangfire.SQLite 1.4.2

public void Configure(IApplicationBuilder app, IHostingEnvironment env)
{
  ...
  RecurringJob.AddOrUpdate("debug", () => Hangfire(), Cron.Minutely);
}

public void Hangfire()
{
   Debug.WriteLine($"{DateTime.Now} - Hangfire");
}

16/08/2018 10:58:14 AM - Hangfire
16/08/2018 10:58:14 AM - Hangfire
16/08/2018 10:58:14 AM - Hangfire
16/08/2018 10:58:15 AM - Hangfire
16/08/2018 10:58:15 AM - Hangfire
16/08/2018 10:58:15 AM - Hangfire
16/08/2018 10:58:15 AM - Hangfire
16/08/2018 10:58:15 AM - Hangfire
16/08/2018 10:58:15 AM - Hangfire
16/08/2018 10:58:15 AM - Hangfire
16/08/2018 10:58:15 AM - Hangfire
@marius-stanescu

This comment has been minimized.

Copy link

marius-stanescu commented Sep 5, 2018

Hello. I have the same issue.

The job starts for every worker. If I set the server worker count to 2 then it starts 2 times, if I set it to 50 then it starts 50 times.

I use the latest Hangfire version (1.6.20) and SQLite for storage.

The job is enqueued from the web application.

BackgroundJob.Enqueue(() => StartDatabaseExport(databaseId));

The server is started from another application (windows service).

var options = new BackgroundJobServerOptions { WorkerCount = 50 };
new BackgroundJobServer(options);

image

Any ideas?

@marius-stanescu

This comment has been minimized.

Copy link

marius-stanescu commented Sep 7, 2018

I also tried with the LiteDB storage, same problem. I then tried with the in-memory storage and it works as expected. So it seems it's related to the storage.

@sheburdos

This comment has been minimized.

Copy link

sheburdos commented Sep 17, 2018

Hangfire 1.6.17.0
MemoryStorage: 1.5.1.0

Tasks are created as:

IState state = new EnqueuedState(QueueName.PRIORITY);
_jobClient.Create(() => TaskFactory.Build(id), state);

Sometimes jobs run multiple times. Log from within the task:

2018-09-12 11:45:43.7292|INFO|44fa0359-e13c-4356-bdb9-9690df16eda0|Export calculation unit 2018-09-12 12:16:14.7399|INFO|44fa0359-e13c-4356-bdb9-9690df16eda0|Export calculation unit
...
2018-09-12 12:27:51.2235|INFO|44fa0359-e13c-4356-bdb9-9690df16eda0|Done!
...
2018-09-12 12:51:39.4573|INFO|44fa0359-e13c-4356-bdb9-9690df16eda0|Done!

Job do access to database. So it could stuck for some time if all db connections from the pool are taken by other jobs. That what happened at the beginning I assume. Then the task was run again, but there were no reports on retry\error. And actually after that, task reported twice about successful completion (as well as about intermediate steps). Feels like the job was retried after some waiting period without canceling previous evaluation.

@pauldotknopf

This comment has been minimized.

Copy link

pauldotknopf commented Nov 22, 2018

Although your messages should be idempotent, this should definitely be fixed in Hangfire.

Is this issue in Hangfire itself, or in the storage providers?

@lukaszgatnicki

This comment has been minimized.

Copy link

lukaszgatnicki commented Nov 22, 2018

Hi,
We have the same problem.
We tried to find a solution and after a long time we probably managed somethink.
The problem is associated with storage beyond any doubt. Mulitiple wrokers run jobs when you use LiteDB, SQLite and similar storages. Everythink is ok with SQL Server. So if it was possible to fix the error in SQL Server Storage it could be possible in others. So this is my shy request to the creators

@pauldotknopf

This comment has been minimized.

Copy link

pauldotknopf commented Nov 22, 2018

I think I found the issue in the Sqlite provider: mobydi/Hangfire.Sqlite#2 (comment)

Maybe this issue is something similar for the SQL Server provider as well?

@Neonkiller

This comment has been minimized.

Copy link

Neonkiller commented Dec 12, 2018

Experiencing the same issue. We use SQL Server as a storage (if it matters).
Any updated when it could be fixed, or at least if root cause is known?
3processings

@srusakov

This comment has been minimized.

Copy link

srusakov commented Dec 20, 2018

Experiencing the same issue. LiteDb as a storage. As a temporary solution I set WrokerCount to 1.
@odinserj Is Pro version is free of this bug?

@sheburdos

This comment has been minimized.

Copy link

sheburdos commented Dec 21, 2018

In my case some workaround was to set extended intervals for MemoryStorage

MemoryStorageOptions storageOpts = new MemoryStorageOptions()
{
    JobExpirationCheckInterval =  TimeSpan.FromMinutes(120),
    FetchNextJobTimeout = TimeSpan.FromMinutes(120)
};
GlobalConfiguration.Configuration.UseMemoryStorage(storageOpts);

But that works more or less till the task can be done in 2 hours time span. In my case I can be sure that at least most of the tasks will be accomplished

@wtfuii

This comment has been minimized.

Copy link

wtfuii commented Jan 9, 2019

Hey, it seems as if #1197 is about the same issue. Everybody who runs into this issue might want to check it out.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment