-
Notifications
You must be signed in to change notification settings - Fork 1.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Maximum hangfire job duration for long running jobs #1197
Comments
What version of Hangfire are you running and what storage option do you use? I keep having a similar issue with version 1.6.19 using MongoDB and my long running background job keeps getting canceled after 30 minutes approximately, might be the same issue |
I also encountered a similar issue. After exactly 30 minutes, a second thread comes in and starts executing the same job. If I just ignore the cancellation token, the first thread keeps going. I'm using the In-memory storage. Let's make an example. I have this job that's supposed to run for 2 hours. It appends a line of text to a file every 3 seconds.
When it's run, these lines of text will appear in the text file. First column is the timestamp, second is the thread id, third is the hash code of the job instance.
As I said, after exactly 30 minutes, a second thread executes the same job. I can tell since I see this in the text file.
As you can see, there's now two threads executing two instances of my job (and this is a problem in my case). |
I am experiencing the same issue, but struggling to reproduce it locally. I will try @BrightSoul's reproduction. Here is a screenshot of what I am experiencing (server name removed for confidentiality). You can see sometimes it is not exactly 30 minutes. I believe this is because Hangfire tries to recruit another worker thread but they are all busy processing other jobs and it must wait for one to become free. |
I managed to solve my problem with this code (it's an ASP.NET Core application).
By setting FetchNextJobTimeout, I can control how long a job can run for before Hangfire starts executing it again on another thread. |
Same issue with MySqlStorage. Thank for this post for the information. |
I have the same problem in Hangfire 1.6.21 with MySqlStorage |
I have the same issue in Hangfire 1.7.1 with Hangfire.PostgreSql 1.5.0. |
I think that the implementation of MySqlStorage and Hangfire.PostgreSql are wrong because on SqlServer, I haven't this issue without configuring anithing. With MySqlStorage, I finaly use the 'InvisibilityTimeout'... even if it's deprecated... it's working :( |
@dhnnjy In
|
I'm using Hangfire.Mongo, this works for me too |
@stevendesrochers even now? |
@ehouarn-perret From what i can tell Maybe we'll see this in a future release of Hangfire.PostgreSql |
@stevendesrochers I'm a little confused. |
So what about for Hangfire Pro w/ redis? How can i configure specific jobs to not be auto restarted? Some jobs just take 2 hours to finish. |
Hey man,.how did you get MysqlStorage to work in your ASP.NET CORE app,. mine fails to create some tables, such hangfire_state and hangfire_set? Can you help a brother out, please? |
this is different. InvisibilityTimeout causes Hangfire to cancel and then restart the job. DisableConcurrentExecution just keeps 1 job instance despite attempts to queue more. (but it still wil be cancelled after InvisibilityTimeout) |
Please try using the SkipWhenPreviousJobIsRunningAttribute job filter from the referenced gist |
This was an absolute nightmare to debug; I had no idea why some of my jobs were behaving strangely and being re-spawned for no apparent reason. |
Hello,
In our project we have a few Hangfire jobs which take a lot time (possibly more than an hour). We noticed a strange behaviour for that jobs, only partial execution is complete (for example only some of the data is removed instead of everything). I was wondering if there's a default timeout on the time job is executing. If yes, could you direct me to the setting where this can be changed (either globally or for specific job), I couldn't find in the docs.
Thanks,
Marcin
The text was updated successfully, but these errors were encountered: