Exception in CheckExists

[viveknuna]

I am getting the below exception intermittently when calling the method below method, Could you please help?

I am also getting similar exceptions in StoreTrigger, StoreJob, and JobExists as well.

services.AddSingleton<IQuartzSchedulersManager, QuartzSchedulersManager>();
services.AddTransient<IQuartzBuilder, QuartzBuilder>();

        private readonly IQuartzBuilder _quartzBuilder;

        /// <summary>
        /// 
        /// </summary>
        /// <param name="logger"></param>
        /// <param name="context"></param>
        /// <param name="quartzBuilder"></param>
        /// <param name="triggerSelector"></param>
        public InternalJobsScheduler(
            IExecutionContextFacade context, 
            IQuartzBuilder quartzBuilder)
        {
            _context = context;
            _quartzBuilder = quartzBuilder;
        }


 private async Task<IScheduler> GetSchedulerForTenant(string tenantId, bool skipCreateIfNotExists)
        {
            try
            {
                _logger.LogInfo($"Quartz - QuartzBuilder.GetSchedulerForTenant tenant '{tenantId}'");
                if (_quartzSchedulerManager.Schedulers.TryGetValue(tenantId, out var scheduler))
                {
                    _logger.LogInfo($"Quartz - QuartzBuilder.GetSchedulerForTenant scheduler for '{tenantId}' name: '{scheduler.SchedulerName}' instance: '{scheduler.SchedulerInstanceId}'");
                    return scheduler;
                }

var scheduler = await _quartzBuilder.GetSchedulerForTenant(_context.TenantId.ToString());
await scheduler.CheckExists(jobKey); //exception on this line

Quartz.net version: 3.5.0
.Net Version: .Net 6

The clean version of exception details:

problemId: System.ComponentModel.Win32Exception at Quartz.Impl.AdoJobStore.JobStoreSupport+d__202.MoveNext

type: System.ComponentModel.Win32Exception

assembly: Quartz, Version=3.5.0.0, Culture=neutral, PublicKeyToken=f6b8c98a402cc8a4

method: Quartz.Impl.AdoJobStore.JobStoreSupport+d__202.MoveNext

outerType: Quartz.JobPersistenceException

outerMessage: Couldn't check for existence of job: Execution Timeout Expired. The timeout period elapsed prior to completion of the operation or the server is not responding.

outerAssembly: Quartz, Version=3.5.0.0, Culture=neutral, PublicKeyToken=f6b8c98a402cc8a4

outerMethod: Quartz.Impl.AdoJobStore.JobStoreSupport+d__202.MoveNext

innermostType: System.ComponentModel.Win32Exception

innermostMessage: Unknown error 258

severityLevel: 3

Exception: Couldn't check for existence of job: Execution Timeout Expired. The timeout period elapsed prior to completion of the operation or the server is not responding.

FormattedMessage: AutomationDiagnostics : Management.Quartz.Service - InternalScheduleCommandHandler (x1) - Exception thrown when handling the schedule command: Couldn't check for existence of job: Execution Timeout Expired. The timeout period elapsed prior to completion of the operation or the server is not responding. : automation-management-quartz-deployment-67dfcc5c6-gkhbd : Couldn't check for existence of job: Execution Timeout Expired. The timeout period elapsed prior to completion of the operation or the server is not responding. : Execution Timeout Expired. The timeout period elapsed prior to completion of the operation or the server is not responding. / Unknown error 258 : at Quartz.Impl.AdoJobStore.JobStoreSupport.CheckExists(ConnectionAndTransactionHolder conn, JobKey jobKey, CancellationToken cancellationToken) at Quartz.Impl.AdoJobStore.JobStoreSupport.ExecuteInNonManagedTXLock[T](String lockName, Func2 txCallback, Func3 txValidator, Nullable1 requestorId, CancellationToken cancellationToken) at Quartz.Impl.AdoJobStore.JobStoreSupport.ExecuteInNonManagedTXLock[T](String lockName, Func2 txCallback, Func3 txValidator, Nullable1 requestorId, CancellationToken cancellationToken) at Automation.Management.Adapters.Repositories.InternalScheduler.InternalJobsScheduler.Schedule(InternalScheduleType type, String jobId, String whenOptions, String payload, IDiagnostics logger) in /src/Management/Automation.Management.Adapters/Repositories/InternalScheduler/InternalJobsScheduler.cs:line 136 at Automation.Management.Quartz.Service.Handlers.InternalScheduleCommandHandler.HandleMessage(InternalScheduleCommand message, IReceivedMessageContext context, CancellationToken cancellationToken) in /src/Management/Automation.Management.Quartz.Service/Handlers/InternalScheduleCommandHandler.cs:line 57 :

InnerExceptions: Execution Timeout Expired. The timeout period elapsed prior to completion of the operation or the server is not responding. / Unknown error 258

Message: Management.Quartz.Service - InternalScheduleCommandHandler (x1) - Exception thrown when handling the schedule command: Couldn't check for existence of job: Execution Timeout Expired. The timeout period elapsed prior to completion of the operation or the server is not responding.

StackTrace at Quartz.Impl.AdoJobStore.JobStoreSupport.CheckExists(ConnectionAndTransactionHolder conn, JobKey jobKey, CancellationToken cancellationToken) at Quartz.Impl.AdoJobStore.JobStoreSupport.ExecuteInNonManagedTXLock[T](String lockName, Func2 txCallback, Func3 txValidator, Nullable1 requestorId, CancellationToken cancellationToken) at Quartz.Impl.AdoJobStore.JobStoreSupport.ExecuteInNonManagedTXLock[T](String lockName, Func2 txCallback, Func3 txValidator, Nullable1 requestorId, CancellationToken cancellationToken) at Automation.Management.Adapters.Repositories.InternalScheduler.InternalJobsScheduler.Schedule(InternalScheduleType type, String jobId, String whenOptions, String payload, IDiagnostics logger) in /src/Management/Automation.Management.Adapters/Repositories/InternalScheduler/InternalJobsScheduler.cs:line 136 at Automation.Management.Quartz.Service.Handlers.InternalScheduleCommandHandler.HandleMessage(InternalScheduleCommand message, IReceivedMessageContext context, CancellationToken cancellationToken) in /src/Management/Automation.Management.Quartz.Service/Handlers/InternalScheduleCommandHandler.cs:line 57

Error Details: (this is CSV, you can copy and open it in an excel sheet for better readability.
"timestamp [UTC]",problemId,handledAt,type,message,assembly,method,outerType,outerMessage,outerAssembly,outerMethod,innermostType,innermostMessage,innermostAssembly,innermostMethod,severityLevel,details,itemType,customDimensions,customMeasurements,"operation_Name","operation_Id","operation_ParentId","operation_SyntheticSource","session_Id","user_Id","user_AuthenticatedId","user_AccountId","application_Version","client_Type","client_Model","client_OS","client_IP","client_City","client_StateOrProvince","client_CountryOrRegion","client_Browser","cloud_RoleName","cloud_RoleInstance",appId,appName,iKey,sdkVersion,itemId,itemCount,"_ResourceId"
"6/19/2023, 9:12:37.199 AM","System.ComponentModel.Win32Exception at Quartz.Impl.AdoJobStore.JobStoreSupport+d__202.MoveNext",,"System.ComponentModel.Win32Exception",,"Quartz, Version=3.5.0.0, Culture=neutral, PublicKeyToken=f6b8c98a402cc8a4","Quartz.Impl.AdoJobStore.JobStoreSupport+d__202.MoveNext","Quartz.JobPersistenceException","Couldn't check for existence of job: Execution Timeout Expired. The timeout period elapsed prior to completion of the operation or the server is not responding.","Quartz, Version=3.5.0.0, Culture=neutral, PublicKeyToken=f6b8c98a402cc8a4","Quartz.Impl.AdoJobStore.JobStoreSupport+d__202.MoveNext","System.ComponentModel.Win32Exception","Unknown error 258",,,3,"[{""id"":""62233295"",""outerId"":""0"",""type"":""Quartz.JobPersistenceException"",""message"":""Couldn't check for existence of job: Execution Timeout Expired. The timeout period elapsed prior to completion of the operation or the server is not responding."",""parsedStack"":[{""level"":0,""method"":""Quartz.Impl.AdoJobStore.JobStoreSupport+d__202.MoveNext"",""assembly"":""Quartz, Version=3.5.0.0, Culture=neutral, PublicKeyToken=f6b8c98a402cc8a4"",""line"":0},{""level"":1,""method"":""System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw"",""assembly"":""System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e"",""line"":0},{""level"":2,""method"":""System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess"",""assembly"":""System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e"",""line"":0},{""level"":3,""method"":""System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification"",""assembly"":""System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e"",""line"":0},{""level"":4,""method"":""Quartz.Impl.AdoJobStore.JobStoreSupport+d__2731.MoveNext"",""assembly"":""Quartz, Version=3.5.0.0, Culture=neutral, PublicKeyToken=f6b8c98a402cc8a4"",""line"":0},{""level"":5,""method"":""System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw"",""assembly"":""System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e"",""line"":0},{""level"":6,""method"":""Quartz.Impl.AdoJobStore.JobStoreSupport+<ExecuteInNonManagedTXLock>d__2731.MoveNext"",""assembly"":""Quartz, Version=3.5.0.0, Culture=neutral, PublicKeyToken=f6b8c98a402cc8a4"",""line"":0},{""level"":7,""method"":""System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw"",""assembly"":""System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e"",""line"":0},{""level"":8,""method"":""System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess"",""assembly"":""System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e"",""line"":0},{""level"":9,""method"":""System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification"",""assembly"":""System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e"",""line"":0},{""level"":10,""method"":""Automation.Management.Adapters.Repositories.InternalScheduler.InternalJobsScheduler+d__12.MoveNext"",""assembly"":""Automation.Management.Adapters, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null"",""line"":136,""fileName"":""/src/Management/Automation.Management.Adapters/Repositories/InternalScheduler/InternalJobsScheduler.cs""},{""level"":11,""method"":""System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw"",""assembly"":""System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e"",""line"":0},{""level"":12,""method"":""System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess"",""assembly"":""System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e"",""line"":0},{""level"":13,""method"":""System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification"",""assembly"":""System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e"",""line"":0},{""level"":14,""method"":""Automation.Management.Quartz.Service.Handlers.InternalScheduleCommandHandler+d__4.MoveNext"",""assembly"":""Automation.Management.Quartz.Service, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null"",""line"":57,""fileName"":""/src/Management/Automation.Management.Quartz.Service/Handlers/InternalScheduleCommandHandler.cs""}],""severityLevel"":""Error""},{""id"":""62371830"",""outerId"":""62233295"",""type"":""Microsoft.Data.SqlClient.SqlException"",""message"":""Execution Timeout Expired. The timeout period elapsed prior to completion of the operation or the server is not responding."",""parsedStack"":[{""level"":0,""method"":""Microsoft.Data.SqlClient.SqlCommand+<>c.b__208_0"",""assembly"":""Microsoft.Data.SqlClient, Version=5.0.0.0, Culture=neutral, PublicKeyToken=23ec7fc2d6eaa4a5"",""line"":0},{""level"":1,""method"":""System.Threading.Tasks.ContinuationResultTaskFromResultTask2.InnerInvoke"",""assembly"":""System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e"",""line"":0},{""level"":2,""method"":""System.Threading.ExecutionContext.RunInternal"",""assembly"":""System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e"",""line"":0},{""level"":3,""method"":""System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw"",""assembly"":""System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e"",""line"":0},{""level"":4,""method"":""System.Threading.Tasks.Task.ExecuteWithThreadLocal"",""assembly"":""System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e"",""line"":0},{""level"":5,""method"":""System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw"",""assembly"":""System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e"",""line"":0},{""level"":6,""method"":""System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess"",""assembly"":""System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e"",""line"":0},{""level"":7,""method"":""System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification"",""assembly"":""System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e"",""line"":0},{""level"":8,""method"":""Quartz.Impl.AdoJobStore.StdAdoDelegate+<JobExists>d__59.MoveNext"",""assembly"":""Quartz, Version=3.5.0.0, Culture=neutral, PublicKeyToken=f6b8c98a402cc8a4"",""line"":0},{""level"":9,""method"":""System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw"",""assembly"":""System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e"",""line"":0},{""level"":10,""method"":""System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess"",""assembly"":""System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e"",""line"":0},{""level"":11,""method"":""System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification"",""assembly"":""System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e"",""line"":0},{""level"":12,""method"":""Quartz.Impl.AdoJobStore.JobStoreSupport+<CheckExists>d__202.MoveNext"",""assembly"":""Quartz, Version=3.5.0.0, Culture=neutral, PublicKeyToken=f6b8c98a402cc8a4"",""line"":0}],""severityLevel"":""Error""},{""id"":""29410686"",""outerId"":""62371830"",""type"":""System.ComponentModel.Win32Exception"",""message"":""Unknown error 258"",""severityLevel"":""Error""}]",exception,"{""FormattedMessage"":""AutomationDiagnostics : Management.Quartz.Service - InternalScheduleCommandHandler (x1) - Exception thrown when handling the schedule command: Couldn't check for existence of job: Execution Timeout Expired. The timeout period elapsed prior to completion of the operation or the server is not responding. : automation-management-quartz-deployment-67dfcc5c6-gkhbd : Couldn't check for existence of job: Execution Timeout Expired. The timeout period elapsed prior to completion of the operation or the server is not responding. : Execution Timeout Expired. The timeout period elapsed prior to completion of the operation or the server is not responding. / Unknown error 258 : at Quartz.Impl.AdoJobStore.JobStoreSupport.CheckExists(ConnectionAndTransactionHolder conn, JobKey jobKey, CancellationToken cancellationToken)\n at Quartz.Impl.AdoJobStore.JobStoreSupport.ExecuteInNonManagedTXLock[T](String lockName, Func2 txCallback, Func3 txValidator, Nullable1 requestorId, CancellationToken cancellationToken)\n at Quartz.Impl.AdoJobStore.JobStoreSupport.ExecuteInNonManagedTXLock[T](String lockName, Func2 txCallback, Func3 txValidator, Nullable1 requestorId, CancellationToken cancellationToken)\n at Automation.Management.Adapters.Repositories.InternalScheduler.InternalJobsScheduler.Schedule(InternalScheduleType type, String jobId, String whenOptions, String payload, IDiagnostics logger) in /src/Management/Automation.Management.Adapters/Repositories/InternalScheduler/InternalJobsScheduler.cs:line 136\n at Automation.Management.Quartz.Service.Handlers.InternalScheduleCommandHandler.HandleMessage(InternalScheduleCommand message, IReceivedMessageContext context, CancellationToken cancellationToken) in /src/Management/Automation.Management.Quartz.Service/Handlers/InternalScheduleCommandHandler.cs:line 57 :"",""OriginalFormat"":""{ServiceName} : {Message} : {HostName} : {Exception} : {InnerExceptions} : {StackTrace} :"",""CategoryName"":""Automation.AutomationManager.Common.Diagnostics.AutomationDiagnostics"",""_MS.ProcessedByMetricExtractors"":""(Name:'Exceptions', Ver:'1.1')"",""CorrelationId"":""ac4d1e96-3476-4bc3-96e6-10a17544bba5"",""TenantId"":""21a4f46d-8abf-40f7-86a0-6de96d319a03"",""Subject"":""Automation.Management.InternalScheduleCommand"",""MessageId"":""3a91e6b7-122d-4095-8907-4efa4d3d9a47"",""StackTrace"":""at Quartz.Impl.AdoJobStore.JobStoreSupport.CheckExists(ConnectionAndTransactionHolder conn, JobKey jobKey, CancellationToken cancellationToken)\n at Quartz.Impl.AdoJobStore.JobStoreSupport.ExecuteInNonManagedTXLock[T](String lockName, Func2 txCallback, Func3 txValidator, Nullable1 requestorId, CancellationToken cancellationToken)\n at Quartz.Impl.AdoJobStore.JobStoreSupport.ExecuteInNonManagedTXLock[T](String lockName, Func2 txCallback, Func3 txValidator, Nullable`1 requestorId, CancellationToken cancellationToken)\n at Automation.Management.Adapters.Repositories.InternalScheduler.InternalJobsScheduler.Schedule(InternalScheduleType type, String jobId, String whenOptions, String payload, IDiagnostics logger) in /src/Management/Automation.Management.Adapters/Repositories/InternalScheduler/InternalJobsScheduler.cs:line 136\n at Automation.Management.Quartz.Service.Handlers.InternalScheduleCommandHandler.HandleMessage(InternalScheduleCommand message, IReceivedMessageContext context, CancellationToken cancellationToken) in /src/Management/Automation.Management.Quartz.Service/Handlers/InternalScheduleCommandHandler.cs:line 57"",""InnerExceptions"":""Execution Timeout Expired. The timeout period elapsed prior to completion of the operation or the server is not responding. / Unknown error 258"",""ServiceName"":""AutomationDiagnostics"",""Exception"":""Couldn't check for existence of job: Execution Timeout Expired. The timeout period elapsed prior to completion of the operation or the server is not responding."",""Message"":""Management.Quartz.Service - InternalScheduleCommandHandler (x1) - Exception thrown when handling the schedule command: Couldn't check for existence of job: Execution Timeout Expired. The timeout period elapsed prior to completion of the operation or the server is not responding."",""HostName"":""automation-management-quartz-deployment-67dfcc5c6-gkhbd""}",,,,,,,,,,"1.0.0",PC,,,"0.0.0.0",Washington,Virginia,"United States",,"automation-management-quartz-service","automation-management-quartz-deployment-67dfcc5c6-gkhbd","4c9b4c87-4c2f-4cf3-8927-65d1a54d5f20","AI-NVU-PRD-DEFAUL-0X0-88","532461f3-5843-4aba-a3b7-b6a88308fd60","il:2.21.0-429","756caad0-0e81-11ee-9a43-d72308a9917b",1,

[jafin]

Execution Timeout Expired. The timeout period elapsed prior to completion of the operation or the server is not responding.

Is your database under load? That looks like a typical db response timeout.
Perhaps rebuild indexes, update statistics on the db.

How many rows are in your qrtz_job_details table?

[viveknuna]

@jafin do you think, upgrading the quartz.net to latest version could solve this problem? I have other tables also in this database apart from the quartz tables. So do you suggest having a separate database for the quartz tables only?

There are 46 rows in table qrtz_job_details at this moment.

[jafin]

@viveknuna I don't think there is anything specific in later versions that would help. Certainly can't hurt to update to a later 3.x package. I'd review here, https://learn.microsoft.com/en-us/troubleshoot/sql/database-engine/connect/timeout-expired-error

Are you seeing timeouts for other parts of your app?
Are you running a cluster? (perhaps it's lock contention)

[viveknuna]

@jafin Is there a way to get rid of the quartz dependency on the SQL database?

Yes, this service is running in a cluster. There are no timeout issues in the other part of this particular app.

But I could see similar exceptions for the other service as well which I don't think is using quartz library.

[lahma]

https://stackoverflow.com/questions/76507294/exception-in-checkexists-of-quartz-net

[viveknuna]

@lahma I have got no response in StackOverflow unfortunately.

[lahma]

And even more unfortunate that you did mention about cross-posting.

[viveknuna]

@lahma I posted on StackOverflow after posting to GitHub, nothing intentional.