How to tell bull queue that the worker is running the job, and not stalled

[vickyRathee]

I have long processing crawling job, it may take hours or days to complete a job.

const queueSettings: AdvancedSettings = {
  lockDuration: 60 * 1000, // Key expiration time for job locks. 5 minutes
  lockRenewTime: 30 * 1000, // Interval on which to acquire the job lock. 1 minutes
  stalledInterval: 60 * 1000, // How often check for stalled jobs (use 0 for never checking).
  maxStalledCount: 1, // Max amount of times a stalled job will be re-processed.
  guardInterval: 5000, // Poll interval for delayed jobs and added jobs.
  retryProcessDelay: 5000, // delay before processing next job in case of internal error.
  backoffStrategies: {}, // A set of custom backoff strategies keyed by name.
  drainDelay: 5, // A timeout for when the queue is in drained state (empty waiting for jobs).
};

const options: QueueOptions = {
  settings: queueSettings,
};

const jobsQueue = new Queue("Jobs", REDIS_URL, options);

I am reporting the progress { } after each request in loop and extended the stalledInterval to 60s, still the job is stalled after around a minute or two while the worker is running the job and no errors.

const urls = ["https://example.com", ""https://example.com".... ];
const data = { total: urls.length, processed: 0 , failed: 0, succeeded: 0};

for (let i = 0; i <urls .length; i++){
   data.processed += 1;
   // Crawl
   // Extract
   // Upload

  data.succeeded += 1;

  // Report progress
  job.progress(data).catch((err) => {
    console.error(err);
  });
}

[manast]

Are you processing the jobs manually like this: https://github.com/OptimalBits/bull/blob/develop/PATTERNS.md#manually-fetching-jobs or using the standard processor function?
You normally do not need to change the stalledInterval setting, and it is better that you do not do it, it has nothing to do with the time it takes for your job to complete. In any case, it is recommended to break your large jobs into smaller jobs, by definition a job can fail, would you lose several days of processing if it fails right at the end? By breaking into smaller jobs you do not have this risk and also you get better visibility.