Multi core running app duplicate schedules saved with the same name

[calebeaires]

I'm submitting a...

[ ] Regression 
[ ] Bug report
[x] Feature request
[ ] Documentation issue or request
[ ] Support request => Please do not submit support request here, instead post your question on Stack Overflow.

Current behavior

Using PM2 with the option -i the app runs a multi multi thread, but when schedule is created at differents moments at a node running app under multi core, some schedule is created twice and more.

Expected behavior

Use redis, like @nestjs/websockets do, to save/create schedules

[twister21]

Unfortunately, node-cron doesn't have a Redis integration (see kelektiv/node-cron#175).
I suggest to depend on a more advanced package like https://github.com/actionhero/node-resque#job-schedules.

I tried to use the Bull package as a workaround. You can define the cronejob options via the repeat property and start the queue immediately after Nest's initialization is completed. However, this isn't really the intended use case.

[twister21]

Shouldn't Nest.js have a single package than can handle jobs and recurring jobs? Currently, there is @nestjs/bull for one-time jobs and @nestjs/schedule for cron jobs, although bull can handle both.(https://github.com/OptimalBits/bull/blob/master/REFERENCE.md#queueadd)

For example, @nestjs/bull could create an abstraction for cron jobs, so that a new bull job (configured with the cron option) is automatically scheduled each time the Nest.js app is started.

It would be great to haves a concept like ActiveJob (Ruby on Rails) for Nest.js.
A job can either be started manually (e.g. on a request) or it can be configured to start automatically. (https://edgeguides.rubyonrails.org/active_job_basics.html)

[calebeaires]

I come up to using bull. Using the process as anonymous I found out that I could replicate what I need from schedule and cron , this is how my proccess looks like:

import { Process, Processor } from '@nestjs/bull';
import { Logger } from '@nestjs/common';
import { Job } from 'bull';

@Processor('audio')
export class AudioProcessor {
  private readonly logger = new Logger(AudioProcessor.name);

  @Process('*')
  async transcode(job: Job<unknown>) {
    this.logger.debug(job.data);
  }
}

the @process('*') decorator with the * make possible call distinct tasks. The rest is just logic.

[twister21]

Do you start the cron job in Nest's onApplicationBootstrap lifecycle event?

[calebeaires]

in my case I sue redis since I need to make tasks uniq and since is tis connected the task is already registered and started every time the app is launched.

Here is the full simple example you can extend with your logic

The module

import { BullModule } from '@nestjs/bull';
import { Module } from '@nestjs/common';
import { AudioController } from './audio.controller';
import { AudioProcessor } from './audio.processor';

@Module({
  imports: [
    BullModule.registerQueue({
      name: 'audio',
    }),
  ],
  controllers: [AudioController],
  providers: [AudioProcessor],
})
export class AudioModule {}

the controller, if you need it to start some taks

import { InjectQueue } from '@nestjs/bull';
import { Controller, Get, Post } from '@nestjs/common';
import { Queue } from 'bull';

@Controller('audio')
export class AudioController {
  constructor(@InjectQueue('audio') private readonly audioQueue: Queue) {
  }

  @Get('transcode')
  async transcode() {
    // this.audioQueue.empty().then(function() {
    //   console.log('done removing jobs');
    // });
    //

    await this.audioQueue.add('teste', {
      file: 'teste-name',
    }, {
      repeat: {
        cron: '* * * * * *',
      },
    });

    return { status: 'ok' };

  }
}

the generic process bull creator

import { Process, Processor } from '@nestjs/bull';
import { Logger } from '@nestjs/common';
import { Job } from 'bull';

@Processor('audio')
export class AudioProcessor {
  private readonly logger = new Logger(AudioProcessor.name);

  @Process('*')
  async transcode(job: Job<unknown>) {
    this.logger.debug(job.data);
  }
}

[twister21]

I need to have a clean-up cron job to delete inactive user accounts. So, it isn't bound to a specific request, but should be executed every 24 hours.

[calebeaires]

To make some service be called every time you turn on the app, just use something like this

// from main.ts
    await app
        .select(CronModule)
        .get(CronService, { strict: true })
        .someMethodHere(someParamIfYouNeed);

[twister21]

I'm using the onApplicationBootstrap method, which seems to do the same.
https://docs.nestjs.com/fundamentals/lifecycle-events

However, I would prefer that Nest.js has a solution to register cron jobs with bull automatically.
#312 (comment)

[kamilmysliwiec]

The solution proposed in this issue (using @nestjs/bull for such jobs when there are multiple apps/processes/threads running in parallel) is valid. node-schedule used by this package doesn't provide any locks-related functionality ootb (so it would have to be added on top, or another package like e.g., node-resque would have to be used).

I'm using the onApplicationBootstrap method, which seems to do the same.

I would personally suggest using the onApplicationBootstrap hook as well. Either approach should work OK though.

Shouldn't Nest.js have a single package than can handle jobs and recurring jobs? Currently, there is @nestjs/bull for one-time jobs and @nestjs/schedule for cron jobs, although bull can handle both.

We wanted to provide integrations for both packages as they are heavily used by the framework users.

[sirbully]

How would I set up the cron job in the onApplicationBootstrap hook? Do I still use @nestjs/schedule?

[kamilmysliwiec]

I'd not recommend using the @nestjs/schedule in this case.

How would I set up the cron job in the onApplicationBootstrap hook?

Depends on what you want to accomplish. If you want to use Bull's repeatable jobs, just add logic for registering such a job in the onApplicationBootstrap hook.