Schedule workflow randomly skipping execution cycles

[xtyinbiao]

Expected Behavior

this is create schedule function:

public String createSchedule(TemporalScheduleBO temporalScheduleBO) {
try {
ScheduleClient client = ScheduleClient.newInstance(
service(), ScheduleClientOptions.newBuilder().setNamespace(temporalScheduleBO.getNamespace()).build());

        Schedule schedule =
                Schedule.newBuilder()
                        .setAction(ScheduleActionStartWorkflow.newBuilder()
                                .setArguments(temporalScheduleBO.getParams())
                                .setWorkflowType(temporalScheduleBO.getWorkflowTypeName())
                                .setOptions(WorkflowOptions.newBuilder()
                                        .setTaskQueue(temporalScheduleBO.getTaskQueueName())
                                        .setWorkflowId(temporalScheduleBO.getWorkflowId())
                                        .build())
                                .build())
                        .setSpec(ScheduleSpec.newBuilder()
                                .setCronExpressions(temporalScheduleBO.getCronString())

// .setIntervals(temporalScheduleBO.getInterval())
// .setCalendars(temporalScheduleBO.getCalendar())
.build())
.setPolicy(SchedulePolicy.newBuilder()
.setOverlap(ScheduleOverlapPolicy.SCHEDULE_OVERLAP_POLICY_ALLOW_ALL)
.setCatchupWindow(Duration.ofSeconds(30))
.build())
.setState(ScheduleState.newBuilder()
.setLimitedAction(false)
.setRemainingActions(30)
.build())
.build();

        ScheduleHandle handle =
                client.createSchedule(temporalScheduleBO.getName(), schedule, ScheduleOptions.newBuilder().build());
        return handle.getId();

    } catch (ScheduleAlreadyRunningException e) {
        log.info("schedule already exists, is running, skip create: {}", temporalScheduleBO.getWorkflowId());
        return temporalScheduleBO.getName();
    } catch (ScheduleException e) {
        if(e.getCause().getMessage().contains("ALREADY_EXISTS")) {
            log.info("schedule already exists, skip create: {}", temporalScheduleBO.getWorkflowId());
            return temporalScheduleBO.getName();
        } else {
            throw e;
        }
    }
}

Expect to run every 5 minutes

Actual Behavior

Randomly miss a run

Steps to Reproduce the Problem

Specifications

• Version: 2.22.3
• Platform: Linux temporal-worker-5f4dfdbb94-dfv8l 5.4.228-1.el7.elrepo.x86_64 Add workspace file to gitignore for vscode development #1 SMP Thu Dec 15 18:03:40 EST 2022 x86_64 Linux

[xtyinbiao]

[xtyinbiao]

{"level":"warn","ts":"2024-11-15T12:56:18.462Z","msg":"Schedule missed catchup window","service":"worker","Namespace":"eagleeye","TaskQueue":"temporal-sys-per-ns-tq","WorkerID":"server-worker@1@temporal-worker-86d8bb7b8c-jhrdp@eagleeye","WorkflowType":"temporal-sys-scheduler-workflow","WorkflowID":"temporal-sys-scheduler:18910#shprod","RunID":"473bc143-08d9-41d5-9d75-c91d96fae837","Attempt":1,"wf-namespace":"eagleeye","schedule-id":"18910#shprod","now":"2024-11-15T12:56:18.226Z","time":"2024-11-15T12:55:00.000Z","logging-call-at":"workflow.go:223"}

how to avoid Schedule missed？

[dnr]

Please use the forums https://community.temporal.io/ or slack https://t.mp/slack for support.

It looks like your cluster has significant performance problems if it's regularly starting scheduled workflows 5 or 6 or 14 seconds after the scheduled time. It's not too surprising that sometimes the delay might exceed the 30 second catchup window that you set. Take a look at some of the self-hosted scaling resources like https://temporal.io/blog/scaling-temporal-the-basics

If you have a large number of schedules in your namespace, you can tune the internal worker similarly to an external worker using the dynamic config setting worker.perNamespaceWorkerOptions

[xtyinbiao]

Please use the forums https://community.temporal.io/ or slack https://t.mp/slack for support.

It looks like your cluster has significant performance problems if it's regularly starting scheduled workflows 5 or 6 or 14 seconds after the scheduled time. It's not too surprising that sometimes the delay might exceed the 30 second catchup window that you set. Take a look at some of the self-hosted scaling resources like https://temporal.io/blog/scaling-temporal-the-basics

If you have a large number of schedules in your namespace, you can tune the internal worker similarly to an external worker using the dynamic config setting worker.perNamespaceWorkerOptions

when i set setJitter(Duration.ofSeconds(30)) then no missed, it's should be heavy load.
At first I set up backfill but it had no effect.
Is there any way to accept delays to avoid missing out rather than scaling ？

[dnr]

setCatchupWindow(Duration.ofSeconds(30)) means "don't start a workflow if it would be more than 30 seconds late". You can leave that at the default (1 year) and then you'll just get a longer delay if the cluster is overloaded.

[xtyinbiao]

setCatchupWindow(Duration.ofSeconds(30)) means "don't start a workflow if it would be more than 30 seconds late". You can leave that at the default (1 year) and then you'll just get a longer delay if the cluster is overloaded.

Thank you very much for your reply