Global Daily Blackout

Global Daily Blackout

You can create a global daily blackout window in xyOps by scheduling two events that call the xyOps REST API. One event disables the scheduler at the beginning of the blackout, and a second delayed event enables it again at the end.

This is useful when you want all scheduled activity to pause during a predictable daily window, such as overnight maintenance, database backups, or a period of reduced infrastructure capacity.

Important

Disabling the scheduler does not abort jobs that are already running. It only prevents new scheduled jobs from launching. This is by design -- active jobs are allowed to finish normally.

How It Works

The two events use the built-in Shell Plugin and curl to call the update_global_state API. The API request changes the global scheduler.enabled state.

There is one important detail: the event that re-enables the scheduler cannot be scheduled to launch during the blackout. Once the scheduler is disabled, it cannot launch that event.

The solution is to schedule the enable event shortly before the blackout starts, then add a Delay trigger modifier so its job waits until the blackout ends. The delayed job is created while the scheduler is still enabled, and disabling the scheduler does not abort it. When its delay expires, it runs and re-enables scheduling.

For example, to create a daily blackout from 4:00 AM through 5:00 AM:

Event	Scheduled Time	Delay	API Call Time
Enable Scheduler	3:59 AM	1 hour, 1 minute	5:00 AM
Disable Scheduler	4:00 AM	None	4:00 AM

The one-minute head start ensures that the delayed enable job is safely created before the scheduler is disabled.

Step 1: Create an Administrator API Key

The update_global_state API is restricted to administrators, so the events need an API Key with full administrator privileges.

1. Open API Keys in the xyOps sidebar.
2. Click New API Key.
3. Give it a clear title, such as Global Scheduler Toggle.
4. In Privileges, select Administrator.
5. Save the API Key.
6. Copy the generated key value when it is displayed.

The full API Key is only shown once. Keep it handy for the next step, but do not paste it directly into either event script.

Step 2: Store the API Key in a Secret Vault

Store the API Key in a Secret Vault, so xyOps can inject it into the jobs without exposing it in the event definitions or scripts.

1. Open Secrets in the xyOps sidebar.
2. Click New Vault.
3. Give it a title, such as Global Scheduler Credentials.
4. Add a variable with these values:
   • Variable Name: XYOPS_API_KEY
   • Variable Value: paste the API Key from Step 1.
5. Make sure the vault is enabled, then save it.

XYOPS_API_KEY is only an example variable name. You may use any valid environment variable name, but make sure both scripts use the same name.

We will assign this vault to both events after they are created.

Step 3: Create the Enable Scheduler Event

Create the enable event first. This event will launch just before the blackout, wait in a delayed state, and then re-enable scheduling at the end of the window.

1. Open Events in the xyOps sidebar and create a new event.
2. Give it a title such as Enable Scheduler.
3. Select the built-in Shell Plugin.
4. Select the target server or group that should execute the API call when the delay expires.
5. Paste this into Script Source:

#!/bin/bash
set -eu

# Ask the xyOps conductor to enable its global scheduler.
curl --fail --silent --show-error \
	-X POST \
	-H "Content-Type: application/json" \
	-H "X-API-Key: $XYOPS_API_KEY" \
	--data '{"scheduler.enabled":true}' \
	"$JOB_BASE_URL/api/app/update_global_state/v1"

Every xyOps job receives JOB_BASE_URL, which points back to the current primary conductor. The Secret Vault will provide XYOPS_API_KEY when the job runs.

Add the Enable Schedule

Add a Schedule trigger for one minute before the blackout begins. For the 4:00 AM example, schedule this event every day at 3:59 AM.

Select the timezone that should govern your blackout window. Both events should use the same timezone.

Add the Delay Modifier

Add a Delay trigger modifier to the same event. Set the delay so the job becomes ready exactly when the blackout should end.

For a blackout beginning at 4:00 AM and ending at 5:00 AM, the enable event is scheduled at 3:59 AM, so its delay should be: 3660 seconds (i.e. 1 hour, 1 minute).

This causes the job to be created at 3:59 AM, remain in its start-delay state throughout the blackout, and execute the API call at 5:00 AM.

Important

Do not schedule this event directly for 5:00 AM. The global scheduler will be disabled at that time, so the event would never launch. It must launch before the blackout and wait using the Delay modifier.

Step 4: Create the Disable Scheduler Event

Now create the event that begins the blackout.

1. Create another event and give it a title such as Disable Scheduler.
2. Select the built-in Shell Plugin.
3. Select a reliable target server.
4. Paste this into Script Source:

#!/bin/bash
set -eu

# Ask the xyOps conductor to disable its global scheduler.
curl --fail --silent --show-error \
	-X POST \
	-H "Content-Type: application/json" \
	-H "X-API-Key: $XYOPS_API_KEY" \
	--data '{"scheduler.enabled":false}' \
	"$JOB_BASE_URL/api/app/update_global_state/v1"

Add a Schedule trigger for the beginning of the blackout. For this example, schedule the event every day at 4:00 AM, using the same timezone as the enable event.

This event does not need a Delay modifier.

Step 5: Assign the Secret Vault to Both Events

Now grant both events access to the API Key:

1. Open Secrets and edit the vault created in Step 2.
2. Find Event Access.
3. Select both Enable Scheduler and Disable Scheduler.
4. Save the vault.

When either event runs, xyOps decrypts the vault in memory and injects XYOPS_API_KEY into the Shell Plugin process as an environment variable.

Step 6: Test the Events

Test the setup before relying on it for a production blackout.

1. Temporarily use a short blackout window, such as 10 or 15 minutes.
2. Schedule the enable event one minute before the test blackout starts.
3. Set its Delay modifier so the API call will run at the end of the test window.
4. Schedule the disable event for the start of the test window.
5. Confirm that the enable job appears in a delayed state before the disable event runs.
6. Confirm that the scheduler becomes disabled at the expected time.
7. Confirm that the delayed job runs and re-enables the scheduler at the end.
8. Restore the intended production schedule and delay.

Running Jobs and Blackout Timing

The blackout only prevents new scheduled jobs from launching. It does not abort jobs that are already active when the scheduler is disabled.

If you have long-running jobs, schedule the blackout to start early enough for those jobs to finish before the period when you need the system to be quiet. For example, if your longest nightly job usually takes 45 minutes and everything should be finished by 4:00 AM, consider starting the blackout at 3:15 AM or earlier.

The delayed enable job is itself already active when the blackout begins, so it is allowed to remain in its start-delay state and run when its delay expires.

Catch-Up Behavior

Events with the Catch-Up trigger modifier will remember schedules missed while the global scheduler was disabled. After the delayed event re-enables the scheduler, those events will automatically catch up and launch all jobs they missed during the blackout window.

This can produce a burst of work immediately after the blackout ends. Review the Catch-Up settings on your scheduled events and make sure your servers can handle the backlog. If an event should simply skip blackout launches, leave Catch-Up disabled for that event.

The two scheduler toggle events do not need Catch-Up enabled.

Recovery and Safety

Because the enable event is your automatic recovery mechanism, configure it carefully:

• Make sure at least one selected target server will be online and enabled when the Delay expires. The job does not need a target server while it is in its start-delay state.
• Do not abort or delete the delayed enable job while the scheduler is disabled.
• Test changes to the schedule and Delay duration with a short window first.
• If the delayed job fails, an administrator can re-enable the scheduler from the xyOps UI.
• Keep both event schedules in the same timezone.

Note

A Delay is a fixed amount of elapsed time and is not adjusted for daylight-saving time. If the selected schedule timezone observes daylight saving and the blackout spans a clock change, a fixed delay may run one hour early or late. Use a timezone without daylight-saving changes, such as UTC, or adjust the Delay duration for those dates if the blackout must end at an exact local wall-clock time.

All calls to update_global_state are recorded in the xyOps Activity Log as state_update transactions, so you can audit when the scheduler was enabled or disabled and which API Key performed the change.