Scheduled Jobs > Crontab corresponds to UTC

[mattmiller87]

Environment

• Nautobot version (Docker tag too if applicable): 1.5.10
• Python version: 3.10.10
• Database platform, version: Postgres: 13-alpine
• Middleware(s): None

Steps to Reproduce

1. Change TIME_ZONE to "America/New_York"
2. Schedule a job using "Recurring custom"
3. Use the Crontab to run a job at a specific time each day ( 0 9 * * *) for 9am

Expected Behavior

The job should be scheduled to run at 9am EST

Observed Behavior

The job runs at 9am UTC

[mattmiller87]

Not sure if I should tack onto this issue, but this is problematic for DST as well. The job continues to run at UTC time and is therefore 1 hour later now that we have sprung forward 1 hour.

[gsnider2195]

Looks like this could be as simple as

from django.utils import timezone
return schedules.crontab(
    minute=minute,
    hour=hour,
    day_of_month=day_of_month,
    month_of_year=month_of_year,
    day_of_week=day_of_week,
    nowfun=timezone.now,
)

[gsnider2195]

@mattmiller87 how are you setting TIME_ZONE in your environment? Are you using the same method for your Nautobot container and your celery beat container?

[mattmiller87]

I am updating the nautobot_config.py using the TIME_ZONE environment variable. such as:
TIME_ZONE='America/New_York'
Then restarting all docker containers.

[lampwins]

Let's document all the places where the timezone settings come into play and a note that we recommend UTC for consistency.

[itdependsnetworks]

Let's document all the places where the timezone settings come into play and a note that we recommend UTC for consistency.

Just off the top of my head

• system timezone on:
  • Nautobot
  • Nautobot Worker
  • Redis
  • Database
  • Celery Beat
• Application Level
  • nautobot-server: TIME_ZONE variable
    • Ensure this is the same in nautobot, celery, and celery beat workers.
  • psql "SET TIME ZONE 'UTC';"
  • mysql my.cnf "default-time-zone='+00:00'"

[mattmiller87]

I have reproduced this problem and have that environment available for review.

Two tests created:

1. Leave time as UTC, create the scheduled job, change the time_zone, restart all containers; job runs at same UTC time.
2. Change the time_zone, create the scheduled job, restart all containers; job runs at UTC time.

[lampwins]

This sounds like it will be deep inside of beat. If there is no easy solution here, I am inclined to document this as an edge case and say that if you change the system timezone, you need to update your scheduled jobs, because changing the system timezone is just such a rare thing to do in the first place.

[mattmiller87]

The issue arose when DST occurred and the job ran at the same UTC time; not adhering to the scheduled time.

[gsnider2195]

Spoke with @mattmiller87 today. It looks like an old cookie was missing the volume mapping nautobot_config.py for the celery beat service so the test deployment that he was using was experiencing the issue where beat's clock was not using the same time zone as nautobot. Will use this issue to update the docs to state that you must mount your nautobot_config.py into your celery beat and celery worker and ensure the time on all of those servers is correct.

I'm going to do one more test to simulate a distributed deployment to see how beat handles that:

• Nautobot server is in eastern TZ
• Celery beat server is in central TZ
• Both servers have the same nautobot_config.py with TIME_ZONE=America/New_York

[gsnider2195]

Confirmed that cron jobs will run in whatever time zone is configured in the TIME_ZONE setting, regardless of the server operating system's configured timezone. So if you want your crontabs to run according to the Eastern time zone you should set your TIME_ZONE to America/New_York on your celery beat server. If your server is in California and its timezone is set to Los Angeles, it will still run the 0 9 * * * scheduled job at 9am eastern.

[itdependsnetworks]

Thanks!! That helps describe the some of the challenges i have seen on Slack over the years.