django-crontab can't access environment variables used in settings.py

[calcutec]

Environment & Versions

• Operating system:
• Python: 3.6.4
• Django: 2.0.2
• django-crontab: 0.7.1

Settings

• My django-crontab settings:

CRONJOBS = [
    ('* * * * *', 'publicity.cron.my_scheduled_job')
]

Details

• Output of crontab -l after running python manage.py crontab add.

* * * * * /Users/billburton/Projects/djangodigital/dpmvenv/bin/python /Users/billburton/Projects/djangodigital/manage.py crontab run 79da9099e73f903adeec54d63e2acd67 # django-cronjobs for djangodigital

• If this output contained a crontab run command and I copy this command to my terminal the job works: yes/no

no

The traceback is below. The issue is that crontab doesn't recognize the os.environ variables i use in settings.py for db connections, aws secret keys, etc. I don't know how to make these available to a cron job and hoped that django-crontab would have a baked-in solution.

The same function works fine when I am running it from the django project itself.

Thanks!

Message 18:
From billburton@Jimmys-iMac.local Fri Aug 10 15:05:00 2018
X-Original-To: billburton
Delivered-To: billburton@Jimmys-iMac.local
From: billburton@Jimmys-iMac.local (Cron Daemon)
To: billburton@Jimmys-iMac.local
Subject: Cron billburton@Jimmys-iMac /Users/billburton/Projects/djangodigital/dpmvenv/bin/python /Users/billburton/Projects/djangodigital/manage.py crontab run 79da9099e73f903adeec54d63e2acd67 # django-cronjobs for djangodigital
X-Cron-Env: <SHELL=/bin/sh>
X-Cron-Env: <PATH=/usr/bin:/bin>
X-Cron-Env: <LOGNAME=billburton>
X-Cron-Env: <USER=billburton>
X-Cron-Env: <HOME=/Users/billburton>
Date: Fri, 10 Aug 2018 15:05:00 -0400 (EDT)

Traceback (most recent call last):
File "/Users/billburton/Projects/djangodigital/manage.py", line 22, in
execute_from_command_line(sys.argv)
File "/Users/billburton/Projects/djangodigital/dpmvenv/lib/python3.6/site-packages/django/core/management/init.py", line 371, in execute_from_command_line
utility.execute()
File "/Users/billburton/Projects/djangodigital/dpmvenv/lib/python3.6/site-packages/django/core/management/init.py", line 317, in execute
settings.INSTALLED_APPS
File "/Users/billburton/Projects/djangodigital/dpmvenv/lib/python3.6/site-packages/django/conf/init.py", line 56, in getattr
self._setup(name)
File "/Users/billburton/Projects/djangodigital/dpmvenv/lib/python3.6/site-packages/django/conf/init.py", line 43, in _setup
self._wrapped = Settings(settings_module)
File "/Users/billburton/Projects/djangodigital/dpmvenv/lib/python3.6/site-packages/django/conf/init.py", line 106, in init
mod = importlib.import_module(self.SETTINGS_MODULE)
File "/Users/billburton/Projects/djangodigital/dpmvenv/lib/python3.6/importlib/init.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "", line 994, in _gcd_import
File "", line 971, in _find_and_load
File "", line 955, in _find_and_load_unlocked
File "", line 665, in _load_unlocked
File "", line 678, in exec_module
File "", line 219, in _call_with_frames_removed
File "/Users/billburton/Projects/djangodigital/djangodigital/settings.py", line 94, in
'HOST': os.environ['RDS_DPM_MAIN_HOSTNAME'],
File "/Users/billburton/Projects/djangodigital/dpmvenv/bin/../lib/python3.6/os.py", line 669, in getitem
raise KeyError(key) from None
KeyError: 'RDS_DPM_MAIN_HOSTNAME'

[calcutec]

Figured out how to inform the crontab of environment variables. Pointed it to the .bash_profile where they are set.

CRONJOBS = [
('* * * * * source $HOME/.bash_profile;', 'publicity.cron.my_scheduled_job')
]

Works great

[kraiz]

Have a look at the README.There's an example for the setting CRONTAB_COMMAND_PREFIX. You should either define your needed values there or place them manually on the top of crontab via crontab -e.

[calcutec]

I glossed over that. Thanks.

Great project!

[txemac]

I solved it editing the crontab, add this at first line:

SHELL=/bin/bash

And change the line created by crontab library with the command:

* * * * * source /Users/billburton/Projects/djangodigital/dpmvenv/bin/activate && /Users/billburton/Projects/djangodigital/manage.py crontab run 79da9099e73f903adeec54d63e2acd67 # django-cronjobs for djangodigital

[johnyoonh]

Yeah, it seems like I have to add that line SHELL=/bin/bash manually in crontab. Setting it as a prefix doesn't make it work with or without export.
One solution to wrap the command after specifying the shell in the settings.py
https://unix.stackexchange.com/a/430478/16814

[luzhongyang]

If use in docker or k8s, SHELL=/bin/bash can not work, my solution is this:

1. export all env in docker cmd or entrypoint.sh

export > /opt/env

2. load env on django-crontab

CRONJOBS = [
    ('*/5 * * * * (source /opt/env || true) &&', 'app.crontab.crontab')
]

[Gradd]

None of the above worked for me, but it pointed me in the right direction.

carestad's answer in this link was what did the trick for me
https://stackoverflow.com/questions/2229825/where-can-i-set-environment-variables-that-crontab-will-use

basically, the crontab doesn't have the env variables in scope, so to do that, we can output our env into the crontab, that'll make them available for the scripts that need to be executed

in my entrypoint.sh:

if [[ "$RUN_CRON" == *"YES"* ]]; then
  service rsyslog start
  echo "$(env ; crontab -l)" | crontab -
  /etc/init.d/cron start
  python /code/manage.py crontab add
fi

then in the settings.py

CRONJOBS = [
    ('0 0 * * *', 'app.management.commands.test_command.test_command', '>> /code/cron_nightly.logs')
]

[yousufctec]

@kraiz is right, you can use the CRONTAB_COMMAND_PREFIX in your settings.py as below.

envs = []
for envkey in os.environ.keys():
    envs.append(envkey + '=' + os.environ[envkey])
CRONTAB_COMMAND_PREFIX = 'env $(echo ' + '\n'.join(envs) + ' | xargs)'

This is working, given the environment variables were not modified/added during runtime.

[Sajzad]

If you are using Ubuntu >=18.0, try with following changes.
I am assuming you have successfully integrated django-crontab with your django application. Now, access to your server via ssh and type "crontab -e", You will see your cronjobs are defined here. Add Your Environment Variables just above your cronjob and save the file using CTRL+o & CTRL+x. See the following piece of lines-

DB_NAME = ""
DB_USER = "*******"
DB_PASSWORD = ""
SECRET_KEY = "************************"

*/1 * * * * /home/sms/sms/env/bin/python /home/sms/sms/sms-backend/website/manage.py crontab run 446aeeebac8c8f82d9e01bb662dc9b21 >> /tmp/scheduled_job.log # django>

Nota Bene: For me, I have 4 variables, Add all your environment variables as it is mentioned and replace all asterisk alpahbet with your information.

[cjaca]

For docker/k8s case:

1. Export all env set previously in docker cmd in entrypoint.sh or any start script to .env file:

printenv > /app_folder/.env

2. Install additional plugin python-dotenv==0.20.0 and add it to requirements.txt

pip install python-dotenv

3. Import python-dotenv in settings.py and import values:

from pathlib import Path
import os
from dotenv import dotenv_values

BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
ENV_FILE=str(Path(BASE_DIR)) + '/.env'
myvars=dotenv_values(ENV_FILE)

4. Reuse .env values across all Django project:

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.postgresql',
        'NAME': myvars['POSTGRES_NAME'],
        'USER': myvars['POSTGRES_USER'],
        'PASSWORD': myvars['POSTGRES_PASSWORD'],
        'HOST': myvars['POSTGRES_HOST'],
        'PORT': 5432,
    }
}

5. No changes required in CRONJOBS

[ayudmin]

Had a similar issue with a project but of course i figured it out just about the environment variables you just need to add the CRONTAB_COMMAND_PREFIX before the list of your crontabs. OK basically that tells crontab where to find the environment it needs to work with ie:

CRONTAB_COMMAND_PREFIX = (
    'STAGE={whether production or development}'
)

CRONJOBS = [
    ('{time for cron}', '{your crontab view}'),

]

[Sajzad]

Solved. Thank you very much for your reply.

…

On Tue, 20 Sep 2022, 7:13 pm Ayume Francis, ***@***.***> wrote: Had a similar issue with a project but of course i figured it out just about the environment variables you just need to add the CRONTAB_COMMAND_PREFIX before the list of your crontabs. OK basically that tells crontab where to find the environment it needs to work with ie: `CRONTAB_COMMAND_PREFIX = ( 'STAGE={whether production or development}' ) CRONJOBS = [ ('* * * * *', '{your crontab view}'), ]` — Reply to this email directly, view it on GitHub <#88 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AKECZZLT7HWOBDVVQ5YEVMTV7GZ7XANCNFSM4FPCXTDA> . You are receiving this because you commented.Message ID: ***@***.***>

[cesarbonadio]

For docker/k8s case:

1. Export all env set previously in docker cmd in entrypoint.sh or any start script to .env file:

printenv > /app_folder/.env

2. Install additional plugin python-dotenv==0.20.0 and add it to requirements.txt

pip install python-dotenv

3. Import python-dotenv in settings.py and import values:

from pathlib import Path
import os
from dotenv import dotenv_values

BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
ENV_FILE=str(Path(BASE_DIR)) + '/.env'
myvars=dotenv_values(ENV_FILE)

4. Reuse .env values across all Django project:

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.postgresql',
        'NAME': myvars['POSTGRES_NAME'],
        'USER': myvars['POSTGRES_USER'],
        'PASSWORD': myvars['POSTGRES_PASSWORD'],
        'HOST': myvars['POSTGRES_HOST'],
        'PORT': 5432,
    }
}

5. No changes required in CRONJOBS

This a good approach and working solution for more complex variables. In my case, I don't use docker but I needed to pass the database configurations, which happens to work unexpectedly using the default approach of CRONTAB_COMMAND_PREFIX. This approach worked for me