ImportError: No module named 'zproject.dev_settings'

[zingmars]

After rebooting the machine I had set up a test instance of docker-zulip on it now refuses to run. The only thing I did on this machine since it last worked was change some unrelated iptables settings and rebooted it.

zulip_1      | === Begin Initial Configuration Phase ===
zulip_1      | Preparing and linking the uploads folder ...
zulip_1      | Prepared and linked the uploads directory.
zulip_1      | Executing nginx configuration ...
zulip_1      | Nginx configuration succeeded.
zulip_1      | Exectuing certificates configuration...
zulip_1      | Certificates configuration succeeded.
zulip_1      | Setting database configuration ...
zulip_1      | Setting key "DATABASES", type "array" in file "/home/zulip/deployments/current/zproject/settings.py".
zulip_1      | Setting key "REMOTE_POSTGRES_HOST", type "string" in file "/etc/zulip/settings.py".
zulip_1      | Setting key "REMOTE_POSTGRES_SSLMODE", type "string" in file "/etc/zulip/settings.py".
zulip_1      | Database configuration succeeded.
zulip_1      | Setting Zulip secrets ...
zulip_1      | Secrets already generated/existing.
zulip_1      | Secret found for "rabbitmq_password".
zulip_1      | Secret found for "email_password".
zulip_1      | Secret found for "secret_key".
zulip_1      | Linking secrets from data dir to etc zulip  ...
zulip_1      | Linked existing secrets from data dir to etc zulip.
zulip_1      | Zulip secrets configuration succeeded.
zulip_1      | Activating authentication backends ...
zulip_1      | Setting key "AUTHENTICATION_BACKENDS", type "array" in file "/etc/zulip/settings.py".
zulip_1      | Adding authentication backend "EmailAuthBackend".
zulip_1      | Authentication backend activation succeeded.
zulip_1      | Executing Zulip configuration ...
zulip_1      | Setting key "EXTERNAL_HOST", type "string" in file "/etc/zulip/settings.py".
zulip_1      | Setting key "EMAIL_HOST", type "string" in file "/etc/zulip/settings.py".
zulip_1      | Setting key "ALLOWED_HOSTS", type "array" in file "/etc/zulip/settings.py".
zulip_1      | Setting key "NOREPLY_EMAIL_ADDRESS", type "string" in file "/etc/zulip/settings.py".
zulip_1      | Setting key "ZULIP_ADMINISTRATOR", type "string" in file "/etc/zulip/settings.py".
zulip_1      | Setting key "REDIS_HOST", type "string" in file "/etc/zulip/settings.py".
zulip_1      | Setting key "RABBITMQ_HOST", type "string" in file "/etc/zulip/settings.py".
zulip_1      | Setting key "PUSH_NOTIFICATION_BOUNCER_URL", type "string" in file "/home/zulip/deployments/current/zproject/settings.py".
zulip_1      | Setting key "EMAIL_HOST_USER", type "string" in file "/etc/zulip/settings.py".
zulip_1      | Setting key "DEFAULT_FROM_EMAIL", type "string" in file "/etc/zulip/settings.py".
zulip_1      | Setting key "MEMCACHED_LOCATION", type "string" in file "/etc/zulip/settings.py".
zulip_1      | Setting key "EMAIL_USE_SSL", type "bool" in file "/etc/zulip/settings.py".
zulip_1      | Setting key "ADMIN_DOMAIN", type "string" in file "/etc/zulip/settings.py".
zulip_1      | Setting key "EMAIL_USE_TLS", type "bool" in file "/etc/zulip/settings.py".
zulip_1      | Setting key "EMAIL_PORT", type "integer" in file "/etc/zulip/settings.py".
zulip_1      | Traceback (most recent call last):
zulip_1      |   File "/home/zulip/deployments/current/manage.py", line 19, in <module>
zulip_1      |     log_management_command(" ".join(sys.argv), settings.MANAGEMENT_LOG_PATH)
zulip_1      |   File "/home/zulip/deployments/2018-05-08-09-59-46/zulip-py3-venv/lib/python3.5/site-packages/django/conf/__init__.py", line 56, in __getattr__
zulip_1      |     self._setup(name)
zulip_1      |   File "/home/zulip/deployments/2018-05-08-09-59-46/zulip-py3-venv/lib/python3.5/site-packages/django/conf/__init__.py", line 41, in _setup
zulip_1      |     self._wrapped = Settings(settings_module)
zulip_1      |   File "/home/zulip/deployments/2018-05-08-09-59-46/zulip-py3-venv/lib/python3.5/site-packages/django/conf/__init__.py", line 110, in __init__
zulip_1      |     mod = importlib.import_module(self.SETTINGS_MODULE)
zulip_1      |   File "/usr/lib/python3.5/importlib/__init__.py", line 126, in import_module
zulip_1      |     return _bootstrap._gcd_import(name[level:], package, level)
zulip_1      |   File "<frozen importlib._bootstrap>", line 986, in _gcd_import
zulip_1      |   File "<frozen importlib._bootstrap>", line 969, in _find_and_load
zulip_1      |   File "<frozen importlib._bootstrap>", line 958, in _find_and_load_unlocked
zulip_1      |   File "<frozen importlib._bootstrap>", line 673, in _load_unlocked
zulip_1      |   File "<frozen importlib._bootstrap_external>", line 665, in exec_module
zulip_1      |   File "<frozen importlib._bootstrap>", line 222, in _call_with_frames_removed
zulip_1      |   File "/home/zulip/deployments/2018-05-08-09-59-46/zproject/settings.py", line 104, in <module>
zulip_1      |     from .dev_settings import *
zulip_1      | ImportError: No module named 'zproject.dev_settings'

Also, reading data/settings/etc-zulip/settings.py, it seems that it re-adds all the settings entry every time it tries to run the instance even if the keys are present. Which is fun, because I now have at least 20 duplicate entries there.

Here's my docker-compose.yml -

version: '2'
services:
  database:
    restart: always
    # https://github.com/zulip/docker-zulip/issues/123
    #    image: "zulip/zulip-postgresql" 
    image: "quay.io/galexrt/postgres-zulip-tsearchextras:latest"
    environment:
      POSTGRES_DB: <removed>
      POSTGRES_USER: <removed>
      POSTGRES_PASSWORD: <removed>
    volumes:
      - "/var/www/zulip/db:/var/lib/postgresql/data:rw"
  memcached:
    image: "quay.io/sameersbn/memcached:latest"
    restart: always
  rabbitmq:
    image: "rabbitmq:3.5.5"
    hostname: zulip-rabbit
    restart: always
    environment:
        RABBITMQ_DEFAULT_USER: "<removed>"
        RABBITMQ_DEFAULT_PASS: "<removed>"
  redis:
    restart: always
    image: "quay.io/sameersbn/redis:latest"
    volumes:
      - "/var/www/zulip/redis:/var/lib/redis:rw"
  zulip:
    restart: always
    image: "quay.io/galexrt/zulip:1.8.1-0"
    ports:
      - "127.0.0.1:8002:443"
    environment:
      DB_HOST: "database"
      DB_USER: "<removed>"
      DB_PASS: "<removed>"
      SSL_CERTIFICATE_GENERATION: "self-signed"
      SETTING_MEMCACHED_LOCATION: "memcached:11211"
      SETTING_RABBITMQ_HOST: "rabbitmq"
      SETTING_REDIS_HOST: "redis"
      SECRETS_rabbitmq_password: "<removed>"
      SECRETS_secret_key: "<removed>"
      SETTING_EXTERNAL_HOST: "<removed>"
      SETTING_PUSH_NOTIFICATION_BOUNCER_URL: "https://push.zulipchat.com"
      LINK_SETTINGS_TO_DATA: "true"
      SETTING_ZULIP_ADMINISTRATOR: "<removed>"
      SETTING_ADMIN_DOMAIN: "<removed>"
      SETTING_NOREPLY_EMAIL_ADDRESS: "<removed>"
      SETTING_DEFAULT_FROM_EMAIL: "<removed>"
      SETTING_EMAIL_HOST: "<removed>"
      SETTING_EMAIL_HOST_USER: "<removed>"
      SECRETS_email_password: "<removed>"
      SETTING_EMAIL_PORT: "587"
      SETTING_EMAIL_USE_SSL: "False"
      SETTING_EMAIL_USE_TLS: "True"
      SETTING_ALLOWED_HOSTS: "[ '*' ]"
      ZULIP_AUTH_BACKENDS: "EmailAuthBackend"
      ZULIP_USER_CREATION_ENABLED: "False"
    volumes:
      - "/var/www/zulip/data:/data:rw"

[zingmars]

Well, this is annoying. Tried re-pulling the image, etc - no change.

Moved the data folder from the host to let it re-create, and I would get variations of different errors when trying to recreate the image -

zulip_1      | Setting key "EMAIL_PORT", type "integer" in file "/etc/zulip/settings.py".
zulip_1      | + unset setting_key setting_var
zulip_1      | + su zulip -c '/home/zulip/deployments/current/manage.py checkconfig'
zulip_1      | Traceback (most recent call last):
zulip_1      |   File "/home/zulip/deployments/current/manage.py", line 19, in <module>
zulip_1      |     log_management_command(" ".join(sys.argv), settings.MANAGEMENT_LOG_PATH)
zulip_1      |   File "/home/zulip/deployments/2018-05-08-09-59-46/zulip-py3-venv/lib/python3.5/site-packages/django/conf/__init__.py", line 56, in __getattr__
zulip_1      |     self._setup(name)
zulip_1      |   File "/home/zulip/deployments/2018-05-08-09-59-46/zulip-py3-venv/lib/python3.5/site-packages/django/conf/__init__.py", line 41, in _setup
zulip_1      |     self._wrapped = Settings(settings_module)
zulip_1      |   File "/home/zulip/deployments/2018-05-08-09-59-46/zulip-py3-venv/lib/python3.5/site-packages/django/conf/__init__.py", line 129, in __init__
zulip_1      |     raise ImproperlyConfigured("The SECRET_KEY setting must not be empty.")
zulip_1      | django.core.exceptions.ImproperlyConfigured: The SECRET_KEY setting must not be empty.

zulip_1      | Preparing and linking the uploads folder ...
zulip_1      | + ln -sfT /data/uploads /home/zulip/uploads
zulip_1      | + chown zulip:zulip -R /data/uploads
zulip_1      | + '[' true = True ']'
zulip_1      | + '[' true = true ']'
zulip_1      | + '[' '!' -d /data/settings ']'
zulip_1      | + '[' '!' -d /data/settings/etc-zulip ']'
zulip_1      | + cp -rf /etc/zulip /data/settings/etc-zulip
zulip_1      | cp: failed to access '/data/settings/etc-zulip': Too many levels of symbolic links

Which one I get seems to be random. I have no idea what's going on with the first one zulip-secrets.conf was set to only be readable by root, but in the container it's read by the zulip account, but in the latter it seems that the etc-zulip folder inside the data folder is actually a symbolic link to /data/settings/etc-zulip for some reason.

I tried to disable LINK_SETTINGS_TO_DATA hoping if I let it handle the config file within the container it would work, but instead it gets stuck trying to create the database during executing first start init phase. which it obviously fails doing because the database already exists (and I would very much like to keep the data there). Obviously having a blank database works.

In desperation I downgraded to the 1.8.0-2 image which seems to work just fine and doesn't seem to choke on the settings.py file. This kind of sucks because having a test instance is kind of pointless if it's running an older version.

[timabbott]

@zingmars sorry for the trouble here!

So the dev_settings error in particular is caused by /etc/zulip/zulip.conf not existing (or perhaps not being visible to) for the zulip user; the others seem to be a similar issue with zulip-secrets.conf.

@galexrt do you understand why the permissions are not being set correctly for the /etc/zulip files for users upgrading?

[galexrt]

@timabbott I don't see a permissions error here. Are you talking about cp: failed to access '/data/settings/etc-zulip': Too many levels of symbolic links?
@zingmars Try to delete the settings/ directory on your host in the Zulip Docker volume dir and restart the image.
Could possibly be linked to https://github.com/zulip/docker-zulip/blob/master/entrypoint.sh#L59-L70.

[Atomique]

Did you guys solve the problem? I just get the same error. I just added the environment variables "LINK_SETTINGS_TO_DATA=True" and "MANUAL_CONFIGURATION=True" to change the settings for LDAP Auth - ImportError: No module named 'zproject.dev_settings'. If I set them to false, everything is working normal, I can start the server, login etc. I just want to change the settings.py to setup LDAP - I dont need more. I tried to set the permissions to UID 1000 without any luck.

Is there another way to configure the settings I am missing maybe?

If you need further information - let me know

[timabbott]

I don't think this particular failure mode of detecting as a development environment is still possible. This code in zproject/config.py in zulip/zulip should crash if zulip.conf is not present:

config_file = configparser.RawConfigParser()                                                            
config_file.read("/etc/zulip/zulip.conf")                                                               
                                                                                                        
# Whether this instance of Zulip is running in a production environment.                                
PRODUCTION = config_file.has_option("machine", "deploy_type")                                           
DEVELOPMENT = not PRODUCTION                                                                            

The actual bug was fixed here: #209 (comment)