fix bug 1123908 - use env vars for django config

[rhelmer]

@peterbe @AdrianGaudebert I just mechanically merged local.py into base.py and converted all settings to use python-decouple (which supports reading configuration from the environment.)

There are a few TODOs in here, and we should discuss whether we want to further merge user-configurable settings with the base.py defaults, but I wanted to do something that was easier to understand in diff form to start the discussion.

• resolve TODOs
• allow local.py overrides to work, while we're still in PHX

[peterbe]

Why not all of these?

The point of having SOUTH_TESTS_MIGRATE (for example) here in settings is so you can override them if you need to.
Well, it's kinda nice to have them written all down in one place too I guess.

[peterbe]

after irc conversation, I'm taking this point back.

[peterbe]

One thing that's missing the .env thing. For local development you don't want to override every single thing with environment variables. Developers have reliable disk and they can use a .env file to (example) to organize theirs.

At the moment I honestly don't know the difference between .env and settings.ini. I guess it's simply a choice of preference.

Either way, this PR is great for production whereby you will never rely on a disk file for settings but if we land this we'll leave developers stranded.

Also, I'm uncertain still why you left out some settings. Human oversight or deliberate?

[rhelmer]

One thing that's missing the .env thing. For local development you don't want to override every single thing with environment variables. Developers have reliable disk and they can use a .env file to (example) to organize theirs.

Hm, so the problem is that python-decouple will look for .env first, then fall back the looking in os.environ, which is what we don't want for production. Could we have a env-dist that people need to copy perhaps?

Also, I'm uncertain still why you left out some settings. Human oversight or deliberate?

I'm not sure how to override some (they are marked with TODOs), some I am not sure of the utility of overriding (ROOT_URLCONF? INSTALLED_APPS?)

Having to edit the base.py is totally doable, we just need to push a new release so I'd like to avoid it for things unlikely to change.

@peterbe do you think we should allow every single thing to be overridden? Not sure how to even approach something like VCS_MAPPINGS

[peterbe]

If you use SECRET_KEY = config('SECRET_KEY') I think it'll raise a UndefinedValueError if it's not provided with a value.

Here's how sugardough handles it

[peterbe]

And .env-dist or something would be good. By the way, here's how sugardough generates a .env when you create a project.

I don't understand why .env has higher priority than environment variables. That just feels strange but maybe the people who've thought it for longer know a good reason.
One potential benefit with .env is that if you generate one, the right settings will be there automatically for mod_wsgi, ./manage.py syncdb, ./manage.py shell, ./manage.py some-thing, etc.

And we will need to start django in other places than just uwsgi. E.g. syncdb, collectstatic and compress.
Could we, similarly, have a script that generates the .env in terraform or something (...that I don't understand). We'd expect this file to live as long as the EC2 instance lives.

We basically want production to be 100% environment variables and local developers to be 100% .env. We can just have a mailing list message or something to train existing devs to convert their aging settings/local.py to a .env but either way we'll need a sustainable message to help new developers.

[rhelmer]

Could we, similarly, have a script that generates the .env in terraform or something (...that I don't understand). We'd expect this file to live as long as the EC2 instance lives.

We want to use environment variables in prod and no config files, it'll be running like this:

envconsul -prefix socorro/webapp uwsgi webapp-django.wsgi.socorro-crashstats

So whenever the key DEBUG changes in the prefix socorro/webapp in consul, envconsul will automatically restart uwsgi.

[rhelmer]

@peterbe you mentioned in IRC possibly having a .env for production to allow for syncdb, collectstatic, compress, and so on.

I wonder if it would be possible instead to inject configuration into the shell.. for instance I could see doing sudo su - socorro and ending up with a shell that had an activated socorro virtualenv, and also all configuration values already set in the environment. So you could simply cd to the right dir and ./manage.py would do the right thing.

We really want to avoid pushing any kind of state to the nodes other than the apps themselves.

[rhelmer]

I wonder if it would be possible instead to inject configuration into the shell.. for instance I could see doing sudo su - socorro and ending up with a shell that had an activated socorro virtualenv, and also all configuration values already set in the environment. So you could simply cd to the right dir and ./manage.py would do the right thing.

To answer my own question - I don't think you can start an interactive shell with envconsul, but we could run it like this:

envconsul -prefix socorro/shell -once \
  /data/socorro/socorro-virtualenv/bin/python \
  /data/socorro/webapp-django/manage.py syncdb --noinput

We could easily have an alias or script that acts as an entry point for manage.py on the nodes.

[rhelmer]

@peterbe actually I realized that we need to not break PHX (we need to keep using local.py there), so we need to have a compromise here in the short-term... I think we should allow local.py overrides to work for now, and make it a warning not fatal if this does not exist.

This gives us some time to move the docs and dev config over, without blocking us using environment variables for production in AWS.

What do you think?

[peterbe]

Can you move this to just after from .local import *? I.e. it's deprecated to use local.py.
Also, instead of print can you change to:

import warnings
warnings.warn(
    "Use environment variables instead or a .env file", 
    DeprecationWarning
)

[rhelmer]

@peterbe sorry I don't understand:

Can you move this to just after from .local import *

Isn't it already just after it?

Also - I changed from print to warnings.warn but I don't see it printed when I do e.g. ./manage.py runserver, is that because logging is disabled by default?

[peterbe]

I mean; if it succeeds to import local.py you're potentially stuck in the old way of doing things. And you need to re-organize yourself away from local.py to a new .env file instead.

[rhelmer]

I mean; if it succeeds to import local.py you're potentially stuck in the old way of doing things. And you need to re-organize yourself away from local.py to a new .env file instead.

Oh! I see, so warn if there is a local.py, not if there is not. Sure, makes sense.

[rhelmer]

@peterbe ah the deprecation warning problem is https://code.djangoproject.com/ticket/18985 and they include an example in their docs (which is the default in manage.py from django 1.5 onward) to re-enable deprecation warnings.

[rhelmer]

@peterbe ok this is ready for review again!

[peterbe]

Isn't this supposed to default to 127.0.0.1:11211?

[rhelmer]

Isn't this supposed to default to 127.0.0.1:11211?

Hm well the default is LocMemCache right?

[peterbe]

r+

The only nit is that I don't see any developer-hand-holding with regards to setting up that .env file.

[rhelmer]

The only nit is that I don't see any developer-hand-holding with regards to setting up that .env file.

@peterbe what hand-holding did you have in mind? Something in the developer section of the docs, or were you thinking of something else?

[peterbe]

Yes, documentation is what I meant. Soft stuff. You're so ahead of me on terms of the beginner documentation so it's rubberstamp=r+ on it if you do anything.