Docker support [WIP] #32

stefankoegl · 2014-11-16T09:07:14Z

Support running gpodder.net in Docker.

Getting started

# Start all the containers
sudo docker-compose up

# Initialize database
sudo docker run -it --env-file=contrib/docker.env --network mygpo_default mygpo_beat python manage.py migrate

# Create an admin user
sudo docker run -it --env-file=contrib/docker.env --network mygpo_default -it mygpo_beat python manage.py createsuperuser

TODO

media files (images / css) is not yet served correctly
split media and static files Split static and media files #28
Startup ordering is not correct yet; beat and worker fail because the database takes too long to start
docs
probably more

JeanFred · 2016-08-09T20:14:22Z

Hello,

I tried to have a look at this to get it working using docker-compose.

I rebased this and here is where I am currently:
JeanFred@e8b3212

Still more to do, but I’m stuck on a Django migration thing (and my Django-fu is not that strong):

Steps to reproduce:

Check out my branch 'docker'
docker-compose up -d
docker-compose run --rm web python manage.py migrate

Traceback (most recent call last):
  File "/usr/local/lib/python3.5/site-packages/django/db/backends/utils.py", line 62, in execute
    return self.cursor.execute(sql)
psycopg2.ProgrammingError: relation "auth_user" does not exist
LINE 3: FROM auth_user
             ^


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "manage.py", line 26, in <module>
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/python3.5/site-packages/django/core/management/__init__.py", line 353, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python3.5/site-packages/django/core/management/__init__.py", line 345, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/lib/python3.5/site-packages/django/core/management/base.py", line 348, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/local/lib/python3.5/site-packages/django/core/management/base.py", line 398, in execute
    self.check()
  File "/usr/local/lib/python3.5/site-packages/django/core/management/base.py", line 426, in check
    include_deployment_checks=include_deployment_checks,
  File "/usr/local/lib/python3.5/site-packages/django/core/checks/registry.py", line 75, in run_checks
    new_errors = check(app_configs=app_configs)
  File "/srv/mygpo/mygpo/users/checks.py", line 17, in check_case_insensitive_users
    cursor.execute(SQL)
  File "/usr/local/lib/python3.5/site-packages/django/db/backends/utils.py", line 64, in execute
    return self.cursor.execute(sql, params)
  File "/usr/local/lib/python3.5/site-packages/django/db/utils.py", line 95, in __exit__
    six.reraise(dj_exc_type, dj_exc_value, traceback)
  File "/usr/local/lib/python3.5/site-packages/django/utils/six.py", line 685, in reraise
    raise value.with_traceback(tb)
  File "/usr/local/lib/python3.5/site-packages/django/db/backends/utils.py", line 62, in execute
    return self.cursor.execute(sql)
django.db.utils.ProgrammingError: relation "auth_user" does not exist
LINE 3: FROM auth_user

Looking around, folks often advise to run manage.py migrate auth, or manage.py makemigrations <snip> − I tried a lot of those but no chance.

Would that ring a bell to anyone by any chance ? :)

JeanFred · 2016-08-22T14:31:42Z

Thoughts @stefankoegl ? :)

morgenroth · 2016-09-04T07:57:15Z

Hello,

I tried to run mygpo in docker-compose and got the same error. It looks like there is a missing initialization of the DB, but "manage.py migrate" did not help.

mygpo_1  | The above exception was the direct cause of the following exception:
mygpo_1  | 
mygpo_1  | Traceback (most recent call last):
mygpo_1  |   File "manage.py", line 26, in <module>
mygpo_1  |     execute_from_command_line(sys.argv)
mygpo_1  |   File "/usr/local/lib/python3.5/dist-packages/django/core/management/__init__.py", line 353, in execute_from_command_line
mygpo_1  |     utility.execute()
mygpo_1  |   File "/usr/local/lib/python3.5/dist-packages/django/core/management/__init__.py", line 345, in execute
mygpo_1  |     self.fetch_command(subcommand).run_from_argv(self.argv)
mygpo_1  |   File "/usr/local/lib/python3.5/dist-packages/django/core/management/base.py", line 348, in run_from_argv
mygpo_1  |     self.execute(*args, **cmd_options)
mygpo_1  |   File "/usr/local/lib/python3.5/dist-packages/django/core/management/base.py", line 398, in execute
mygpo_1  |     self.check()
mygpo_1  |   File "/usr/local/lib/python3.5/dist-packages/django/core/management/base.py", line 426, in check
mygpo_1  |     include_deployment_checks=include_deployment_checks,
mygpo_1  |   File "/usr/local/lib/python3.5/dist-packages/django/core/checks/registry.py", line 75, in run_checks
mygpo_1  |     new_errors = check(app_configs=app_configs)
mygpo_1  |   File "/mygpo/mygpo/users/checks.py", line 17, in check_case_insensitive_users
mygpo_1  |     cursor.execute(SQL)
mygpo_1  |   File "/usr/local/lib/python3.5/dist-packages/django/db/backends/utils.py", line 64, in execute
mygpo_1  |     return self.cursor.execute(sql, params)
mygpo_1  |   File "/usr/local/lib/python3.5/dist-packages/django/db/utils.py", line 95, in __exit__
mygpo_1  |     six.reraise(dj_exc_type, dj_exc_value, traceback)
mygpo_1  |   File "/usr/local/lib/python3.5/dist-packages/django/utils/six.py", line 685, in reraise
mygpo_1  |     raise value.with_traceback(tb)
mygpo_1  |   File "/usr/local/lib/python3.5/dist-packages/django/db/backends/utils.py", line 62, in execute
mygpo_1  |     return self.cursor.execute(sql)
mygpo_1  | django.db.utils.ProgrammingError: relation "auth_user" does not exist
mygpo_1  | LINE 3: FROM auth_user
mygpo_1  |              ^
mygpo_1  |

How can I initialize the database properly?

fabolhak · 2016-09-04T23:46:45Z

With the Dockerfile from @morgenroth I get the same error. I think it is maybe because of python 3.

So I tried it with an older ubuntu version and python 2.7. Unfortunately without any luck :(

root@ecc3f2d6657b:/mygpo# python manage.py migrate
Traceback (most recent call last):
  File "manage.py", line 26, in <module>
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/python2.7/dist-packages/django/core/management/__init__.py", line 353, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python2.7/dist-packages/django/core/management/__init__.py", line 327, in execute
    django.setup()
  File "/usr/local/lib/python2.7/dist-packages/django/__init__.py", line 18, in setup
    apps.populate(settings.INSTALLED_APPS)
  File "/usr/local/lib/python2.7/dist-packages/django/apps/registry.py", line 85, in populate
    app_config = AppConfig.create(entry)
  File "/usr/local/lib/python2.7/dist-packages/django/apps/config.py", line 90, in create
    module = import_module(entry)
  File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module
    __import__(name)
  File "/mygpo/mygpo/api/__init__.py", line 27, in <module>
    from mygpo.utils import parse_request_body
  File "/mygpo/mygpo/utils.py", line 236
    print('\r', end=' ', file=stream)
                   ^
SyntaxError: invalid syntax

JeanFred · 2016-09-30T22:36:43Z

Digging through StackOverflow, that might rather be something due to a Django upgrade?

fabolhak · 2016-12-26T01:39:08Z

I tried various commands, including:
python3 manage.py migrate
python3 manage.py migrate auth
python3 manage.py migrate mygpo
python3 manage.py makemigrations auth
python3 manage.py makemigrations mypgo

Without any luck :( .
Always the same error from above.

elelay · 2017-04-08T15:01:05Z

This is due to changed exception thrown on by django when a table is missing from the DB.

In mygpo/users/checks.py, import ProgrammingError instead of OperationalError and catch it:

-    except OperationalError as oe:
-        if 'no such table: auth_user' in str(oe):
+    except ProgrammingError as oe:
+        if 'no such table: auth_user' in str(oe) or 'relation "auth_user" does not exist' in str(oe):

This worked for me

@elelay

As suggested by @elelay on gpodder#32 (comment)

JeanFred · 2017-07-01T15:51:14Z

Thanks @elelay ! That did the trick. I got a bit further on in JeanFred@d8d89a2

elelay · 2017-07-02T12:58:12Z

Great!

coveralls · 2017-07-02T13:38:52Z

Coverage decreased (-46.3%) to 0.14% when pulling f180435 on docker into 5cd52a5 on master.

coveralls · 2017-07-02T13:49:23Z

Coverage decreased (-46.3%) to 0.14% when pulling 3a719f0 on docker into 5cd52a5 on master.

coveralls · 2017-07-02T14:10:21Z

Coverage increased (+0.03%) to 46.424% when pulling 9ce0fa2 on docker into 5cd52a5 on master.

stefankoegl · 2017-07-23T06:24:33Z

Hi everyone! I appreciate your interest in that topic, and I'd like to continue working on that pull request. Is anyone of you still working on that? Where did you get stuck exactly?

elelay · 2017-07-23T08:37:30Z

My struggles were with compatibility with newer django version (ProgrammingError instead of ProgrammingError) and python 2 compatibility (broken in a few files). I've not tried to dockerize.
My second issues were with setting up periodic tasks to parse feeds, etc..

elelay · 2017-07-23T08:38:31Z

Great to hear you're working on it, btw. Thanks!

This is useful for development, when there is no nginx server to serve assets.

@elelay

As suggested by @elelay on #32 (comment)

This keeps crashing with ` ModuleNotFoundError: No module named 'django_nose'` even though the package is installed. Let's ignore for now.

JeanFred · 2017-07-23T10:13:15Z

@stefankoegl I don’t really work actively on it (I just thought it would be a fun app to dockerize :). Glad to hear you are on it :)

With my current setup I got as far as correctly running mygpo using docker-compose. Static assets are served, I could create user account from command line.
What does not work is:

e-mail for registration, but I don’t think that’s really needed
ElasticSearch setup − I could correctly add an ES container, but it would need a couple of fixtures (indices etc.) − since PostgreSQL fulltext search #38 may get rid of ES, I did not look into it further

All in all, you may be interested in these two commits:

JeanFred@89d689f is a complete docker-compose setup, with a base image based on python:3 rather than ubuntu
JeanFred@cdcb34f serves static assets using Whitenoise, which avoid having to throw an nginx webserver

Hope that helps! (Note that I’m not very familiar with mygpo so I definitely may have missed obvious things)

Python 3.7 is not yet supported

SiqingYu · 2020-02-24T05:52:44Z

I am currenly triaging this pull request. For more discussions about Docker containerization, please comment under #355 😄

There is no sudo available in the Docker environment. Let's just move the `apt-get` call command to a variable, and override it in Makefile call in the Dockerfile.

There was a breaking change recently in Postgres Docker image: ``` Error: Database is uninitialized and superuser password is not specified. You must specify POSTGRES_PASSWORD for the superuser. Use "-e POSTGRES_PASSWORD=password" to set it in "docker run". You may also use POSTGRES_HOST_AUTH_METHOD=trust to allow all connections without a password. This is *not* recommended. See PostgreSQL documentation about "trust": https://www.postgresql.org/docs/current/auth-trust.html ``` This may not be the best setting, but let's go for this. See docker-library/postgres#681

Without it, the step `compilemessages` fails.

JeanFred · 2020-04-14T12:57:28Z

I tried spinning up this and ran into a couple of issues, which I submitted as #397. Hope this helps!

This gets me as far as a clean Docker build, and an accessible UI. Static assets (CSS etc.) are however not served.

EDIT: Added one more commit, static assets are now correctly served.

This is one of the few requirements for whitenoise to just work. See http://whitenoise.evans.io/en/stable/django.html (This had been done in cdcb34f, but probably lost in a master merge)

Using (gpodder#32) and (gpodder#397) as a starting point, we implement some changes to get mygpo to run correctly under Docker. To actually make use of this, we need an as-yet-unpublished `Dockerfile` and `docker-compose.yaml`.

Some fixes for the Docker setup

oliverkane · 2020-11-17T00:25:59Z

Hi there. I figure I'd offer this wonderful script I use in my projects to wait for other containers to be there before you start

https://github.com/vishnubob/wait-for-it

Add this to the container that you want to hold up for a service.

Start Docker support [WIP]

f3f4c52

stefankoegl force-pushed the docker branch from 7cc9b96 to f3f4c52 Compare November 19, 2014 21:23

stefankoegl force-pushed the master branch from 95a9bf8 to fff2404 Compare June 13, 2015 08:21

JeanFred added a commit to JeanFred/mygpo that referenced this pull request Jul 1, 2017

Avoid crash during initial migration

d8d89a2

As suggested by @elelay on gpodder#32 (comment)

Merge branch 'master' into docker

f180435

Move dependency installation from Dockerfile to makefile

3a719f0

Simplify settings

9ce0fa2

Merge branch 'master' into docker

c3268bc

stefankoegl and others added 6 commits July 23, 2017 10:44

Start Docker support [WIP]

247b23e

Docker-compose setup

89d689f

Serve static assets using Whitenoise

cdcb34f

This is useful for development, when there is no nginx server to serve assets.

Avoid crash during initial migration

44e40e4

As suggested by @elelay on #32 (comment)

Update Dockerfile, makefile to fix build

6f74bbd

Comment out django-nose from INSTALLED_APPS

d3580b3

This keeps crashing with ` ModuleNotFoundError: No module named 'django_nose'` even though the package is installed. Let's ignore for now.

stefankoegl added 4 commits June 28, 2018 21:44

Merge branch 'master' into docker

86836ed

Specify Python 3.6 in Dockerfile

7611dbc

Python 3.7 is not yet supported

Merge branch 'master' into docker

85343ec

Merge branch 'master' into docker

2567edc

JeanFred mentioned this pull request Oct 8, 2019

Environment setup automation #262

Closed

stefankoegl added 3 commits October 15, 2019 21:53

Merge branch 'master' into docker

15c1204

Fix code formatting

a15262e

Merge branch 'master' into docker

44b0499

stefankoegl changed the title ~~Docker support~~ Docker support [WIP] Jan 30, 2020

SiqingYu self-requested a review February 23, 2020 22:58

SiqingYu self-assigned this Feb 23, 2020

SiqingYu added the configuration label Feb 24, 2020

SiqingYu mentioned this pull request Feb 24, 2020

Docker containerization #355

Open

4 tasks

SiqingYu linked an issue Feb 24, 2020 that may be closed by this pull request

Docker containerization #355

Open

4 tasks

SiqingYu and others added 7 commits February 24, 2020 15:46

Merge branch 'master' into docker

560f3eb

Fix makefile tab indentation.

31365aa

Upgrade whitenoise to fix django.utils.six ModuleNotFoundError

406f1d5

Allow to run Makefile command without sudo

8484c3e

There is no sudo available in the Docker environment. Let's just move the `apt-get` call command to a variable, and override it in Makefile call in the Dockerfile.

Add gettext to makefile dependencies

93131ae

Without it, the step `compilemessages` fails.

Fix variable in French locale

9f4ac38

JeanFred mentioned this pull request Apr 14, 2020

Some fixes for the Docker setup #397

Merged

Set STATIC_ROOT for serving static assets using Whitenoise

a7045b9

This is one of the few requirements for whitenoise to just work. See http://whitenoise.evans.io/en/stable/django.html (This had been done in cdcb34f, but probably lost in a master merge)

Merge pull request #397 from JeanFred/docker

06a1b80

Some fixes for the Docker setup

mattcen mentioned this pull request Feb 5, 2021

Update Docker WIP (master merge, static files fix) #559

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Docker support [WIP] #32

Docker support [WIP] #32

stefankoegl commented Nov 16, 2014 •

edited

Loading

JeanFred commented Aug 9, 2016

JeanFred commented Aug 22, 2016

morgenroth commented Sep 4, 2016

fabolhak commented Sep 4, 2016 •

edited

Loading

JeanFred commented Sep 30, 2016

fabolhak commented Dec 26, 2016

elelay commented Apr 8, 2017

JeanFred commented Jul 1, 2017

elelay commented Jul 2, 2017

coveralls commented Jul 2, 2017 •

edited

Loading

coveralls commented Jul 2, 2017 •

edited

Loading

coveralls commented Jul 2, 2017 •

edited

Loading

stefankoegl commented Jul 23, 2017

elelay commented Jul 23, 2017

elelay commented Jul 23, 2017

JeanFred commented Jul 23, 2017 •

edited

Loading

SiqingYu commented Feb 24, 2020

JeanFred commented Apr 14, 2020 •

edited

Loading

oliverkane commented Nov 17, 2020

Docker support [WIP] #32

Are you sure you want to change the base?

Docker support [WIP] #32

Conversation

stefankoegl commented Nov 16, 2014 • edited Loading

Getting started

TODO

JeanFred commented Aug 9, 2016

JeanFred commented Aug 22, 2016

morgenroth commented Sep 4, 2016

fabolhak commented Sep 4, 2016 • edited Loading

JeanFred commented Sep 30, 2016

fabolhak commented Dec 26, 2016

elelay commented Apr 8, 2017

JeanFred commented Jul 1, 2017

elelay commented Jul 2, 2017

coveralls commented Jul 2, 2017 • edited Loading

coveralls commented Jul 2, 2017 • edited Loading

coveralls commented Jul 2, 2017 • edited Loading

stefankoegl commented Jul 23, 2017

elelay commented Jul 23, 2017

elelay commented Jul 23, 2017

JeanFred commented Jul 23, 2017 • edited Loading

SiqingYu commented Feb 24, 2020

JeanFred commented Apr 14, 2020 • edited Loading

oliverkane commented Nov 17, 2020

stefankoegl commented Nov 16, 2014 •

edited

Loading

fabolhak commented Sep 4, 2016 •

edited

Loading

coveralls commented Jul 2, 2017 •

edited

Loading

coveralls commented Jul 2, 2017 •

edited

Loading

coveralls commented Jul 2, 2017 •

edited

Loading

JeanFred commented Jul 23, 2017 •

edited

Loading

JeanFred commented Apr 14, 2020 •

edited

Loading