Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

django.db.utils.OperationalError: (1045, "Access denied for user 'xqueue'@'172.18.0.13' #34

Closed
natea opened this issue Jun 8, 2018 · 25 comments
Closed

django.db.utils.OperationalError: (1045, "Access denied for user 'xqueue'@'172.18.0.13' #34

natea opened this issue Jun 8, 2018 · 25 comments
Labels
question General questions are welcome

Comments

@natea
Copy link

natea commented Jun 8, 2018

Hi Regis,

I did a git pull, configure and make build and now when I run make up i'm seeing this permissions error with the xqueue service.

And when I go to www.myopenedx.com and studio.myopenedx.com, I'm seeing "400 Bad Request".

xqueue_consumer_1  | Traceback (most recent call last):
xqueue_consumer_1  |   File "./manage.py", line 10, in <module>
xqueue_consumer_1  |     execute_from_command_line(sys.argv)
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/django/core/management/__init__.py", line 354, in execute_from_command_line
xqueue_consumer_1  |     utility.execute()
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/django/core/management/__init__.py", line 346, in execute
xqueue_consumer_1  |     self.fetch_command(subcommand).run_from_argv(self.argv)
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/django/core/management/base.py", line 394, in run_from_argv
xqueue_consumer_1  |     self.execute(*args, **cmd_options)
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/django/core/management/base.py", line 444, in execute
xqueue_consumer_1  |     self.check()
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/django/core/management/base.py", line 482, in check
xqueue_consumer_1  |     include_deployment_checks=include_deployment_checks,
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/django/core/checks/registry.py", line 72, in run_checks
xqueue_consumer_1  |     new_errors = check(app_configs=app_configs)
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/django/core/checks/model_checks.py", line 28, in check_all_models
xqueue_consumer_1  |     errors.extend(model.check(**kwargs))
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/django/db/models/base.py", line 1205, in check
xqueue_consumer_1  |     errors.extend(cls._check_fields(**kwargs))
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/django/db/models/base.py", line 1282, in _check_fields
xqueue_consumer_1  |     errors.extend(field.check(**kwargs))
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/django/db/models/fields/__init__.py", line 934, in check
xqueue_consumer_1  |     errors = super(AutoField, self).check(**kwargs)
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/django/db/models/fields/__init__.py", line 207, in check
xqueue_consumer_1  |     errors.extend(self._check_backend_specific_checks(**kwargs))
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/django/db/models/fields/__init__.py", line 306, in _check_backend_specific_checks
xqueue_consumer_1  |     return connection.validation.check_field(self, **kwargs)
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/django/db/backends/mysql/validation.py", line 18, in check_field
xqueue_consumer_1  |     field_type = field.db_type(connection)
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/django/db/models/fields/__init__.py", line 614, in db_type
xqueue_consumer_1  |     return connection.data_types[self.get_internal_type()] % data
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/django/db/__init__.py", line 36, in __getattr__
xqueue_consumer_1  |     return getattr(connections[DEFAULT_DB_ALIAS], item)
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/django/utils/functional.py", line 59, in __get__
xqueue_consumer_1  |     res = instance.__dict__[self.name] = self.func(instance)
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/django/db/backends/mysql/base.py", line 196, in data_types
xqueue_consumer_1  |     if self.features.supports_microsecond_precision:
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/django/utils/functional.py", line 59, in __get__
xqueue_consumer_1  |     res = instance.__dict__[self.name] = self.func(instance)
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/django/db/backends/mysql/features.py", line 52, in supports_microsecond_precision
xqueue_consumer_1  |     return self.connection.mysql_version >= (5, 6, 4) and Database.version_info >= (1, 2, 5)
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/django/utils/functional.py", line 59, in __get__
xqueue_consumer_1  |     res = instance.__dict__[self.name] = self.func(instance)
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/django/db/backends/mysql/base.py", line 371, in mysql_version
xqueue_consumer_1  |     with self.temporary_connection():
xqueue_consumer_1  |   File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
xqueue_consumer_1  |     return self.gen.next()
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/django/db/backends/base/base.py", line 462, in temporary_connection
xqueue_consumer_1  |     cursor = self.cursor()
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/django/db/backends/base/base.py", line 164, in cursor
xqueue_consumer_1  |     cursor = self.make_cursor(self._cursor())
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/django/db/backends/base/base.py", line 135, in _cursor
xqueue_consumer_1  |     self.ensure_connection()
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/django/db/backends/base/base.py", line 130, in ensure_connection
xqueue_consumer_1  |     self.connect()
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/django/db/utils.py", line 98, in __exit__
xqueue_consumer_1  |     six.reraise(dj_exc_type, dj_exc_value, traceback)
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/django/db/backends/base/base.py", line 130, in ensure_connection
xqueue_consumer_1  |     self.connect()
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/django/db/backends/base/base.py", line 119, in connect
xqueue_consumer_1  |     self.connection = self.get_new_connection(conn_params)
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/django/db/backends/mysql/base.py", line 276, in get_new_connection
xqueue_consumer_1  |     conn = Database.connect(**conn_params)
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/MySQLdb/__init__.py", line 81, in Connect
xqueue_consumer_1  |     return Connection(*args, **kwargs)
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/MySQLdb/connections.py", line 193, in __init__
xqueue_consumer_1  |     super(Connection, self).__init__(*args, **kwargs2)
xqueue_consumer_1  | django.db.utils.OperationalError: (1045, "Access denied for user 'xqueue'@'172.18.0.13' (using password: YES)")
xqueue_consumer_1  | Traceback (most recent call last):
xqueue_consumer_1  |   File "./manage.py", line 10, in <module>
xqueue_consumer_1  |     execute_from_command_line(sys.argv)
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/django/core/management/__init__.py", line 354, in execute_from_command_line
xqueue_consumer_1  |     utility.execute()
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/django/core/management/__init__.py", line 346, in execute
xqueue_consumer_1  |     self.fetch_command(subcommand).run_from_argv(self.argv)
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/django/core/management/base.py", line 394, in run_from_argv
xqueue_consumer_1  |     self.execute(*args, **cmd_options)
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/django/core/management/base.py", line 444, in execute
xqueue_consumer_1  |     self.check()
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/django/core/management/base.py", line 482, in check
xqueue_consumer_1  |     include_deployment_checks=include_deployment_checks,
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/django/core/checks/registry.py", line 72, in run_checks
xqueue_consumer_1  |     new_errors = check(app_configs=app_configs)
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/django/core/checks/model_checks.py", line 28, in check_all_models
xqueue_consumer_1  |     errors.extend(model.check(**kwargs))
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/django/db/models/base.py", line 1205, in check
xqueue_consumer_1  |     errors.extend(cls._check_fields(**kwargs))
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/django/db/models/base.py", line 1282, in _check_fields
xqueue_consumer_1  |     errors.extend(field.check(**kwargs))
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/django/db/models/fields/__init__.py", line 934, in check
xqueue_consumer_1  |     errors = super(AutoField, self).check(**kwargs)
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/django/db/models/fields/__init__.py", line 207, in check
xqueue_consumer_1  |     errors.extend(self._check_backend_specific_checks(**kwargs))
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/django/db/models/fields/__init__.py", line 306, in _check_backend_specific_checks
xqueue_consumer_1  |     return connection.validation.check_field(self, **kwargs)
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/django/db/backends/mysql/validation.py", line 18, in check_field
xqueue_consumer_1  |     field_type = field.db_type(connection)
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/django/db/models/fields/__init__.py", line 614, in db_type
xqueue_consumer_1  |     return connection.data_types[self.get_internal_type()] % data
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/django/db/__init__.py", line 36, in __getattr__
xqueue_consumer_1  |     return getattr(connections[DEFAULT_DB_ALIAS], item)
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/django/utils/functional.py", line 59, in __get__
xqueue_consumer_1  |     res = instance.__dict__[self.name] = self.func(instance)
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/django/db/backends/mysql/base.py", line 196, in data_types
xqueue_consumer_1  |     if self.features.supports_microsecond_precision:
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/django/utils/functional.py", line 59, in __get__
xqueue_consumer_1  |     res = instance.__dict__[self.name] = self.func(instance)
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/django/db/backends/mysql/features.py", line 52, in supports_microsecond_precision
xqueue_consumer_1  |     return self.connection.mysql_version >= (5, 6, 4) and Database.version_info >= (1, 2, 5)
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/django/utils/functional.py", line 59, in __get__
xqueue_consumer_1  |     res = instance.__dict__[self.name] = self.func(instance)
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/django/db/backends/mysql/base.py", line 371, in mysql_version
xqueue_consumer_1  |     with self.temporary_connection():
xqueue_consumer_1  |   File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
xqueue_consumer_1  |     return self.gen.next()
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/django/db/backends/base/base.py", line 462, in temporary_connection
xqueue_consumer_1  |     cursor = self.cursor()
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/django/db/backends/base/base.py", line 164, in cursor
xqueue_consumer_1  |     cursor = self.make_cursor(self._cursor())
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/django/db/backends/base/base.py", line 135, in _cursor
xqueue_consumer_1  |     self.ensure_connection()
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/django/db/backends/base/base.py", line 130, in ensure_connection
xqueue_consumer_1  |     self.connect()
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/django/db/utils.py", line 98, in __exit__
xqueue_consumer_1  |     six.reraise(dj_exc_type, dj_exc_value, traceback)
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/django/db/backends/base/base.py", line 130, in ensure_connection
xqueue_consumer_1  |     self.connect()
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/django/db/backends/base/base.py", line 119, in connect
xqueue_consumer_1  |     self.connection = self.get_new_connection(conn_params)
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/django/db/backends/mysql/base.py", line 276, in get_new_connection
xqueue_consumer_1  |     conn = Database.connect(**conn_params)
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/MySQLdb/__init__.py", line 81, in Connect
xqueue_consumer_1  |     return Connection(*args, **kwargs)
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/MySQLdb/connections.py", line 193, in __init__
xqueue_consumer_1  |     super(Connection, self).__init__(*args, **kwargs2)
xqueue_consumer_1  | django.db.utils.OperationalError: (1045, "Access denied for user 'xqueue'@'172.18.0.13' (using password: YES)")
xqueue_consumer_1  | Traceback (most recent call last):
xqueue_consumer_1  |   File "./manage.py", line 10, in <module>
xqueue_consumer_1  |     execute_from_command_line(sys.argv)
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/django/core/management/__init__.py", line 354, in execute_from_command_line
xqueue_consumer_1  |     utility.execute()
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/django/core/management/__init__.py", line 346, in execute
xqueue_consumer_1  |     self.fetch_command(subcommand).run_from_argv(self.argv)
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/django/core/management/base.py", line 394, in run_from_argv
xqueue_consumer_1  |     self.execute(*args, **cmd_options)
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/django/core/management/base.py", line 444, in execute
xqueue_consumer_1  |     self.check()
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/django/core/management/base.py", line 482, in check
xqueue_consumer_1  |     include_deployment_checks=include_deployment_checks,
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/django/core/checks/registry.py", line 72, in run_checks
xqueue_consumer_1  |     new_errors = check(app_configs=app_configs)
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/django/core/checks/model_checks.py", line 28, in check_all_models
xqueue_consumer_1  |     errors.extend(model.check(**kwargs))
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/django/db/models/base.py", line 1205, in check
xqueue_consumer_1  |     errors.extend(cls._check_fields(**kwargs))
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/django/db/models/base.py", line 1282, in _check_fields
xqueue_consumer_1  |     errors.extend(field.check(**kwargs))
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/django/db/models/fields/__init__.py", line 934, in check
xqueue_consumer_1  |     errors = super(AutoField, self).check(**kwargs)
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/django/db/models/fields/__init__.py", line 207, in check
xqueue_consumer_1  |     errors.extend(self._check_backend_specific_checks(**kwargs))
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/django/db/models/fields/__init__.py", line 306, in _check_backend_specific_checks
xqueue_consumer_1  |     return connection.validation.check_field(self, **kwargs)
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/django/db/backends/mysql/validation.py", line 18, in check_field
xqueue_consumer_1  |     field_type = field.db_type(connection)
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/django/db/models/fields/__init__.py", line 614, in db_type
xqueue_consumer_1  |     return connection.data_types[self.get_internal_type()] % data
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/django/db/__init__.py", line 36, in __getattr__
xqueue_consumer_1  |     return getattr(connections[DEFAULT_DB_ALIAS], item)
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/django/utils/functional.py", line 59, in __get__
xqueue_consumer_1  |     res = instance.__dict__[self.name] = self.func(instance)
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/django/db/backends/mysql/base.py", line 196, in data_types
xqueue_consumer_1  |     if self.features.supports_microsecond_precision:
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/django/utils/functional.py", line 59, in __get__
xqueue_consumer_1  |     res = instance.__dict__[self.name] = self.func(instance)
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/django/db/backends/mysql/features.py", line 52, in supports_microsecond_precision
xqueue_consumer_1  |     return self.connection.mysql_version >= (5, 6, 4) and Database.version_info >= (1, 2, 5)
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/django/utils/functional.py", line 59, in __get__
xqueue_consumer_1  |     res = instance.__dict__[self.name] = self.func(instance)
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/django/db/backends/mysql/base.py", line 371, in mysql_version
xqueue_consumer_1  |     with self.temporary_connection():
xqueue_consumer_1  |   File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
xqueue_consumer_1  |     return self.gen.next()
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/django/db/backends/base/base.py", line 462, in temporary_connection
xqueue_consumer_1  |     cursor = self.cursor()
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/django/db/backends/base/base.py", line 164, in cursor
xqueue_consumer_1  |     cursor = self.make_cursor(self._cursor())
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/django/db/backends/base/base.py", line 135, in _cursor
xqueue_consumer_1  |     self.ensure_connection()
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/django/db/backends/base/base.py", line 130, in ensure_connection
xqueue_consumer_1  |     self.connect()
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/django/db/utils.py", line 98, in __exit__
xqueue_consumer_1  |     six.reraise(dj_exc_type, dj_exc_value, traceback)
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/django/db/backends/base/base.py", line 130, in ensure_connection
xqueue_consumer_1  |     self.connect()
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/django/db/backends/base/base.py", line 119, in connect
xqueue_consumer_1  |     self.connection = self.get_new_connection(conn_params)
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/django/db/backends/mysql/base.py", line 276, in get_new_connection
xqueue_consumer_1  |     conn = Database.connect(**conn_params)
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/MySQLdb/__init__.py", line 81, in Connect
xqueue_consumer_1  |     return Connection(*args, **kwargs)
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/MySQLdb/connections.py", line 193, in __init__
xqueue_consumer_1  |     super(Connection, self).__init__(*args, **kwargs2)
xqueue_consumer_1  | django.db.utils.OperationalError: (1045, "Access denied for user 'xqueue'@'172.18.0.13' (using password: YES)")
@regisb
Copy link
Contributor

regisb commented Jun 8, 2018

Yes, the database associated with the xqueue service needs to be created. Please run make provision, make migrate and try again.

@regisb
Copy link
Contributor

regisb commented Jun 8, 2018

Concerning the 400 error, what is the exact domain name you used to configure the LMS?

@natea
Copy link
Author

natea commented Jun 8, 2018

i ran the two commands you suggested, but I'm still getting the same error.

i just realized that when i re-ran the ./configure command i just accepted the defaults which were localhost:8000 instead of www.myopenedx.com. and localhost:8001 for Studio instead of studio.myopenedx.com.

But when I go to localhost:8000 and localhost:8001, it doesn't return the site.

@regisb
Copy link
Contributor

regisb commented Jun 8, 2018

Did you stop all services prior to running both commands? (sorry, I should have mentioned that earlier)

To test the images locally, specify "localhost" and "studio.localhost" during configuration. I realised this is a common mistake, so I modified the message displayed on configuration.

@natea
Copy link
Author

natea commented Jun 8, 2018

No, I don't believe I had stopped all the services prior to running those commands. Will do so now.

If the default for testing locally is localhost and studio.localhost wouldn't you want to make those the defaults, rather than localhost:8000 and localhost:8001, or are the defaults for running it in production?

@natea
Copy link
Author

natea commented Jun 8, 2018

The docs are still referring to localhost for local development: https://github.com/regisb/openedx-docker#for-developers

@regisb
Copy link
Contributor

regisb commented Jun 8, 2018

During configuration, the defaults are "www.myopenedx.com" and "studio.myopenedx.com". I wanted to set defaults for a production environment, which explains why I didn't set "localhost" or "localhost:8000" as defaults.

Let's see what's going on here (and then I'll add more information to the README because you're not the first one with this problem): when the platform is running and we try to access it on port 80, we reach the Nginx server, which then forwards the request to the gunicorn process. The production settings of the gunicorn process are defined to allow access only to the configured domain name, through the ALLOWED_HOSTS django setting: https://github.com/regisb/openedx-docker/blob/master/config/openedx/universal/lms/production.py#L21
This is a security setting. But it is not the case in development: development settings allow access from any domain name (ALLOWED_HOSTS = ['*'] is set in aws.py): https://github.com/regisb/openedx-docker/blob/master/config/openedx/universal/lms/development.py

This is an issue that confuses many people, and I'm not 100% sure how we can improve the situation. A good start would be to allow access to "localhost:8000" and "localhost:8001", even in production. It's not a big deal, in terms of security, because ports 8000 and 8001 are not open from the docker-compose configuration. But I wonder if this is going to generate other issues, for instance with static assets.

@natea
Copy link
Author

natea commented Jun 8, 2018

I just re-ran the commands setting these environment variables. Before I was using the default settings file, which is for production, not for development.

export EDX_PLATFORM_PATH=/Users/nateaune/Dropbox/code/natea-edx-platform-ginkgo
export EDX_PLATFORM_SETTINGS=development

Now the LMS is responding at localhost and Studio is responding at studio.localhost but I still see the django.db.utils.OperationalError: (1045, "Access denied for user 'xqueue'@'172.18.0.13' (using password: YES)") error message in the logs.

regisb added a commit that referenced this issue Jun 8, 2018
@regisb
Allow direct access to gunicorn process in production
5cb51e0 
This is not a security issue because ports 8000-8001 are not open to the
world; it should also drastically simplify the life of many people. See
for instance issues #30 and #34.

Also, we allow access to nginx on hostnames "localhost" and
"studio.localhost" for lms and cms, respectively. Again, this will
remove much of the confusion for many users.
@regisb
Copy link
Contributor

regisb commented Jun 8, 2018

Nate, I just made changes to the production settings (in nginx and django) which should allow anyone to access "localhost" and "studio.localhost" both in development and producton mode. I also fixed the documentation.

The xqueue error message that you are seeing means that the xqueue MySQL database was not created, or not with the right credentials. I suspect that multiple mysql containers have been run concurrently. And here we reach the bottom of my ignorance about docker 😅 For some reason, when running make stop, most containers (including mysql) remain up. Can you please try to docker stop ... and docker rm ... each one of them manually? No container should be left running. Then run make provision again, and try to run the platform.

@regisb
Copy link
Contributor

regisb commented Jun 8, 2018

(I just updated the make stop command to make sure all containers are really stopped and removed)

@natea
Copy link
Author

natea commented Jun 9, 2018

Yeah, the xqueue docker container was not stopped for some reason, which is probably why i was getting that error before:

Stopping openedx-docker_lms_worker_1      ... done
Stopping openedx-docker_cms_worker_1      ... done
Stopping openedx-docker_lms_1             ... done
Stopping openedx-docker_cms_1             ... done
Stopping openedx-docker_xqueue_1          ... done
Stopping openedx-docker_xqueue_consumer_1 ... done
Stopping openedx-docker_mysql_1           ...
Stopping openedx-docker_mysql_1           ... done
Stopping openedx-docker_forum_1           ... done
Stopping openedx-docker_elasticsearch_1   ... done
Stopping openedx-docker_smtp_1            ... done
Stopping openedx-docker_rabbitmq_1        ... done
Stopping openedx-docker_mongodb_1         ... done
Stopping openedx-docker_memcached_1       ...
Stopping openedx-docker_memcached_1       ... done


nateaune at natemacbook in ~/Dropbox/code/openedx-docker (master●)
$ git pull
remote: Counting objects: 55, done.
remote: Compressing objects: 100% (18/18), done.
remote: Total 55 (delta 30), reused 55 (delta 30), pack-reused 0
Unpacking objects: 100% (55/55), done.
From https://github.com/regisb/openedx-docker
   8ba0887724..9d4743f00e  master     -> origin/master
Updating 8ba0887724..9d4743f00e
Fast-forward
 .gitignore                                 |  4 ++++
 Makefile                                   |  8 ++++----
 README.md                                  | 13 ++-----------
 config/mysql/templates/auth.env.templ      |  2 +-
 config/nginx/templates/cms.conf.templ      |  2 +-
 config/nginx/templates/lms.conf.templ      |  2 +-
 config/openedx/universal/cms/production.py |  3 +++
 config/openedx/universal/lms/production.py |  2 ++
 config/xqueue/templates/universal.py.templ |  1 +
 configure                                  | 16 +++++++++++-----
 openedx/Dockerfile                         |  7 ++++++-
 openedx/bin/wait-for-greenlight.sh         | 10 ----------
 12 files changed, 36 insertions(+), 34 deletions(-)
 delete mode 100755 openedx/bin/wait-for-greenlight.sh

nateaune at natemacbook in ~/Dropbox/code/openedx-docker (master●)
$ docker ps
CONTAINER ID        IMAGE                         COMMAND                  CREATED             STATUS                              PORTS               NAMES
4778136496cc        regis/openedx-xqueue:ginkgo   "./manage.py run_con…"   7 hours ago         Restarting (0) About a minute ago                       openedx-docker_xqueue_consumer_1

nateaune at natemacbook in ~/Dropbox/code/openedx-docker (master●)
$ make stop
docker-compose rm --stop --force
Stopping openedx-docker_xqueue_consumer_1 ... done
Going to remove openedx-docker_lms_worker_1, openedx-docker_cms_worker_1, openedx-docker_lms_1, openedx-docker_cms_1, openedx-docker_xqueue_1, openedx-docker_xqueue_consumer_1, openedx-docker_mysql_1, openedx-docker_forum_1, openedx-docker_elasticsearch_1, openedx-docker_smtp_1, openedx-docker_rabbitmq_1, openedx-docker_mongodb_1, openedx-docker_nginx_1, openedx-docker_memcached_1
Removing openedx-docker_lms_worker_1      ... done
Removing openedx-docker_cms_worker_1      ... done
Removing openedx-docker_lms_1             ... done
Removing openedx-docker_cms_1             ... done
Removing openedx-docker_xqueue_1          ... done
Removing openedx-docker_xqueue_consumer_1 ... done
Removing openedx-docker_mysql_1           ... done
Removing openedx-docker_forum_1           ... done
Removing openedx-docker_elasticsearch_1   ... done
Removing openedx-docker_smtp_1            ... done
Removing openedx-docker_rabbitmq_1        ... done
Removing openedx-docker_mongodb_1         ... done
Removing openedx-docker_nginx_1           ... done
Removing openedx-docker_memcached_1       ... done

nateaune at natemacbook in ~/Dropbox/code/openedx-docker (master●)
$ docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES

nateaune at natemacbook in ~/Dropbox/code/openedx-docker (master●)

@natea
Copy link
Author

natea commented Jun 9, 2018

Should I run the make update command or use make build as before? Is there any advantage to building the images myself, besides waiting for the slow images download on my home internet connection. ;)

@natea
Copy link
Author

natea commented Jun 9, 2018

I pulled down the images from your Docker Hub, and re-ran all the commands, and now getting this error:

xqueue_consumer_1  | Traceback (most recent call last):
xqueue_consumer_1  |   File "./manage.py", line 10, in <module>
xqueue_consumer_1  |     execute_from_command_line(sys.argv)
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/django/core/management/__init__.py", line 354, in execute_from_command_line
xqueue_consumer_1  |     utility.execute()
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/django/core/management/__init__.py", line 346, in execute
xqueue_consumer_1  |     self.fetch_command(subcommand).run_from_argv(self.argv)
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/django/core/management/base.py", line 394, in run_from_argv
xqueue_consumer_1  |     self.execute(*args, **cmd_options)
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/django/core/management/base.py", line 444, in execute
xqueue_consumer_1  |     self.check()
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/django/core/management/base.py", line 482, in check
xqueue_consumer_1  |     include_deployment_checks=include_deployment_checks,
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/django/core/checks/registry.py", line 72, in run_checks
xqueue_consumer_1  |     new_errors = check(app_configs=app_configs)
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/django/core/checks/model_checks.py", line 28, in check_all_models
xqueue_consumer_1  |     errors.extend(model.check(**kwargs))
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/django/db/models/base.py", line 1205, in check
xqueue_consumer_1  |     errors.extend(cls._check_fields(**kwargs))
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/django/db/models/base.py", line 1282, in _check_fields
xqueue_consumer_1  |     errors.extend(field.check(**kwargs))
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/django/db/models/fields/__init__.py", line 934, in check
xqueue_consumer_1  |     errors = super(AutoField, self).check(**kwargs)
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/django/db/models/fields/__init__.py", line 207, in check
xqueue_consumer_1  |     errors.extend(self._check_backend_specific_checks(**kwargs))
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/django/db/models/fields/__init__.py", line 306, in _check_backend_specific_checks
xqueue_consumer_1  |     return connection.validation.check_field(self, **kwargs)
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/django/db/backends/mysql/validation.py", line 18, in check_field
xqueue_consumer_1  |     field_type = field.db_type(connection)
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/django/db/models/fields/__init__.py", line 614, in db_type
xqueue_consumer_1  |     return connection.data_types[self.get_internal_type()] % data
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/django/db/__init__.py", line 36, in __getattr__
xqueue_consumer_1  |     return getattr(connections[DEFAULT_DB_ALIAS], item)
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/django/utils/functional.py", line 59, in __get__
xqueue_consumer_1  |     res = instance.__dict__[self.name] = self.func(instance)
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/django/db/backends/mysql/base.py", line 196, in data_types
xqueue_consumer_1  |     if self.features.supports_microsecond_precision:
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/django/utils/functional.py", line 59, in __get__
xqueue_consumer_1  |     res = instance.__dict__[self.name] = self.func(instance)
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/django/db/backends/mysql/features.py", line 52, in supports_microsecond_precision
xqueue_consumer_1  |     return self.connection.mysql_version >= (5, 6, 4) and Database.version_info >= (1, 2, 5)
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/django/utils/functional.py", line 59, in __get__
xqueue_consumer_1  |     res = instance.__dict__[self.name] = self.func(instance)
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/django/db/backends/mysql/base.py", line 371, in mysql_version
xqueue_consumer_1  |     with self.temporary_connection():
xqueue_consumer_1  |   File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
xqueue_consumer_1  |     return self.gen.next()
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/django/db/backends/base/base.py", line 462, in temporary_connection
xqueue_consumer_1  |     cursor = self.cursor()
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/django/db/backends/base/base.py", line 164, in cursor
xqueue_consumer_1  |     cursor = self.make_cursor(self._cursor())
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/django/db/backends/base/base.py", line 135, in _cursor
xqueue_consumer_1  |     self.ensure_connection()
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/django/db/backends/base/base.py", line 130, in ensure_connection
xqueue_consumer_1  |     self.connect()
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/django/db/utils.py", line 98, in __exit__
xqueue_consumer_1  |     six.reraise(dj_exc_type, dj_exc_value, traceback)
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/django/db/backends/base/base.py", line 130, in ensure_connection
xqueue_consumer_1  |     self.connect()
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/django/db/backends/base/base.py", line 119, in connect
xqueue_consumer_1  |     self.connection = self.get_new_connection(conn_params)
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/django/db/backends/mysql/base.py", line 276, in get_new_connection
xqueue_consumer_1  |     conn = Database.connect(**conn_params)
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/MySQLdb/__init__.py", line 81, in Connect
xqueue_consumer_1  |     return Connection(*args, **kwargs)
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/MySQLdb/connections.py", line 193, in __init__
xqueue_consumer_1  |     super(Connection, self).__init__(*args, **kwargs2)
xqueue_consumer_1  | django.db.utils.OperationalError: (2003, "Can't connect to MySQL server on 'mysql' (111)")

@regisb
Copy link
Contributor

regisb commented Jun 9, 2018 •
edited

What is the output of the make provision command?

(and building or pulling should just be a matter of personal preference: in theory, there shouldn't be any difference.)

@natea
Copy link
Author

natea commented Jun 9, 2018

For some reason, I can't stop the xqueue_consumer container.

^CGracefully stopping... (press Ctrl+C again to force)
Stopping openedx-docker_lms_worker_1      ... done
Stopping openedx-docker_lms_1             ... done
Stopping openedx-docker_forum_1           ... done
Stopping openedx-docker_cms_worker_1      ... done
Stopping openedx-docker_xqueue_1          ... done
Stopping openedx-docker_cms_1             ... done
Stopping openedx-docker_xqueue_consumer_1 ... done
Stopping openedx-docker_mysql_1           ... done
Stopping openedx-docker_nginx_1           ... done
Stopping openedx-docker_smtp_1            ... done
Stopping openedx-docker_memcached_1       ... done
Stopping openedx-docker_elasticsearch_1   ... done
Stopping openedx-docker_rabbitmq_1        ... done
Stopping openedx-docker_mongodb_1         ... done


nateaune at natemacbook in ~/Dropbox/code/openedx-docker (master●)
$ docker ps
CONTAINER ID        IMAGE                         COMMAND                  CREATED             STATUS                          PORTS               NAMES
b00de9457a66        regis/openedx-xqueue:ginkgo   "./manage.py run_con…"   14 hours ago        Restarting (0) 45 minutes ago                       openedx-docker_xqueue_consumer_1

nateaune at natemacbook in ~/Dropbox/code/openedx-docker (master●)
$ docker stop b00de9457a66
b00de9457a66

nateaune at natemacbook in ~/Dropbox/code/openedx-docker (master●)
$ docker ps
CONTAINER ID        IMAGE                         COMMAND                  CREATED             STATUS                          PORTS               NAMES
b00de9457a66        regis/openedx-xqueue:ginkgo   "./manage.py run_con…"   14 hours ago        Restarting (0) 45 minutes ago                       openedx-docker_xqueue_consumer_1

@natea
Copy link
Author

natea commented Jun 9, 2018

I just removed it, so now it's not running anymore:

nateaune at natemacbook in ~/Dropbox/code/openedx-docker (master●)
$ docker rm b00de9457a66
b00de9457a66

nateaune at natemacbook in ~/Dropbox/code/openedx-docker (master●)
$ docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES

nateaune at natemacbook in ~/Dropbox/code/openedx-docker (master●)

@natea
Copy link
Author

natea commented Jun 9, 2018

Here's the output of the make provision command:

$ make provision
docker-compose run --rm lms bash -c "dockerize -wait tcp://mysql:3306 -timeout 20s && bash /openedx/config/provision.sh"
Starting openedx-docker_mysql_1         ... done
Starting openedx-docker_rabbitmq_1      ... done
Starting openedx-docker_elasticsearch_1 ... done
Starting openedx-docker_mongodb_1       ... done
Starting openedx-docker_smtp_1          ... done
Starting openedx-docker_memcached_1     ... done
Starting openedx-docker_forum_1         ... done
2018/06/09 16:08:17 Waiting for: tcp://mysql:3306
2018/06/09 16:08:17 Problem with dial: dial tcp 172.18.0.4:3306: connect: connection refused. Sleeping 1s
2018/06/09 16:08:18 Problem with dial: dial tcp 172.18.0.4:3306: connect: connection refused. Sleeping 1s
2018/06/09 16:08:19 Connected to tcp://mysql:3306
mysql: [Warning] Using a password on the command line interface can be insecure.
mysql: [Warning] Using a password on the command line interface can be insecure.
mysql: [Warning] Using a password on the command line interface can be insecure.
mysql: [Warning] Using a password on the command line interface can be insecure.

@regisb
Copy link
Contributor

regisb commented Jun 9, 2018

Hmmm and you're still getting that xqueue error? I need to think about this...

@natea
Copy link
Author

natea commented Jun 11, 2018 via email

@regisb
Copy link
Contributor

regisb commented Jun 11, 2018

Let's try to diagnose this better. The error you are getting is still the following, right?

xqueue_consumer_1  | django.db.utils.OperationalError: (2003, "Can't connect to MySQL server on 'mysql' (111)")

Let's try the following steps:

  1. Log in to the xqueue_consumer container:

    docker-compose run --rm xqueue_consumer bash

  2. Try to ping the mysql container:

    apt install iputils-ping
    ping mysql

  3. Try to connect to mysql instance: (replace "mypassword" by the XQUEUE_MYSQL_PASSWORD value you will find inside config/config.json)

    apt install mysql-client
    mysql --user=xqueue --password=mypassword --host=mysql xqueue

Which steps failed?

@regisb regisb added the help wanted Easy issues that can be tackled by newcomers label Jun 11, 2018
@natea
Copy link
Author

natea commented Jun 13, 2018

@regisb i'm wondering if this had something to do with it... Notice the two xqueue containers, one which is restarting every few minutes?

$ docker ps
CONTAINER ID        IMAGE                         COMMAND                  CREATED             STATUS                          PORTS                NAMES
c592fc649d21        regis/openedx-xqueue:ginkgo   "./manage.py run_con…"   3 days ago          Restarting (0) 17 seconds ago                        openedx-docker_xqueue_consumer_1
4e92a05909c2        regis/openedx:ginkgo          "docker-entrypoint.s…"   4 days ago          Up 31 hours                     8000/tcp             openedx-docker_lms_worker_1
ca1d03eeb986        regis/openedx:ginkgo          "docker-entrypoint.s…"   4 days ago          Up 31 hours                     8000/tcp             openedx-docker_lms_1
a713493aab1f        regis/openedx-forum:ginkgo    "/bin/sh -c './bin/u…"   4 days ago          Up 31 hours                     4567/tcp             openedx-docker_forum_1
7f99c44df023        regis/openedx:ginkgo          "docker-entrypoint.s…"   4 days ago          Up 31 hours                     8000/tcp             openedx-docker_cms_worker_1
b6c48330d883        regis/openedx-xqueue:ginkgo   "/bin/sh -c 'gunicor…"   4 days ago          Up 31 hours                     8040/tcp             openedx-docker_xqueue_1
70ac45dc4093        regis/openedx:ginkgo          "docker-entrypoint.s…"   4 days ago          Up 31 hours                     8000/tcp             openedx-docker_cms_1
b2fd3947c711        mysql:5.6.36                  "docker-entrypoint.s…"   4 days ago          Up 31 hours                     3306/tcp             openedx-docker_mysql_1
20e910dbfa1a        nginx:1.13                    "nginx -g 'daemon of…"   4 days ago          Up 31 hours                     0.0.0.0:80->80/tcp   openedx-docker_nginx_1
94a075800bf2        namshi/smtp                   "/bin/entrypoint.sh …"   4 days ago          Up 31 hours                     25/tcp               openedx-docker_smtp_1
261a354f8f3e        memcached:1.4.38              "docker-entrypoint.s…"   4 days ago          Up 31 hours                     11211/tcp            openedx-docker_memcached_1
938abd73c96f        elasticsearch:1.5.2           "/docker-entrypoint.…"   4 days ago          Up 31 hours                     9200/tcp, 9300/tcp   openedx-docker_elasticsearch_1
4cc6b6bf6d24        mongo:3.0.14                  "docker-entrypoint.s…"   4 days ago          Up 31 hours                     27017/tcp            openedx-docker_mongodb_1

so I stopped and removed that container:

$ docker stop c592fc649d21
c592fc649d21

nateaune at natemacbook in ~/Dropbox/code/docker-vscode (master●)
$ docker rm c592fc649d21
c592fc649d21

nateaune at natemacbook in ~/Dropbox/code/docker-vscode (master●)
$ docker ps
CONTAINER ID        IMAGE                         COMMAND                  CREATED             STATUS              PORTS                NAMES
4e92a05909c2        regis/openedx:ginkgo          "docker-entrypoint.s…"   4 days ago          Up 31 hours         8000/tcp             openedx-docker_lms_worker_1
ca1d03eeb986        regis/openedx:ginkgo          "docker-entrypoint.s…"   4 days ago          Up 31 hours         8000/tcp             openedx-docker_lms_1
a713493aab1f        regis/openedx-forum:ginkgo    "/bin/sh -c './bin/u…"   4 days ago          Up 31 hours         4567/tcp             openedx-docker_forum_1
7f99c44df023        regis/openedx:ginkgo          "docker-entrypoint.s…"   4 days ago          Up 31 hours         8000/tcp             openedx-docker_cms_worker_1
b6c48330d883        regis/openedx-xqueue:ginkgo   "/bin/sh -c 'gunicor…"   4 days ago          Up 31 hours         8040/tcp             openedx-docker_xqueue_1
70ac45dc4093        regis/openedx:ginkgo          "docker-entrypoint.s…"   4 days ago          Up 31 hours         8000/tcp             openedx-docker_cms_1
b2fd3947c711        mysql:5.6.36                  "docker-entrypoint.s…"   4 days ago          Up 31 hours         3306/tcp             openedx-docker_mysql_1
20e910dbfa1a        nginx:1.13                    "nginx -g 'daemon of…"   4 days ago          Up 31 hours         0.0.0.0:80->80/tcp   openedx-docker_nginx_1
94a075800bf2        namshi/smtp                   "/bin/entrypoint.sh …"   4 days ago          Up 31 hours         25/tcp               openedx-docker_smtp_1
261a354f8f3e        memcached:1.4.38              "docker-entrypoint.s…"   4 days ago          Up 31 hours         11211/tcp            openedx-docker_memcached_1
938abd73c96f        elasticsearch:1.5.2           "/docker-entrypoint.…"   4 days ago          Up 31 hours         9200/tcp, 9300/tcp   openedx-docker_elasticsearch_1
4cc6b6bf6d24        mongo:3.0.14                  "docker-entrypoint.s…"   4 days ago          Up 31 hours         27017/tcp            openedx-docker_mongodb_1

@natea
Copy link
Author

natea commented Jun 13, 2018

I switched back to universal.production settings rather than my development settings to rule out that as the culprit. And sure enough once I did that, then the LMS and CMS came up fine without any problems. So I need to go debug my local environment to find out why it's not working.

I did run the commands you suggested, and everything worked as expected:

$ docker-compose run --rm xqueue_consumer bash
Starting openedx-docker_mysql_1 ... done
root@d5765b4b9e86:/openedx/xqueue# apt install iputils-ping
Reading package lists... Done
Building dependency tree
Processing triggers for libc-bin (2.23-0ubuntu10) ...
root@d5765b4b9e86:/openedx/xqueue# ping mysql
PING mysql (172.18.0.13) 56(84) bytes of data.
64 bytes from openedx-docker_mysql_1.openedx-docker_default (172.18.0.13): icmp_seq=1 ttl=64 time=0.229 ms
64 bytes from openedx-docker_mysql_1.openedx-docker_default (172.18.0.13): icmp_seq=2 ttl=64 time=0.102 ms
64 bytes from openedx-docker_mysql_1.openedx-docker_default (172.18.0.13): icmp_seq=3 ttl=64 time=0.109 ms
^C
--- mysql ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2027ms
rtt min/avg/max/mdev = 0.102/0.146/0.229/0.059 ms
root@d5765b4b9e86:/openedx/xqueue# apt install mysql-client
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following additional packages will be installed:
  libaio1 libnuma1 mysql-client-5.7 mysql-client-core-5.7
The following NEW packages will be installed:
  libaio1 libnuma1 mysql-client mysql-client-5.7 mysql-client-core-5.7
0 upgraded, 5 newly installed, 0 to remove and 0 not upgraded.
Need to get 8136 kB of archives.
After this operation, 65.6 MB of additional disk space will be used.
Do you want to continue? [Y/n] y
Get:1 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 libnuma1 amd64 2.0.11-1ubuntu1.1 [21.0 kB]
Get:2 http://archive.ubuntu.com/ubuntu xenial/main amd64 libaio1 amd64 0.3.110-2 [6356 B]
Get:3 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 mysql-client-core-5.7 amd64 5.7.22-0ubuntu0.16.04.1 [6425 kB]
Get:4 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 mysql-client-5.7 amd64 5.7.22-0ubuntu0.16.04.1 [1673 kB]
Get:5 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 mysql-client all 5.7.22-0ubuntu0.16.04.1 [10.0 kB]
Fetched 8136 kB in 11s (691 kB/s)
debconf: delaying package configuration, since apt-utils is not installed
Selecting previously unselected package libnuma1:amd64.
(Reading database ... 16714 files and directories currently installed.)
Preparing to unpack .../libnuma1_2.0.11-1ubuntu1.1_amd64.deb ...
Unpacking libnuma1:amd64 (2.0.11-1ubuntu1.1) ...
Selecting previously unselected package libaio1:amd64.
Preparing to unpack .../libaio1_0.3.110-2_amd64.deb ...
Unpacking libaio1:amd64 (0.3.110-2) ...
Selecting previously unselected package mysql-client-core-5.7.
Preparing to unpack .../mysql-client-core-5.7_5.7.22-0ubuntu0.16.04.1_amd64.deb ...
Unpacking mysql-client-core-5.7 (5.7.22-0ubuntu0.16.04.1) ...
Selecting previously unselected package mysql-client-5.7.
Preparing to unpack .../mysql-client-5.7_5.7.22-0ubuntu0.16.04.1_amd64.deb ...
Unpacking mysql-client-5.7 (5.7.22-0ubuntu0.16.04.1) ...
Selecting previously unselected package mysql-client.
Preparing to unpack .../mysql-client_5.7.22-0ubuntu0.16.04.1_all.deb ...
Unpacking mysql-client (5.7.22-0ubuntu0.16.04.1) ...
Processing triggers for libc-bin (2.23-0ubuntu10) ...
Setting up libnuma1:amd64 (2.0.11-1ubuntu1.1) ...
Setting up libaio1:amd64 (0.3.110-2) ...
Setting up mysql-client-core-5.7 (5.7.22-0ubuntu0.16.04.1) ...
Setting up mysql-client-5.7 (5.7.22-0ubuntu0.16.04.1) ...
Setting up mysql-client (5.7.22-0ubuntu0.16.04.1) ...
Processing triggers for libc-bin (2.23-0ubuntu10) ...
root@d5765b4b9e86:/openedx/xqueue# mysql --user=xqueue --password=HNpmPB1j --host=mysql xqueue
mysql: [Warning] Using a password on the command line interface can be insecure.
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 611
Server version: 5.6.36 MySQL Community Server (GPL)

Copyright (c) 2000, 2018, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql>

@regisb
Copy link
Contributor

regisb commented Jun 13, 2018

The xqueue_consumer keeps restasrting because it is configured to restart on failure: see restart: unless-stopped in docker-compose.yml.

Can you post your development settings?

@regisb regisb closed this as completed Jun 18, 2018
@natea
Copy link
Author

natea commented Jul 16, 2018

This one seems to have reared its ugly head again.

xqueue_consumer_1  | Traceback (most recent call last):
xqueue_consumer_1  |   File "./manage.py", line 10, in <module>
xqueue_consumer_1  |     execute_from_command_line(sys.argv)
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/django/core/management/__init__.py", line 354, in execute_from_command_line
xqueue_consumer_1  |     utility.execute()
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/django/core/management/__init__.py", line 346, in execute
xqueue_consumer_1  |     self.fetch_command(subcommand).run_from_argv(self.argv)
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/django/core/management/base.py", line 394, in run_from_argv
xqueue_consumer_1  |     self.execute(*args, **cmd_options)
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/django/core/management/base.py", line 444, in execute
xqueue_consumer_1  |     self.check()
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/django/core/management/base.py", line 482, in check
xqueue_consumer_1  |     include_deployment_checks=include_deployment_checks,
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/django/core/checks/registry.py", line 72, in run_checks
xqueue_consumer_1  |     new_errors = check(app_configs=app_configs)
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/django/core/checks/model_checks.py", line 28, in check_all_models
xqueue_consumer_1  |     errors.extend(model.check(**kwargs))
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/django/db/models/base.py", line 1205, in check
xqueue_consumer_1  |     errors.extend(cls._check_fields(**kwargs))
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/django/db/models/base.py", line 1282, in _check_fields
xqueue_consumer_1  |     errors.extend(field.check(**kwargs))
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/django/db/models/fields/__init__.py", line 934, in check
xqueue_consumer_1  |     errors = super(AutoField, self).check(**kwargs)
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/django/db/models/fields/__init__.py", line 207, in check
xqueue_consumer_1  |     errors.extend(self._check_backend_specific_checks(**kwargs))
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/django/db/models/fields/__init__.py", line 306, in _check_backend_specific_checks
xqueue_consumer_1  |     return connection.validation.check_field(self, **kwargs)
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/django/db/backends/mysql/validation.py", line 18, in check_field
xqueue_consumer_1  |     field_type = field.db_type(connection)
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/django/db/models/fields/__init__.py", line 614, in db_type
xqueue_consumer_1  |     return connection.data_types[self.get_internal_type()] % data
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/django/db/__init__.py", line 36, in __getattr__
xqueue_consumer_1  |     return getattr(connections[DEFAULT_DB_ALIAS], item)
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/django/utils/functional.py", line 59, in __get__
xqueue_consumer_1  |     res = instance.__dict__[self.name] = self.func(instance)
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/django/db/backends/mysql/base.py", line 196, in data_types
xqueue_consumer_1  |     if self.features.supports_microsecond_precision:
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/django/utils/functional.py", line 59, in __get__
xqueue_consumer_1  |     res = instance.__dict__[self.name] = self.func(instance)
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/django/db/backends/mysql/features.py", line 52, in supports_microsecond_precision
xqueue_consumer_1  |     return self.connection.mysql_version >= (5, 6, 4) and Database.version_info >= (1, 2, 5)
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/django/utils/functional.py", line 59, in __get__
xqueue_consumer_1  |     res = instance.__dict__[self.name] = self.func(instance)
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/django/db/backends/mysql/base.py", line 371, in mysql_version
xqueue_consumer_1  |     with self.temporary_connection():
xqueue_consumer_1  |   File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
xqueue_consumer_1  |     return self.gen.next()
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/django/db/backends/base/base.py", line 462, in temporary_connection
xqueue_consumer_1  |     cursor = self.cursor()
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/django/db/backends/base/base.py", line 164, in cursor
xqueue_consumer_1  |     cursor = self.make_cursor(self._cursor())
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/django/db/backends/base/base.py", line 135, in _cursor
xqueue_consumer_1  |     self.ensure_connection()
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/django/db/backends/base/base.py", line 130, in ensure_connection
xqueue_consumer_1  |     self.connect()
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/django/db/utils.py", line 98, in __exit__
xqueue_consumer_1  |     six.reraise(dj_exc_type, dj_exc_value, traceback)
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/django/db/backends/base/base.py", line 130, in ensure_connection
xqueue_consumer_1  |     self.connect()
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/django/db/backends/base/base.py", line 119, in connect
xqueue_consumer_1  |     self.connection = self.get_new_connection(conn_params)
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/django/db/backends/mysql/base.py", line 276, in get_new_connection
xqueue_consumer_1  |     conn = Database.connect(**conn_params)
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/MySQLdb/__init__.py", line 81, in Connect
xqueue_consumer_1  |     return Connection(*args, **kwargs)
xqueue_consumer_1  |   File "/usr/local/lib/python2.7/dist-packages/MySQLdb/connections.py", line 193, in __init__
xqueue_consumer_1  |     super(Connection, self).__init__(*args, **kwargs2)
xqueue_consumer_1  | django.db.utils.OperationalError: (2003, "Can't connect to MySQL server on 'mysql' (111)")

I got a mysql> prompt by following the steps you outlined above, so the xqueue consumer appears to be able to ping and connect to the MySQL server using the password as defined in config.json.
Why then the django.db.utils.OperationalError?

Does anything seem strange about the containers that are running:

CONTAINER ID        IMAGE                         COMMAND                  CREATED             STATUS                          PORTS                                NAMES
73ab6d877376        regis/openedx-xqueue:ginkgo   "./manage.py run_con…"   5 days ago          Restarting (0) 13 seconds ago                                        openedx-docker_xqueue_consumer_1
12ca516d4d9d        elasticsearch:1.5.2           "/docker-entrypoint.…"   5 days ago          Up 18 minutes                   9200/tcp, 9300/tcp                   openedx-docker_elasticsearch_1
9a439a8f3a8a        regis/openedx:ginkgo          "docker-entrypoint.s…"   5 days ago          Up 18 minutes                   8000/tcp                             openedx-docker_lms_worker_1
cfbb6da369e3        regis/openedx:ginkgo          "docker-entrypoint.s…"   5 days ago          Up 18 minutes                   8000/tcp                             openedx-docker_cms_worker_1
bd2d928b0880        regis/openedx:ginkgo          "docker-entrypoint.s…"   5 days ago          Up 18 minutes                   8000/tcp                             openedx-docker_lms_1
b602f3a0bd29        regis/openedx:ginkgo          "docker-entrypoint.s…"   5 days ago          Up 18 minutes                   8000/tcp                             openedx-docker_cms_1
83518f0378a2        regis/openedx-xqueue:ginkgo   "/bin/sh -c 'gunicor…"   5 days ago          Up 18 minutes                   8040/tcp                             openedx-docker_xqueue_1
1f65380c911b        regis/openedx-forum:ginkgo    "/bin/sh -c './bin/u…"   5 days ago          Up 17 minutes                   4567/tcp                             openedx-docker_forum_1
20a1f4529090        memcached:1.4.38              "docker-entrypoint.s…"   5 days ago          Up 18 minutes                   11211/tcp                            openedx-docker_memcached_1
661853b842bc        namshi/smtp                   "/bin/entrypoint.sh …"   5 days ago          Up 18 minutes                   25/tcp                               openedx-docker_smtp_1
453065872e92        nginx:1.13                    "nginx -g 'daemon of…"   5 days ago          Up 18 minutes                   0.0.0.0:80->80/tcp                   openedx-docker_nginx_1
616ca8b2e307        mysql:5.6.36                  "docker-entrypoint.s…"   5 days ago          Up 18 minutes                   3306/tcp                             openedx-docker_mysql_1
a1d400358a44        rabbitmq:3.6.10               "docker-entrypoint.s…"   5 days ago          Up 18 minutes                   4369/tcp, 5671-5672/tcp, 25672/tcp   openedx-docker_rabbitmq_1
d9c3110a26a2        mongo:3.0.14                  "docker-entrypoint.s…"   5 days ago          Up 18 minutes                   27017/tcp                            openedx-docker_mongodb_1

@regisb
Copy link
Contributor

regisb commented Jul 16, 2018

You have two different containers that run xqueue: 83518f0378a2 and 73ab6d877376. The problem is that you should have only one. From my understanding, the container with the mysql error is not inside the open edx "cluster" created from all the containers (lms, cms, mysql, etc.), so it does not have access to mysql. Try to kill the xqueue containers and restart it.

By the way, do you need the xqueue container? I'm thinking of removing it from the project.

@regisb regisb added question General questions are welcome and removed help wanted Easy issues that can be tackled by newcomers labels Mar 16, 2019
@regisb regisb mentioned this issue May 2, 2019
Closed
@mrvaghani mrvaghani mentioned this issue Nov 9, 2021
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
question General questions are welcome
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@regisb @natea