Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Zulip Server not creating users when using Reverse Proxy #12741

Open
worlddevops opened this issue Jul 9, 2019 · 12 comments

Comments

@worlddevops
Copy link

@worlddevops worlddevops commented Jul 9, 2019

I have set-up a vanilla Zulip Server and a NGINX Reverse Proxy but when I try to add the first organisation - I can view the registration page, and when I create the organisation, user and submit, it comes with Internal 500 web error, and the following message appears in the log.

The setup is like this -

            ┌─────────────┐      ┌────────────┐
┌────┐      │    NGINX    │      │ZULIP SERVER│
│User│─────▶│Reverse Proxy│─────▶│NGINX       │
└────┘      └─────────────┘      └────────────┘

The Error

2019-07-08 21:44:26.661 ERR  [django.request] Internal Server Error: /accounts/register/
Traceback (most recent call last):
  File "/home/zulip/deployments/2019-07-08-14-44-59/zulip-py3-venv/lib/python3.6/site-packages/django/db/models/fields/related_descriptors.py", line 178, in __get__
    rel_obj = getattr(instance, self.cache_name)
AttributeError: 'UserProfile' object has no attribute '_bot_owner_cache'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/zulip/deployments/2019-07-08-14-44-59/zulip-py3-venv/lib/python3.6/site-packages/django/core/handlers/exception.py", line 41, in inner
    response = get_response(request)
  File "/home/zulip/deployments/2019-07-08-14-44-59/zulip-py3-venv/lib/python3.6/site-packages/django/core/handlers/base.py", line 187, in _get_response
    response = self.process_exception_by_middleware(e, request)
  File "/home/zulip/deployments/2019-07-08-14-44-59/zulip-py3-venv/lib/python3.6/site-packages/django/core/handlers/base.py", line 185, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "./zerver/decorator.py", line 128, in wrapper
    return func(request, *args, **kwargs)
  File "./zerver/views/registration.py", line 196, in accounts_register
    realm = do_create_realm(string_id, realm_name)
  File "./zerver/lib/actions.py", line 3549, in do_create_realm
    "signups", realm.display_subdomain, signup_message)
  File "./zerver/lib/actions.py", line 2393, in internal_send_message
    content=content,
  File "./zerver/lib/actions.py", line 2316, in _internal_prep_message
    content, realm=realm)
  File "./zerver/lib/actions.py", line 2223, in check_message
    stream = validate_stream_name_with_pm_notification(stream_name, realm, sender)
  File "./zerver/lib/actions.py", line 2171, in validate_stream_name_with_pm_notification
    send_pm_if_empty_stream(sender, stream, stream_name, realm)
  File "./zerver/lib/actions.py", line 2097, in send_pm_if_empty_stream
    if not sender.is_bot or sender.bot_owner is None:
  File "/home/zulip/deployments/2019-07-08-14-44-59/zulip-py3-venv/lib/python3.6/site-packages/django/db/models/fields/related_descriptors.py", line 184, in __get__
    rel_obj = self.get_object(instance)
  File "/home/zulip/deployments/2019-07-08-14-44-59/zulip-py3-venv/lib/python3.6/site-packages/django/db/models/fields/related_descriptors.py", line 159, in get_object
    return qs.get(self.field.get_reverse_related_filter(instance))
  File "/home/zulip/deployments/2019-07-08-14-44-59/zulip-py3-venv/lib/python3.6/site-packages/django/db/models/query.py", line 380, in get
    self.model._meta.object_name
zerver.models.DoesNotExist: UserProfile matching query does not exist.

Nginx Reverse Proxy

nginx@dmz:~$ cat /etc/nginx/sites-available/mydomain.com 
map $http_upgrade $connection_upgrade {
        default upgrade;
        ''      close;
}
server {
        listen                  443 ssl;
        server_name             mydomain.com;

        ssl                     on;
        ssl_certificate         /etc/ssl/ntgclouds.com/mydomain.com.chain.crt;
       	ssl_certificate_key     /etc/ssl/ntgclouds.com/mydomain.com.key;
        access_log              /var/log/nginx/mydomain.com.access.log;

        location / {
                proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
                proxy_set_header        Host $http_host;
                proxy_set_header        Upgrade $http_upgrade;
                proxy_set_header        Connection $connection_upgrade;
                proxy_http_version      1.1;
                proxy_buffering         off;
                proxy_read_timeout      20m;
                proxy_pass              http://internal.zulip;
        }
}

Zulip Server

user@zulip-server:$ cat /etc/nginx/sites-enabled/zulip-enterprise 

include /etc/nginx/zulip-include/upstreams;

server {
    listen 80;
    listen [::]:80;

    location /user_avatars {
        add_header X-Content-Type-Options nosniff;
        add_header Content-Security-Policy "default-src 'none' img-src 'self'";
        include /etc/nginx/zulip-include/uploads.types;
        alias /home/zulip/uploads/avatars;
    }

    location /local-static {
        alias /home/zulip/local-static;
    }

    include /etc/nginx/zulip-include/certbot;
    include /etc/nginx/zulip-include/app;
}
@worlddevops

This comment has been minimized.

Copy link
Author

@worlddevops worlddevops commented Jul 9, 2019

I get the same error with HAProxy instead of NGINX

2019-07-09 17:36:00.979 ERR  [django.request] Internal Server Error: /accounts/register/
Traceback (most recent call last):
  File "/home/zulip/deployments/2019-07-08-14-44-59/zulip-py3-venv/lib/python3.6/site-packages/django/db/models/fields/related_descriptors.py", line 178, in __get__
    rel_obj = getattr(instance, self.cache_name)
AttributeError: 'UserProfile' object has no attribute '_bot_owner_cache'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/zulip/deployments/2019-07-08-14-44-59/zulip-py3-venv/lib/python3.6/site-packages/django/core/handlers/exception.py", line 41, in inner
    response = get_response(request)
  File "/home/zulip/deployments/2019-07-08-14-44-59/zulip-py3-venv/lib/python3.6/site-packages/django/core/handlers/base.py", line 187, in _get_response
    response = self.process_exception_by_middleware(e, request)
  File "/home/zulip/deployments/2019-07-08-14-44-59/zulip-py3-venv/lib/python3.6/site-packages/django/core/handlers/base.py", line 185, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "./zerver/decorator.py", line 128, in wrapper
    return func(request, *args, **kwargs)
  File "./zerver/views/registration.py", line 196, in accounts_register
    realm = do_create_realm(string_id, realm_name)
  File "./zerver/lib/actions.py", line 3549, in do_create_realm
    "signups", realm.display_subdomain, signup_message)
  File "./zerver/lib/actions.py", line 2393, in internal_send_message
    content=content,
  File "./zerver/lib/actions.py", line 2316, in _internal_prep_message
    content, realm=realm)
  File "./zerver/lib/actions.py", line 2223, in check_message
    stream = validate_stream_name_with_pm_notification(stream_name, realm, sender)
  File "./zerver/lib/actions.py", line 2171, in validate_stream_name_with_pm_notification
    send_pm_if_empty_stream(sender, stream, stream_name, realm)
  File "./zerver/lib/actions.py", line 2097, in send_pm_if_empty_stream
    if not sender.is_bot or sender.bot_owner is None:
  File "/home/zulip/deployments/2019-07-08-14-44-59/zulip-py3-venv/lib/python3.6/site-packages/django/db/models/fields/related_descriptors.py", line 184, in __get__
    rel_obj = self.get_object(instance)
  File "/home/zulip/deployments/2019-07-08-14-44-59/zulip-py3-venv/lib/python3.6/site-packages/django/db/models/fields/related_descriptors.py", line 159, in get_object
    return qs.get(self.field.get_reverse_related_filter(instance))
  File "/home/zulip/deployments/2019-07-08-14-44-59/zulip-py3-venv/lib/python3.6/site-packages/django/db/models/query.py", line 380, in get
    self.model._meta.object_name
zerver.models.DoesNotExist: UserProfile matching query does not exist.
@timabbott

This comment has been minimized.

Copy link
Member

@timabbott timabbott commented Jul 9, 2019

I've never seen this exception before. I suspect it isn't actually related to using a reverse proxy; it looks a lot more like some sort of weird issue with how your database or memcached is setup. How are you running your Zulip server?

Specifically, it kinda looks like the early-stage Zulip database initialization run by the installer was never run.

@worlddevops

This comment has been minimized.

Copy link
Author

@worlddevops worlddevops commented Jul 10, 2019

@timabbott Its a standard install on AWS Ubuntu Server

root@zulip-server:~# add-apt-repository universe
root@zulip-server:~# apt update
root@zulip-server:~# cd $(mktemp -d)
root@zulip-server:~# wget https://www.zulip.org/dist/releases/zulip-server-latest.tar.gz
root@zulip-server:~# tar -xf zulip-server-latest.tar.gz
root@zulip-server:~# ./zulip-server-*/scripts/setup/install --email=noreply@mydomain.tld --hostname=zulip.mydomain.tld --self-signed-cert

...
...
Please visit the following secure single-use link to register your 
new Zulip organization:

    https://zulip.mydomain.tld/new/plled9heblima09zgmlwp013

Is there a way to "register" a new Zulip organisation from the CLI instead of the webpage?

@timabbott

This comment has been minimized.

Copy link
Member

@timabbott timabbott commented Jul 10, 2019

We don't offer a new organization creation / registration code path entirely from the CLI; even if we did, I suspect whatever is unusual about your setup that is causing that exception would cause it there too.

I would try rerunning the installer (which should delete and reinitialize the database) and seeing if that helps. I've never seen that exception before, and it looks like some sort of internal exception in Django.

@zulipbot

This comment has been minimized.

Copy link
Member

@zulipbot zulipbot commented Jul 11, 2019

Hello @zulip/server-production members, this issue was labeled with the "area: production" label, so you may want to check it out!

@t3hmrman

This comment has been minimized.

Copy link

@t3hmrman t3hmrman commented Nov 4, 2019

Here's the error I get:

Logger django.request, from module django.core.handlers.exception line 135:
Error generated by Anonymous user (not logged in) on chat.domain.tld deployment

Traceback (most recent call last):
  File "/home/zulip/deployments/2019-09-12-01-25-54/zulip-py3-venv/lib/python3.6/site-packages/django/core/handlers/exception.py", line 41, in inner
    response = get_response(request)
  File "/home/zulip/deployments/2019-09-12-01-25-54/zulip-py3-venv/lib/python3.6/site-packages/django/core/handlers/base.py", line 187, in _get_response
    response = self.process_exception_by_middleware(e, request)
  File "/home/zulip/deployments/2019-09-12-01-25-54/zulip-py3-venv/lib/python3.6/site-packages/django/core/handlers/base.py", line 185, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "./zerver/decorator.py", line 128, in wrapper
    return func(request, *args, **kwargs)
  File "./zerver/views/registration.py", line 196, in accounts_register
    realm = do_create_realm(string_id, realm_name)
  File "./zerver/lib/actions.py", line 3547, in do_create_realm
    admin_realm = get_system_bot(settings.NOTIFICATION_BOT).realm
  File "./zerver/lib/cache.py", line 167, in func_with_caching
    val = func(*args, **kwargs)
  File "./zerver/models.py", line 1868, in get_system_bot
    return UserProfile.objects.select_related().get(email__iexact=email.strip())
  File "/home/zulip/deployments/2019-09-12-01-25-54/zulip-py3-venv/lib/python3.6/site-packages/django/db/models/query.py", line 380, in get
    self.model._meta.object_name
zerver.models.DoesNotExist: UserProfile matching query does not exist.

This error gets emailed to me after creating the initial Zulip organization and entering username and password.

When I check the DB I find that zerver_userprofile is empty (which explains the error). So the user creation failed, and no user profile got created.

After that if I attempt to use manage.py createsuperuser I get the following error:

zulip@zulip-589db58c5b-bzr75:/$ /home/zulip/deployments/current/manage.py createsuperuser
Email: user@domain.tld
Password: 
Password (again): 
Traceback (most recent call last):
  File "/home/zulip/deployments/current/manage.py", line 46, in <module>
    execute_from_command_line(sys.argv)
  File "/home/zulip/deployments/2019-09-12-01-25-54/zulip-py3-venv/lib/python3.6/site-packages/django/core/management/__init__.py", line 364, in execute_from_command_line
    utility.execute()
  File "/home/zulip/deployments/2019-09-12-01-25-54/zulip-py3-venv/lib/python3.6/site-packages/django/core/management/__init__.py", line 356, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/home/zulip/deployments/2019-09-12-01-25-54/zulip-py3-venv/lib/python3.6/site-packages/django/core/management/base.py", line 283, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/home/zulip/deployments/2019-09-12-01-25-54/zulip-py3-venv/lib/python3.6/site-packages/django/contrib/auth/management/commands/createsuperuser.py", line 63, in execute
    return super(Command, self).execute(*args, **options)
  File "/home/zulip/deployments/2019-09-12-01-25-54/zulip-py3-venv/lib/python3.6/site-packages/django/core/management/base.py", line 330, in execute
    output = self.handle(*args, **options)
  File "/home/zulip/deployments/2019-09-12-01-25-54/zulip-py3-venv/lib/python3.6/site-packages/django/contrib/auth/management/commands/createsuperuser.py", line 183, in handle
    self.UserModel._default_manager.db_manager(database).create_superuser(**user_data)
TypeError: create_superuser() missing 1 required positional argument: 'username'

After that failed I tried to create a regular user instead:

$ /home/zulip/deployments/current/manage.py create_user --this-user-has-accepted-the-tos --password-file /tmp/password user@domain.tld "User Name" -r 1

The first time I run that, I get the same old UserProfile matching query does not exist error. The second time (and after checking zerver_userprofile in the DB and seeing a row in there), I get User already exists.

After this, I wasn't able to log in with the password I set (either through ENV variables or via the CLI commands), but I was able to reset my password from the web side, and log in.

It feels like I'm running into multiple timing/race issues, but this is what worked for me.

I had earlier issues with 500 error pages, and realized that my reverse proxying wasn't redirecting all the pages it should have been. So for those that are having this problem maybe that will help. In my very specific case, I specified a path (i.e. path: /) on my Kubernetes Ingress object where I should have not specified path at all IIRC and some services did not have endpoints showing up properly.

@t3hmrman

This comment has been minimized.

Copy link

@t3hmrman t3hmrman commented Nov 4, 2019

After trying to sign up with a different email on my instance I'm convinced that this is actually a timing issue -- when users get initially created, there's a bit of a delay until the user profile actually gets made it seems, but the process tries to move along too quickly. I get a 500, but if I wait about ~30 seconds I can go back and log in

@timabbott

This comment has been minimized.

Copy link
Member

@timabbott timabbott commented Nov 6, 2019

@t3hmrman can you provide more detail on precisely what you did to setup your Zulip installation? It sounds like you're using Kubernetes, is it possible you accidentally deleted your Zulip database in your postgres container (or changed paths in a way where it lost access to its storage)?

That would be a simple explanation for this issue -- the Zulip app's Docker image has some first-time-startup logic to initialize the database, but that logic only runs once (using a stamp file to determine whether to do so), so if you destroy and rebuild the database container but not the zulip container's storage, you'll end up with an empty/uninitialized database where that work needs to be done.

I'd like to understand what precisely you did to get into that situation, since this seems like something we should address through changes to our Docker image or its documentation. In any case, this is unrelated to the original report, which appears to be a version mismatch between the Zulip code and its database configuration (likely resulting from installing different versions at different times on the same system and unknown other fiddling)

(As a sidenote, manage.py createsuperuser is part of Django's default auth module that we don't use; so that's not expected to do anything useful.)

@t3hmrman

This comment has been minimized.

Copy link

@t3hmrman t3hmrman commented Nov 6, 2019

Hi @timabbott thanks for the detailed response, sorry to take up some of your likely busy day.

@t3hmrman can you provide more detail on precisely what you did to setup your Zulip installation? It sounds like you're using Kubernetes, is it possible you accidentally deleted your Zulip database in your postgres container (or changed paths in a way where it lost access to its storage)?

I recently spent some time reworking and breaking out the Zulip pieces into their own Deployments and Services, etc (after using what the docker-zulip project provided). I know that I changed over PVCs when doing the switch, but I think it is the case that I cleared out the database but did not clear out Zulip's storage as well -- I deleted the database PVC on but I don't lucidly remember deleting zulip's data PVC as well as I was working out how to put everything together.

I've actually written about it in a blog post, and I was wondering if I should make a PR to docker-zulip with the broken out (but slightly more recent k8s-convention-wise) resource configurations.

I'd like to understand what precisely you did to get into that situation, since this seems like something we should address through changes to our Docker image or its documentation. In any case, this is unrelated to the original report, which appears to be a version mismatch between the Zulip code and its database configuration (likely resulting from installing different versions at different times on the same system and unknown other fiddling)

Part of my issue was an incorrect Ingress specification (some of the 500s were 502s AKA Bad Gateway errors), but the only persistent issue at present is when new users sign up I get exactly one UserProfile not found error for the user that just signed up.

Conveniently (yay k8s), I can just re-apply my setup to a completely new namespace and see how it goes there, and I'll get completely new PVCs, Deployments, Services, etc. I will do that and see how the flow differs or if any of these bugs are persistent (I assume they won't be except for the UserProfile does not exist). I do want to re-iterate that the UserProfile does not exist error occurs but the user can log in -- and the row in zerver_userprofile representing the user that was just created is present within seconds (at least faster than I can log in w/ psql and check for it).

Thanks for clarifying that this is unrelated, would you prefer I made a new ticket? Maybe one in docker-zulip instead?

@timabbott

This comment has been minimized.

Copy link
Member

@timabbott timabbott commented Nov 6, 2019

@timabbott

This comment has been minimized.

Copy link
Member

@timabbott timabbott commented Nov 6, 2019

@t3hmrman

This comment has been minimized.

Copy link

@t3hmrman t3hmrman commented Nov 6, 2019

Will do, I'll find some time this week and attempt to reproduce if I can and get a PR up to help with docker-zulip!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
5 participants
You can’t perform that action at this time.