Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

middleware: Detect reverse proxy misconfigurations. #26046

Merged
merged 3 commits into from
Jul 2, 2023

Conversation

alexmv
Copy link
Collaborator

@alexmv alexmv commented Jun 16, 2023

Combine nginx and Django middlware to stop putting misleading warnings
about CSRF_TRUSTED_ORIGINS when the issue is untrusted proxies.
This attempts to, in the error logs, diagnose and suggest next steps
to fix common proxy misconfigurations.

See also #24599 and zulip/docker-zulip#403.

Self-review checklist
  • Self-reviewed the changes for clarity and maintainability
    (variable names, code reuse, readability, etc.).

Communicate decisions, questions, and potential concerns.

  • Explains differences from previous plans (e.g., issue description).
  • Highlights technical choices and bugs encountered.
  • Calls out remaining decisions and concerns.
  • Automated tests verify logic where appropriate.

Individual commits are ready for review (see commit discipline).

  • Each commit is a coherent idea.
  • Commit message(s) explain reasoning and motivation for changes.

Completed manual review and testing of the following:

  • Visual appearance of the changes.
  • Responsiveness and internationalization.
  • Strings and tooltips.
  • End-to-end functionality of buttons, interactions and flows.
  • Corner cases, error conditions, and easily imagined bugs.

@alexmv alexmv force-pushed the proxy-misconfiguration branch 2 times, most recently from b1271f7 to ab3c02b Compare June 16, 2023 19:26
@alexmv alexmv added the integration review Added by maintainers when a PR may be ready for integration. label Jun 21, 2023
@alexmv
Copy link
Collaborator Author

alexmv commented Jun 22, 2023

This seems to reliably wedge the Debian 12 (Python 3.11, backend) testsuite, which is fascinating.

@alexmv
Copy link
Collaborator Author

alexmv commented Jun 22, 2023

Oh, it apparently segfaults Python:

Fatal Python error: Segmentation fault

Current thread 0x00007f4bf6f162c0 (most recent call first):
  Garbage-collecting
  File "/srv/zulip-py3-venv/lib/python3.11/site-packages/django/db/models/sql/query.py", line 2439 in extra_select
  File "/srv/zulip-py3-venv/lib/python3.11/site-packages/django/db/models/sql/compiler.py", line 527 in quote_name_unless_alias
  File "/srv/zulip-py3-venv/lib/python3.11/site-packages/django/db/models/expressions.py", line 1133 in as_sql
  File "/srv/zulip-py3-venv/lib/python3.11/site-packages/django/db/models/sql/compiler.py", line 544 in compile
  File "/srv/zulip-py3-venv/lib/python3.11/site-packages/django/db/models/sql/compiler.py", line 296 in get_select
  File "/srv/zulip-py3-venv/lib/python3.11/site-packages/django/db/models/sql/compiler.py", line 73 in setup_query
  File "/srv/zulip-py3-venv/lib/python3.11/site-packages/django/db/models/sql/compiler.py", line 84 in pre_sql_setup
  File "/srv/zulip-py3-venv/lib/python3.11/site-packages/django/db/models/sql/compiler.py", line 734 in as_sql
  File "/srv/zulip-py3-venv/lib/python3.11/site-packages/django/db/models/sql/compiler.py", line 1547 in execute_sql
  File "/srv/zulip-py3-venv/lib/python3.11/site-packages/django/db/models/query.py", line 91 in __iter__
  File "/srv/zulip-py3-venv/lib/python3.11/site-packages/django/db/models/query.py", line 1881 in _fetch_all
  File "/srv/zulip-py3-venv/lib/python3.11/site-packages/django/db/models/query.py", line 380 in __len__
  File "/srv/zulip-py3-venv/lib/python3.11/site-packages/django/db/models/query.py", line 633 in get
  File "/__w/zulip/zulip/zerver/models.py", line 3836 in get_user_profile_by_id
  File "/__w/zulip/zulip/zerver/lib/cache.py", line 157 in func_with_caching
  File "/__w/zulip/zulip/zproject/backends.py", line 419 in get_user
  File "/srv/zulip-py3-venv/lib/python3.11/site-packages/django/contrib/auth/__init__.py", line 198 in get_user
  File "/srv/zulip-py3-venv/lib/python3.11/site-packages/django/contrib/auth/middleware.py", line 11 in get_user
  File "/srv/zulip-py3-venv/lib/python3.11/site-packages/django/contrib/auth/middleware.py", line 25 in <lambda>
  File "/srv/zulip-py3-venv/lib/python3.11/site-packages/django/utils/functional.py", line 419 in _setup
  File "/srv/zulip-py3-venv/lib/python3.11/site-packages/django/utils/functional.py", line 310 in __setattr__
  File "/srv/zulip-py3-venv/lib/python3.11/site-packages/django_otp/middleware.py", line 41 in _verify_user
  File "/srv/zulip-py3-venv/lib/python3.11/site-packages/django/utils/functional.py", line 419 in _setup
  File "/srv/zulip-py3-venv/lib/python3.11/site-packages/django/utils/functional.py", line 266 in inner
  File "/__w/zulip/zulip/zerver/lib/rest.py", line 170 in rest_dispatch
  File "/srv/zulip-py3-venv/lib/python3.11/site-packages/django/views/decorators/csrf.py", line 56 in wrapper_view
  File "/__w/zulip/zulip/zerver/lib/rest.py", line 39 in _wrapped_view_func
  File "/srv/zulip-py3-venv/lib/python3.11/site-packages/django/core/handlers/base.py", line 197 in _get_response
  File "/srv/zulip-py3-venv/lib/python3.11/site-packages/django/core/handlers/exception.py", line 55 in inner
  File "/srv/zulip-py3-venv/lib/python3.11/site-packages/django/utils/deprecation.py", line 134 in __call__
  File "/srv/zulip-py3-venv/lib/python3.11/site-packages/django/core/handlers/exception.py", line 55 in inner
  File "/srv/zulip-py3-venv/lib/python3.11/site-packages/two_factor/middleware/threadlocals.py", line 19 in __call__
  File "/srv/zulip-py3-venv/lib/python3.11/site-packages/django/core/handlers/exception.py", line 55 in inner
  File "/srv/zulip-py3-venv/lib/python3.11/site-packages/django_otp/middleware.py", line 35 in __call__
  File "/srv/zulip-py3-venv/lib/python3.11/site-packages/django/core/handlers/exception.py", line 55 in inner
  File "/srv/zulip-py3-venv/lib/python3.11/site-packages/django/utils/deprecation.py", line 134 in __call__
  File "/srv/zulip-py3-venv/lib/python3.11/site-packages/django/core/handlers/exception.py", line 55 in inner
  File "/srv/zulip-py3-venv/lib/python3.11/site-packages/django/utils/deprecation.py", line 134 in __call__
  File "/srv/zulip-py3-venv/lib/python3.11/site-packages/django/core/handlers/exception.py", line 55 in inner
  File "/srv/zulip-py3-venv/lib/python3.11/site-packages/django/utils/deprecation.py", line 134 in __call__
  File "/srv/zulip-py3-venv/lib/python3.11/site-packages/django/core/handlers/exception.py", line 55 in inner
  File "/srv/zulip-py3-venv/lib/python3.11/site-packages/django/utils/deprecation.py", line 134 in __call__
  File "/srv/zulip-py3-venv/lib/python3.11/site-packages/django/core/handlers/exception.py", line 55 in inner
  File "/srv/zulip-py3-venv/lib/python3.11/site-packages/django/utils/deprecation.py", line 134 in __call__
  File "/srv/zulip-py3-venv/lib/python3.11/site-packages/django/core/handlers/exception.py", line 55 in inner
  File "/srv/zulip-py3-venv/lib/python3.11/site-packages/django/utils/deprecation.py", line 134 in __call__
  File "/srv/zulip-py3-venv/lib/python3.11/site-packages/django/core/handlers/exception.py", line 55 in inner
  File "/srv/zulip-py3-venv/lib/python3.11/site-packages/django/utils/deprecation.py", line 134 in __call__
  File "/srv/zulip-py3-venv/lib/python3.11/site-packages/django/core/handlers/exception.py", line 55 in inner
  File "/srv/zulip-py3-venv/lib/python3.11/site-packages/django/utils/deprecation.py", line 134 in __call__
  File "/srv/zulip-py3-venv/lib/python3.11/site-packages/django/core/handlers/exception.py", line 55 in inner
  File "/srv/zulip-py3-venv/lib/python3.11/site-packages/django/utils/deprecation.py", line 134 in __call__
  File "/srv/zulip-py3-venv/lib/python3.11/site-packages/django/core/handlers/exception.py", line 55 in inner
  File "/srv/zulip-py3-venv/lib/python3.11/site-packages/django/utils/deprecation.py", line 134 in __call__
  File "/srv/zulip-py3-venv/lib/python3.11/site-packages/django/core/handlers/exception.py", line 55 in inner
  File "/srv/zulip-py3-venv/lib/python3.11/site-packages/django/utils/deprecation.py", line 134 in __call__
  File "/srv/zulip-py3-venv/lib/python3.11/site-packages/django/core/handlers/exception.py", line 55 in inner
  File "/__w/zulip/zulip/zerver/middleware.py", line 245 in __call__
  File "/srv/zulip-py3-venv/lib/python3.11/site-packages/django/core/handlers/exception.py", line 55 in inner
  File "/srv/zulip-py3-venv/lib/python3.11/site-packages/django/utils/deprecation.py", line 134 in __call__
  File "/srv/zulip-py3-venv/lib/python3.11/site-packages/django/core/handlers/exception.py", line 55 in inner
  File "/srv/zulip-py3-venv/lib/python3.11/site-packages/django/utils/deprecation.py", line 134 in __call__
  File "/srv/zulip-py3-venv/lib/python3.11/site-packages/django/core/handlers/exception.py", line 55 in inner
  File "/srv/zulip-py3-venv/lib/python3.11/site-packages/django/core/handlers/base.py", line 140 in get_response
  File "/srv/zulip-py3-venv/lib/python3.11/site-packages/django/test/client.py", line 176 in __call__
  File "/srv/zulip-py3-venv/lib/python3.11/site-packages/django/test/client.py", line 886 in request
  File "/srv/zulip-py3-venv/lib/python3.11/site-packages/django/test/client.py", line 609 in generic
  File "/srv/zulip-py3-venv/lib/python3.11/site-packages/django/test/client.py", line 553 in patch
  File "/srv/zulip-py3-venv/lib/python3.11/site-packages/django/test/client.py", line 1054 in patch
  File "/__w/zulip/zulip/zerver/lib/test_classes.py", line 320 in client_patch
  File "/__w/zulip/zulip/zerver/lib/test_helpers.py", line 407 in wrapper
  File "/__w/zulip/zulip/zerver/tests/test_custom_profile_data.py", line 616 in assert_error_update_invalid_value
  File "/__w/zulip/zulip/zerver/tests/test_custom_profile_data.py", line 635 in test_update_invalid_short_text
  File "/usr/lib/python3.11/unittest/case.py", line [579](https://github.com/zulip/zulip/actions/runs/5293519081/jobs/9672947227#step:13:580) in _callTestMethod
  File "/usr/lib/python3.11/unittest/case.py", line [623](https://github.com/zulip/zulip/actions/runs/5293519081/jobs/9672947227#step:13:624) in run
  File "/__w/zulip/zulip/zerver/lib/test_classes.py", line 179 in run
  File "/usr/lib/python3.11/unittest/case.py", line 678 in __call__
  File "/srv/zulip-py3-venv/lib/python3.11/site-packages/django/test/testcases.py", line 416 in _setup_and_call
  File "/srv/zulip-py3-venv/lib/python3.11/site-packages/django/test/testcases.py", line 381 in __call__
  File "/usr/lib/python3.11/unittest/suite.py", line 122 in run
  File "/usr/lib/python3.11/unittest/suite.py", line 84 in __call__
  File "/srv/zulip-py3-venv/lib/python3.11/site-packages/django/test/runner.py", line 366 in run
  File "/__w/zulip/zulip/zerver/lib/test_runner.py", line 117 in run_subsuite
  File "/usr/lib/python3.11/multiprocessing/pool.py", line 125 in worker
  File "/usr/lib/python3.11/multiprocessing/process.py", line 108 in run
  File "/usr/lib/python3.11/multiprocessing/process.py", line 314 in _bootstrap
  File "/srv/zulip-py3-venv/lib/python3.11/site-packages/coverage/multiproc.py", line 44 in _bootstrap
  File "/usr/lib/python3.11/multiprocessing/popen_fork.py", line 71 in _launch
  File "/usr/lib/python3.11/multiprocessing/popen_fork.py", line 19 in __init__
  File "/usr/lib/python3.11/multiprocessing/context.py", line 281 in _Popen
  File "/usr/lib/python3.11/multiprocessing/process.py", line 121 in start
  File "/usr/lib/python3.11/multiprocessing/pool.py", line 329 in _repopulate_pool_static
  File "/usr/lib/python3.11/multiprocessing/pool.py", line 306 in _repopulate_pool
  File "/usr/lib/python3.11/multiprocessing/pool.py", line 215 in __init__
  File "/usr/lib/python3.11/multiprocessing/context.py", line 119 in Pool
  File "/srv/zulip-py3-venv/lib/python3.11/site-packages/django/test/runner.py", line 510 in run
  File "/usr/lib/python3.11/unittest/suite.py", line 84 in __call__
  File "/usr/lib/python3.11/unittest/runner.py", line 217 in run
  File "/srv/zulip-py3-venv/lib/python3.11/site-packages/django/test/runner.py", line 983 in run_suite
  File "/__w/zulip/zulip/zerver/lib/test_runner.py", line 396 in run_tests
  ...

Extension modules: yaml._yaml, charset_normalizer.md, markupsafe._speedups, _ldap, psycopg2._psycopg, tornado.speedups, _cffi_backend, PIL._imaging, ahocorasick, lxml._elementpath, lxml.etree, cchardet._cchardet, sqlalchemy.cimmutabledict, greenlet._greenlet, sqlalchemy.cprocessors, sqlalchemy.cresultproxy, lxml.builder, xmlsec, ujson, lazy_object_proxy.cext, _time_machine, bson._cbson, lxml.html.diff (total: 23)

@alexmv
Copy link
Collaborator Author

alexmv commented Jun 22, 2023

CZO thread for the crash.

alexmv referenced this pull request Jun 26, 2023
Previously, `X-Forwarded-Proto` did not need to be set, and failure to
set `loadbalancer.ips` would merely result in bad IP-address
rate-limiting and incorrect access logs; after 0935d38, however,
failure to do either of those, if Zulip is deployed with `http_only`,
will lead to infinite redirect loops after login.  These are
accompanied by a misleading error, from Tornado, of:

    Forbidden (Origin checking failed - https://zulip.example.com does not match any trusted origins.): /json/events

This is most common with Docker deployments, where deployments use
another docker container, such as nginx or Traefik, to do SSL
termination.  See zulip/docker-zulip#403.

Update the documentation to reinforce that `loadbalancer.ips` also
controls trust of `X-Forwarded-Proto`, and that failure to set it will
cause the application to not function correctly.
@alexmv alexmv force-pushed the proxy-misconfiguration branch 2 times, most recently from 8722517 to 1b73158 Compare June 26, 2023 20:21
zerver/middleware.py Outdated Show resolved Hide resolved
@timabbott
Copy link
Sponsor Member

This lgtm; posted one small comment on readability. Given the intent to immediately backport this, it'd be great if @andersk did a read as well; I am not very good at reading the nginx configuration language.

Having exactly 17 or 18 middlewares, on Python 3.11.0 and above,
causes python to segfault when running tests with coverage; see
python/cpython#106092

Work around this by adding one or two no-op middlewares if we would
hit those unlucky numbers.  We only add them in testing, since
coverage is a requirement to trigger it, and there is no reason to
burden production with additional wrapping.
Combine nginx and Django middlware to stop putting misleading warnings
about `CSRF_TRUSTED_ORIGINS` when the issue is untrusted proxies.
This attempts to, in the error logs, diagnose and suggest next steps
to fix common proxy misconfigurations.

See also zulip#24599 and zulip/docker-zulip#403.
@timabbott timabbott added the backport candidate Items we should consider backporting to the bug fix release series. label Jun 30, 2023
Copy link
Member

@andersk andersk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I haven’t tested this, but it looks reasonable to me.

@timabbott timabbott merged commit 8a77cca into zulip:main Jul 2, 2023
17 checks passed
@timabbott
Copy link
Sponsor Member

Merged, thanks @alexmv!

@alexmv
Copy link
Collaborator Author

alexmv commented Jul 3, 2023

Backported as 8f98071, 2f91471, and 738429c.

@alexmv alexmv removed the backport candidate Items we should consider backporting to the bug fix release series. label Jul 3, 2023
@alexmv alexmv deleted the proxy-misconfiguration branch July 6, 2023 23:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
integration review Added by maintainers when a PR may be ready for integration. size: L
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants