Skip to content
This repository has been archived by the owner on Apr 26, 2024. It is now read-only.

allow specifying https:// proxy #10411

Merged
merged 31 commits into from
Jul 27, 2021
Merged

Conversation

dklimpel
Copy link
Contributor

@dklimpel dklimpel commented Jul 16, 2021

Replace and follow up: #9119
Fixes: #9090

I have tried to do some smaller commits for better reading.

Pull Request Checklist

  • Pull request is based on the develop branch
  • Pull request includes a changelog file. The entry should:
    • Be a short description of your change which makes sense to users. "Fixed a bug that prevented receiving messages from other servers." instead of "Moved X method from EventStore to EventWorkerStore.".
    • Use markdown where necessary, mostly for code blocks.
    • End with either a period (.) or an exclamation mark (!).
    • Start with a capital letter.
  • Pull request includes a sign off
  • Code style is correct (run the linters)

Signed-off-by: Dirk Klimpel dirk@klimpel.org

@dklimpel dklimpel marked this pull request as ready for review July 16, 2021 07:35
@richvdh richvdh requested a review from a team July 16, 2021 08:02
@richvdh richvdh mentioned this pull request Jul 20, 2021
4 tasks
Copy link
Member

@richvdh richvdh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this looks excellent. A couple of nits. I'll have a look to see what the problem with the tests is.

synapse/http/proxyagent.py Outdated Show resolved Hide resolved
synapse/http/proxyagent.py Outdated Show resolved Hide resolved
synapse/http/proxyagent.py Outdated Show resolved Hide resolved
If no credentials were found, the ProxyCredentials instance is replaced with None.

Raise:
RuntimeError if proxy has no hostname or unsupported scheme.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why RuntimeError rather than ValueError?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok, but I see that as a bug in sygnal rather than something we ought to replicate here.

synapse/http/proxyagent.py Outdated Show resolved Hide resolved
synapse/http/proxyagent.py Outdated Show resolved Hide resolved
tests/http/test_proxyagent.py Outdated Show resolved Hide resolved
dklimpel and others added 2 commits July 20, 2021 16:00
Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
@richvdh richvdh self-requested a review July 20, 2021 15:32
Copy link
Member

@richvdh richvdh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@dklimpel I've pushed a change which fixes up the tests. The problem was that the way we were setting up the connections meant that the traffic from the client was going straight to the server, rather than first being un-SSLed by the proxy.

Could you switch the exceptions back to ValueError and then I think we can merge this?

@dklimpel
Copy link
Contributor Author

@dklimpel I've pushed a change which fixes up the tests. The problem was that the way we were setting up the connections meant that the traffic from the client was going straight to the server, rather than first being un-SSLed by the proxy.

Could you switch the exceptions back to ValueError and then I think we can merge this?

Great.
This test is a little bit to crazy for me. :)

Copy link
Member

@richvdh richvdh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm. thanks both @dklimpel and @Bubu !

@richvdh richvdh merged commit 076dead into matrix-org:develop Jul 27, 2021
@dklimpel dklimpel deleted the https_proxy_new branch July 27, 2021 21:09
aaronraimist added a commit to aaronraimist/synapse that referenced this pull request Aug 13, 2021
Synapse 1.40.0 (2021-08-10)
===========================

No significant changes.

Synapse 1.40.0rc3 (2021-08-09)
==============================

Features
--------

- Support [MSC3289: room version 8](matrix-org/matrix-spec-proposals#3289). ([\matrix-org#10449](matrix-org#10449))

Bugfixes
--------

- Mark the experimental room version from [MSC2716](matrix-org/matrix-spec-proposals#2716) as unstable. ([\matrix-org#10449](matrix-org#10449))

Improved Documentation
----------------------

- Fix broken links in `upgrade.md`. Contributed by @dklimpel. ([\matrix-org#10543](matrix-org#10543))

Synapse 1.40.0rc2 (2021-08-04)
==============================

Bugfixes
--------

- Fix the `PeriodicallyFlushingMemoryHandler` inhibiting application shutdown because of its background thread. ([\matrix-org#10517](matrix-org#10517))
- Fix a bug introduced in Synapse v1.40.0rc1 that could cause Synapse to respond with an error when clients would update read receipts. ([\matrix-org#10531](matrix-org#10531))

Internal Changes
----------------

- Fix release script to open the correct URL for the release. ([\matrix-org#10516](matrix-org#10516))

Synapse 1.40.0rc1 (2021-08-03)
==============================

Features
--------

- Add support for [MSC2033](matrix-org/matrix-spec-proposals#2033): `device_id` on `/account/whoami`. ([\matrix-org#9918](matrix-org#9918))
- Update support for [MSC2716 - Incrementally importing history into existing rooms](matrix-org/matrix-spec-proposals#2716). ([\matrix-org#10245](matrix-org#10245), [\matrix-org#10432](matrix-org#10432), [\matrix-org#10463](matrix-org#10463))
- Update support for [MSC3083](matrix-org/matrix-spec-proposals#3083) to consider changes in the MSC around which servers can issue join events. ([\matrix-org#10254](matrix-org#10254), [\matrix-org#10447](matrix-org#10447), [\matrix-org#10489](matrix-org#10489))
- Initial support for [MSC3244](matrix-org/matrix-spec-proposals#3244), Room version capabilities over the /capabilities API. ([\matrix-org#10283](matrix-org#10283))
- Add a buffered logging handler which periodically flushes itself. ([\matrix-org#10407](matrix-org#10407), [\matrix-org#10515](matrix-org#10515))
- Add support for https connections to a proxy server. Contributed by @Bubu and @dklimpel. ([\matrix-org#10411](matrix-org#10411))
- Support for [MSC2285 (hidden read receipts)](matrix-org/matrix-spec-proposals#2285). Contributed by @SimonBrandner. ([\matrix-org#10413](matrix-org#10413))
- Email notifications now state whether an invitation is to a room or a space. ([\matrix-org#10426](matrix-org#10426))
- Allow setting transaction limit for database connections. ([\matrix-org#10440](matrix-org#10440), [\matrix-org#10511](matrix-org#10511))
- Add `creation_ts` to "list users" admin API. ([\matrix-org#10448](matrix-org#10448))

Bugfixes
--------

- Improve character set detection in URL previews by supporting underscores (in addition to hyphens). Contributed by @srividyut. ([\matrix-org#10410](matrix-org#10410))
- Fix events being incorrectly rejected over federation if they reference auth events that the server needed to fetch. ([\matrix-org#10439](matrix-org#10439))
- Fix `synapse_federation_server_oldest_inbound_pdu_in_staging` Prometheus metric to not report a max age of 51 years when the queue is empty. ([\matrix-org#10455](matrix-org#10455))
- Fix a bug which caused an explicit assignment of power-level 0 to a user to be misinterpreted in rare circumstances. ([\matrix-org#10499](matrix-org#10499))

Improved Documentation
----------------------

- Fix hierarchy of providers on the OpenID page. ([\matrix-org#10445](matrix-org#10445))
- Consolidate development documentation to `docs/development/`. ([\matrix-org#10453](matrix-org#10453))
- Add some developer docs to explain room DAG concepts like `outliers`, `state_groups`, `depth`, etc. ([\matrix-org#10464](matrix-org#10464))
- Document how to use Complement while developing a new Synapse feature. ([\matrix-org#10483](matrix-org#10483))

Internal Changes
----------------

- Prune inbound federation queues for a room if they get too large. ([\matrix-org#10390](matrix-org#10390))
- Add type hints to `synapse.federation.transport.client` module. ([\matrix-org#10408](matrix-org#10408))
- Remove shebang line from module files. ([\matrix-org#10415](matrix-org#10415))
- Drop backwards-compatibility code that was required to support Ubuntu Xenial. ([\matrix-org#10429](matrix-org#10429))
- Use a docker image cache for the prerequisites for the debian package build. ([\matrix-org#10431](matrix-org#10431))
- Improve servlet type hints. ([\matrix-org#10437](matrix-org#10437), [\matrix-org#10438](matrix-org#10438))
- Replace usage of `or_ignore` in `simple_insert` with `simple_upsert` usage, to stop spamming postgres logs with spurious ERROR messages. ([\matrix-org#10442](matrix-org#10442))
- Update the `tests-done` Github Actions status. ([\matrix-org#10444](matrix-org#10444), [\matrix-org#10512](matrix-org#10512))
- Update type annotations to work with forthcoming Twisted 21.7.0 release. ([\matrix-org#10446](matrix-org#10446), [\matrix-org#10450](matrix-org#10450))
- Cancel redundant GHA workflows when a new commit is pushed. ([\matrix-org#10451](matrix-org#10451))
- Mitigate media repo XSS attacks on IE11 via the non-standard X-Content-Security-Policy header. ([\matrix-org#10468](matrix-org#10468))
- Additional type hints in the state handler. ([\matrix-org#10482](matrix-org#10482))
- Update syntax used to run complement tests. ([\matrix-org#10488](matrix-org#10488))
- Fix up type annotations to work with Twisted 21.7. ([\matrix-org#10490](matrix-org#10490))
- Improve type annotations for `ObservableDeferred`. ([\matrix-org#10491](matrix-org#10491))
- Extend release script to also tag and create GitHub releases. ([\matrix-org#10496](matrix-org#10496))
- Fix a bug which caused production debian packages to be incorrectly marked as 'prerelease'. ([\matrix-org#10500](matrix-org#10500))
babolivier added a commit to matrix-org/synapse-dinsic that referenced this pull request Sep 1, 2021
Synapse 1.40.0 (2021-08-10)
===========================

No significant changes.

Synapse 1.40.0rc3 (2021-08-09)
==============================

Features
--------

- Support [MSC3289: room version 8](matrix-org/matrix-spec-proposals#3289). ([\#10449](matrix-org/synapse#10449))

Bugfixes
--------

- Mark the experimental room version from [MSC2716](matrix-org/matrix-spec-proposals#2716) as unstable. ([\#10449](matrix-org/synapse#10449))

Improved Documentation
----------------------

- Fix broken links in `upgrade.md`. Contributed by @dklimpel. ([\#10543](matrix-org/synapse#10543))

Synapse 1.40.0rc2 (2021-08-04)
==============================

Bugfixes
--------

- Fix the `PeriodicallyFlushingMemoryHandler` inhibiting application shutdown because of its background thread. ([\#10517](matrix-org/synapse#10517))
- Fix a bug introduced in Synapse v1.40.0rc1 that could cause Synapse to respond with an error when clients would update read receipts. ([\#10531](matrix-org/synapse#10531))

Internal Changes
----------------

- Fix release script to open the correct URL for the release. ([\#10516](matrix-org/synapse#10516))

Synapse 1.40.0rc1 (2021-08-03)
==============================

Features
--------

- Add support for [MSC2033](matrix-org/matrix-spec-proposals#2033): `device_id` on `/account/whoami`. ([\#9918](matrix-org/synapse#9918))
- Update support for [MSC2716 - Incrementally importing history into existing rooms](matrix-org/matrix-spec-proposals#2716). ([\#10245](matrix-org/synapse#10245), [\#10432](matrix-org/synapse#10432), [\#10463](matrix-org/synapse#10463))
- Update support for [MSC3083](matrix-org/matrix-spec-proposals#3083) to consider changes in the MSC around which servers can issue join events. ([\#10254](matrix-org/synapse#10254), [\#10447](matrix-org/synapse#10447), [\#10489](matrix-org/synapse#10489))
- Initial support for [MSC3244](matrix-org/matrix-spec-proposals#3244), Room version capabilities over the /capabilities API. ([\#10283](matrix-org/synapse#10283))
- Add a buffered logging handler which periodically flushes itself. ([\#10407](matrix-org/synapse#10407), [\#10515](matrix-org/synapse#10515))
- Add support for https connections to a proxy server. Contributed by @Bubu and @dklimpel. ([\#10411](matrix-org/synapse#10411))
- Support for [MSC2285 (hidden read receipts)](matrix-org/matrix-spec-proposals#2285). Contributed by @SimonBrandner. ([\#10413](matrix-org/synapse#10413))
- Email notifications now state whether an invitation is to a room or a space. ([\#10426](matrix-org/synapse#10426))
- Allow setting transaction limit for database connections. ([\#10440](matrix-org/synapse#10440), [\#10511](matrix-org/synapse#10511))
- Add `creation_ts` to "list users" admin API. ([\#10448](matrix-org/synapse#10448))

Bugfixes
--------

- Improve character set detection in URL previews by supporting underscores (in addition to hyphens). Contributed by @srividyut. ([\#10410](matrix-org/synapse#10410))
- Fix events being incorrectly rejected over federation if they reference auth events that the server needed to fetch. ([\#10439](matrix-org/synapse#10439))
- Fix `synapse_federation_server_oldest_inbound_pdu_in_staging` Prometheus metric to not report a max age of 51 years when the queue is empty. ([\#10455](matrix-org/synapse#10455))
- Fix a bug which caused an explicit assignment of power-level 0 to a user to be misinterpreted in rare circumstances. ([\#10499](matrix-org/synapse#10499))

Improved Documentation
----------------------

- Fix hierarchy of providers on the OpenID page. ([\#10445](matrix-org/synapse#10445))
- Consolidate development documentation to `docs/development/`. ([\#10453](matrix-org/synapse#10453))
- Add some developer docs to explain room DAG concepts like `outliers`, `state_groups`, `depth`, etc. ([\#10464](matrix-org/synapse#10464))
- Document how to use Complement while developing a new Synapse feature. ([\#10483](matrix-org/synapse#10483))

Internal Changes
----------------

- Prune inbound federation queues for a room if they get too large. ([\#10390](matrix-org/synapse#10390))
- Add type hints to `synapse.federation.transport.client` module. ([\#10408](matrix-org/synapse#10408))
- Remove shebang line from module files. ([\#10415](matrix-org/synapse#10415))
- Drop backwards-compatibility code that was required to support Ubuntu Xenial. ([\#10429](matrix-org/synapse#10429))
- Use a docker image cache for the prerequisites for the debian package build. ([\#10431](matrix-org/synapse#10431))
- Improve servlet type hints. ([\#10437](matrix-org/synapse#10437), [\#10438](matrix-org/synapse#10438))
- Replace usage of `or_ignore` in `simple_insert` with `simple_upsert` usage, to stop spamming postgres logs with spurious ERROR messages. ([\#10442](matrix-org/synapse#10442))
- Update the `tests-done` Github Actions status. ([\#10444](matrix-org/synapse#10444), [\#10512](matrix-org/synapse#10512))
- Update type annotations to work with forthcoming Twisted 21.7.0 release. ([\#10446](matrix-org/synapse#10446), [\#10450](matrix-org/synapse#10450))
- Cancel redundant GHA workflows when a new commit is pushed. ([\#10451](matrix-org/synapse#10451))
- Mitigate media repo XSS attacks on IE11 via the non-standard X-Content-Security-Policy header. ([\#10468](matrix-org/synapse#10468))
- Additional type hints in the state handler. ([\#10482](matrix-org/synapse#10482))
- Update syntax used to run complement tests. ([\#10488](matrix-org/synapse#10488))
- Fix up type annotations to work with Twisted 21.7. ([\#10490](matrix-org/synapse#10490))
- Improve type annotations for `ObservableDeferred`. ([\#10491](matrix-org/synapse#10491))
- Extend release script to also tag and create GitHub releases. ([\#10496](matrix-org/synapse#10496))
- Fix a bug which caused production debian packages to be incorrectly marked as 'prerelease'. ([\#10500](matrix-org/synapse#10500))
Fizzadar pushed a commit to Fizzadar/synapse that referenced this pull request Oct 26, 2021
Synapse 1.40.0 (2021-08-10)
===========================

No significant changes.

Synapse 1.40.0rc3 (2021-08-09)
==============================

Features
--------

- Support [MSC3289: room version 8](matrix-org/matrix-spec-proposals#3289). ([\matrix-org#10449](matrix-org#10449))

Bugfixes
--------

- Mark the experimental room version from [MSC2716](matrix-org/matrix-spec-proposals#2716) as unstable. ([\matrix-org#10449](matrix-org#10449))

Improved Documentation
----------------------

- Fix broken links in `upgrade.md`. Contributed by @dklimpel. ([\matrix-org#10543](matrix-org#10543))

Synapse 1.40.0rc2 (2021-08-04)
==============================

Bugfixes
--------

- Fix the `PeriodicallyFlushingMemoryHandler` inhibiting application shutdown because of its background thread. ([\matrix-org#10517](matrix-org#10517))
- Fix a bug introduced in Synapse v1.40.0rc1 that could cause Synapse to respond with an error when clients would update read receipts. ([\matrix-org#10531](matrix-org#10531))

Internal Changes
----------------

- Fix release script to open the correct URL for the release. ([\matrix-org#10516](matrix-org#10516))

Synapse 1.40.0rc1 (2021-08-03)
==============================

Features
--------

- Add support for [MSC2033](matrix-org/matrix-spec-proposals#2033): `device_id` on `/account/whoami`. ([\matrix-org#9918](matrix-org#9918))
- Update support for [MSC2716 - Incrementally importing history into existing rooms](matrix-org/matrix-spec-proposals#2716). ([\matrix-org#10245](matrix-org#10245), [\matrix-org#10432](matrix-org#10432), [\matrix-org#10463](matrix-org#10463))
- Update support for [MSC3083](matrix-org/matrix-spec-proposals#3083) to consider changes in the MSC around which servers can issue join events. ([\matrix-org#10254](matrix-org#10254), [\matrix-org#10447](matrix-org#10447), [\matrix-org#10489](matrix-org#10489))
- Initial support for [MSC3244](matrix-org/matrix-spec-proposals#3244), Room version capabilities over the /capabilities API. ([\matrix-org#10283](matrix-org#10283))
- Add a buffered logging handler which periodically flushes itself. ([\matrix-org#10407](matrix-org#10407), [\matrix-org#10515](matrix-org#10515))
- Add support for https connections to a proxy server. Contributed by @Bubu and @dklimpel. ([\matrix-org#10411](matrix-org#10411))
- Support for [MSC2285 (hidden read receipts)](matrix-org/matrix-spec-proposals#2285). Contributed by @SimonBrandner. ([\matrix-org#10413](matrix-org#10413))
- Email notifications now state whether an invitation is to a room or a space. ([\matrix-org#10426](matrix-org#10426))
- Allow setting transaction limit for database connections. ([\matrix-org#10440](matrix-org#10440), [\matrix-org#10511](matrix-org#10511))
- Add `creation_ts` to "list users" admin API. ([\matrix-org#10448](matrix-org#10448))

Bugfixes
--------

- Improve character set detection in URL previews by supporting underscores (in addition to hyphens). Contributed by @srividyut. ([\matrix-org#10410](matrix-org#10410))
- Fix events being incorrectly rejected over federation if they reference auth events that the server needed to fetch. ([\matrix-org#10439](matrix-org#10439))
- Fix `synapse_federation_server_oldest_inbound_pdu_in_staging` Prometheus metric to not report a max age of 51 years when the queue is empty. ([\matrix-org#10455](matrix-org#10455))
- Fix a bug which caused an explicit assignment of power-level 0 to a user to be misinterpreted in rare circumstances. ([\matrix-org#10499](matrix-org#10499))

Improved Documentation
----------------------

- Fix hierarchy of providers on the OpenID page. ([\matrix-org#10445](matrix-org#10445))
- Consolidate development documentation to `docs/development/`. ([\matrix-org#10453](matrix-org#10453))
- Add some developer docs to explain room DAG concepts like `outliers`, `state_groups`, `depth`, etc. ([\matrix-org#10464](matrix-org#10464))
- Document how to use Complement while developing a new Synapse feature. ([\matrix-org#10483](matrix-org#10483))

Internal Changes
----------------

- Prune inbound federation queues for a room if they get too large. ([\matrix-org#10390](matrix-org#10390))
- Add type hints to `synapse.federation.transport.client` module. ([\matrix-org#10408](matrix-org#10408))
- Remove shebang line from module files. ([\matrix-org#10415](matrix-org#10415))
- Drop backwards-compatibility code that was required to support Ubuntu Xenial. ([\matrix-org#10429](matrix-org#10429))
- Use a docker image cache for the prerequisites for the debian package build. ([\matrix-org#10431](matrix-org#10431))
- Improve servlet type hints. ([\matrix-org#10437](matrix-org#10437), [\matrix-org#10438](matrix-org#10438))
- Replace usage of `or_ignore` in `simple_insert` with `simple_upsert` usage, to stop spamming postgres logs with spurious ERROR messages. ([\matrix-org#10442](matrix-org#10442))
- Update the `tests-done` Github Actions status. ([\matrix-org#10444](matrix-org#10444), [\matrix-org#10512](matrix-org#10512))
- Update type annotations to work with forthcoming Twisted 21.7.0 release. ([\matrix-org#10446](matrix-org#10446), [\matrix-org#10450](matrix-org#10450))
- Cancel redundant GHA workflows when a new commit is pushed. ([\matrix-org#10451](matrix-org#10451))
- Mitigate media repo XSS attacks on IE11 via the non-standard X-Content-Security-Policy header. ([\matrix-org#10468](matrix-org#10468))
- Additional type hints in the state handler. ([\matrix-org#10482](matrix-org#10482))
- Update syntax used to run complement tests. ([\matrix-org#10488](matrix-org#10488))
- Fix up type annotations to work with Twisted 21.7. ([\matrix-org#10490](matrix-org#10490))
- Improve type annotations for `ObservableDeferred`. ([\matrix-org#10491](matrix-org#10491))
- Extend release script to also tag and create GitHub releases. ([\matrix-org#10496](matrix-org#10496))
- Fix a bug which caused production debian packages to be incorrectly marked as 'prerelease'. ([\matrix-org#10500](matrix-org#10500))
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Support for https traffic to the proxy
3 participants