Community feedback, testing and improvements for Apache2 config docs

[jasonblais]

From https://forum.mattermost.org/t/websocket-apache-2-2-forwards-to-ws-127-0-0-1-8065-api-v3-users-websocket/3437

Summary

I updated to the latest mattermost (3.10.0). I want to use my Apache 2 (2.22.2) with my own SSL certificates. I use this Apache2 with SSL configuration from your documentation without the "Require all granted" lines, because they are for Apache 2.4.

Because I can't update to Apache 2.4, I had to setup the proxy_wstunnel manually.

All requests in Mattermost are answered properly, but 'https://mm.analyse-konzepte.de/api/v3/users/websocket' gets a 301 to the location: "/ws:/127.0.0.1:8065/api/v3/users/websocket". When I enter the URL 'mm.analyse-konzepte.de/api/v3/users/websocket' in my browser, it takes me to: mm.analyse-konzepte.de/ws:/127.0.0.1:8065/api/v3/users/websocket and obviously i get an error.

Expected behavior

I would expect, that https://mm.analyse-konzepte.de/api/v3/users/websocket forwards internally to 127.0.0.1:8065.

Observed behavior

[jasonblais]

@MikeDaniel18 @funkyfuture

A report from a community member after trying to set up Apahce2 proxy with SSL. Any thoughts or anything you might be able to help with?

[funkyfuture]

sorry, no clue.

[comradedakota]

I, too, am having the same issue. Any news on this?

[prixone]

@jasonblais could you ask for his apache configs and if he have any .htaccess taking place for those as well? thx

It could be potentially be some rule redirecting things wrongly.

[jasonblais]

@drcurry2010 Would you be able to provide the information mentioned above?

could you ask for his apache configs and if he have any .htaccess taking place for those as well? thx

[corinroyal]

I'm getting the dreaded "Please check connection, Mattermost unreachable. If issue persists, ask administrator to check WebSocket port." error after following the unofficial Apache2 docs, including certbot SSL with a redirect to port 443. This is on Apache 2.4 and Ubuntu 16.04 LTS.

These docs don't really say what packages and Apache modules we need to install/enable. I'm wondering if that's the problem. For Apache, I enabled proxy, proxy_http, proxy_wstunnel, and rewrite_module. Should there be others?

[MikeDaniel18]

@corinroyal hey I contributed to those docs and I agree they're in dire need of which packages need to be installed. My problem is, I did it so long ago now that I'll struggle to remember all of them. I'll have a think and see if I can't figure out what you're missing. The instructions should have worked fine for you.

@jasonblais Apologies for missing this, it seems I don't get notified by email when I'm mentioned in a comment. This looks odd. I don't think it's a .htaccess issue so I'm thinking it's arising from a difference in the way Apache 2.4 and Apache 2.2 handle the rewrite rules (or other parts) in the .conf. I don't think I'd be qualified to go much further than that - but I'll have a little think and a tinker. Edit: Or, there's just a small mistake in the person's config. Might be worth seeing what his is.

[corinroyal]

Thanks @MikeDaniel18 and @jasonblais for your great work on the documentation. Here's are pastes of the apache virtual host configurations for the websockets problem:

/etc/apache2/sites-available/talk.museapps.online.conf: http://termbin.com/w77v
/etc/apache2/sites-available/talk.museapps.online.conf-le-ssl.conf: http://termbin.com/21m3

I hope it's just a simple mistake on my part. I do have another domain hosted on this VPS. Holler if you want to see that config. Everything works great except for the web sockets, which is where I tend to trip up with Mattermost.

[jasonblais]

Thanks @corinroyal and @MikeDaniel18!

Wondering @MikeDaniel18 if you'd have any thoughts on the latest note above?

[MikeDaniel18]

@corinroyal & @jasonblais Yea this is a little tricky, it's very different from the original issue (which I'm relatively convinced is due to a misconfiguration or a typo somewhere - it's appending the URL rather than redirecting it). However, with this issue, I think its because you seem to have copied the non-ssl config to the ssl config when there are subtle differences. For instance, in the SSL config, try changing this:
RewriteRule .* wss://127.0.0.1:8065%{REQUEST_URI} [P,QSA,L]
to
RewriteRule .* ws://127.0.0.1:8065%{REQUEST_URI} [P,QSA,L].

You also seem to have missed these two lines:

RewriteCond %{DOCUMENT_ROOT}/%{REQUEST_FILENAME} !-f
RewriteRule .* http://127.0.0.1:8065%{REQUEST_URI} [P,QSA,L]

Let me know if this helps.
Mike

[MikeDaniel18]

Also, @jasonblais thanks for the reminder to respond again. I know sometimes it's hard to ask random people on the internet to follow up on things, but I promise my tardiness isn't because of an unwillingness to help! Keep pestering me until I respond if needed!!

[jasonblais]

@MikeDaniel18 Thank you! We really appreciate your help here. Many people have used your docs to configure Apache2 proxy so you've made quite an impact here. :)

[MikeDaniel18]

@jasonblais My pleasure. Let's see if it fixes it first :)

[corinroyal]

Thank you @MikeDaniel18 and @jasonblais. Unfortunately I'm no longer able to test as I just finished ripping out Apache for Nginx, which fortunately fixed the issue. I suspect your fix would work. Let's hope the next person with the same problem gives it a try.

[thawn]

I ran into the same problem as @jasonblais.
I was able to fix it by installing and enabling the following apache modules mod_rewrite , mod_proxy, mod_proxy_http and mod_proxy_wstunnel. (I am on gentoo so I did this by adding proxy proxy_http proxy_wstunnel rewrite to the APACHE2_MODULES variable in make.conf).

I also added the following lines in my apache configuration (under the <VirtualHost *:443> section):

# Begin mattermost configuration
ProxyPreserveHost On
RewriteEngine On
RewriteCond %{REQUEST_URI} /api/v[0-9]+/(users/)?websocket [NC,OR]
RewriteCond %{HTTP:UPGRADE} ^WebSocket$ [NC,OR]
RewriteCond %{HTTP:CONNECTION} ^Upgrade$ [NC]
RewriteRule .* ws://127.0.0.1:8065%{REQUEST_URI} [P,QSA,L]

<Location />
        Require all granted
        ProxyPass http://127.0.0.1:8065/
        ProxyPassReverse http://127.0.0.1:8065/
        ProxyPassReverseCookieDomain 127.0.0.1 mysubdomain.mydomain.com
</Location>
# End mattermost configuration

I did extensive testing and all of the lines above were essential to get the websockets to work.
However, I found the following lines to be obsolete (In fact the obsolete Rewrite lines were harmful to other services running on the same server):

# setup the proxy 
# I don't think this can work with apache 2.4 
# because the 'Order allow,deny' and 'Allow from all' syntax has been deprecated
<Proxy *>
        Order allow,deny
        Allow from all
</Proxy>
RewriteCond %{DOCUMENT_ROOT}/%{REQUEST_FILENAME} !-f
RewriteRule .* http://127.0.0.1:8065%{REQUEST_URI} [P,QSA,L]
<LocationMatch "^/api/v(?<apiversion>[0-9]+)/(?<apiusers>users/)?websocket">
        Require all granted
        ProxyPass ws://127.0.0.1:8065/api/v%{env:MATCH_APIVERSION}/%{env:MATCH_APIUSERS}websocket
        ProxyPassReverse ws://127.0.0.1:8065/api/v%{env:MATCH_APIVERSION}/%{env:MATCH_APIUSERS}websocket
        ProxyPassReverseCookieDomain 127.0.0.1 mysubdomain.mydomain.com
</LocationMatch>

@MikeDaniel18 Should I update the documentation accordingly (i.e. add the documentation of which modules are required and remove the obsolete configuration lines)?

@MikeDaniel18 I assume the same applies to the non-ssl configuration but I cannot test this in my setup. Should I update the non-ssl documentation anyways?

[thawn]

Let me know If I should also make the (untested but very likely to work) changes to the non-ssl configuration.

[MikeDaniel18]

@thawn Great stuff. We've been meaning to add the modules to the documentation. None of those modules would negatively impact the non-ssl configuration.

Thanks for removing the obsolete code, it was a relic from the very first test a user did.

[thawn]

@MikeDaniel18 glad I could help :) It took me the better part of a day to figure out what was wrong with my setup until I realized that I was simply missing the mod_proxy_wstunnel module - d'oh.

I now also changed the non-ssl configuration accordingly (pull request #2261). If possible, it would be great if someone with a non-ssl apache setup could test the apache configuration although I don't see a reason why it should not work.

[jasonblais]

Thank you @thawn!! This is great stuff. @amyblais has helped queue your PRs for review. Appreciate your help contributing the changes back to the community :)

[rothirschtec]

If you want to open mattermost inside an iframe you have to add following inside your apache vHost configuration.

Header append Content-Security-Policy "frame-ancestors 'self' https://*.your.domain.local"
Header unset Content-Security-Policy
Header append X-Frame-Options "ALLOW-FROM 'https://*.your.domain.local'"
Header unset X-Frame-Options

An example: https://documentation.kopano.io/kopano_mattermost_manual/first_steps.html#configuration-for-kopano-webapp-deskapp

[gracion]

I successfully set up mattermost alongside my existing apache2 cloud server thanks to your instructions! I'd suggest two additional notes.

(1) If proxying a new port (e.g., 8443), you may need to add "Listen 8443" on a separate line before the <VirtualHost *:8443> entry in the .conf file.

(2) In the mattermost config.json file, Mattermost docs suggest this: "ListenAddress": ":8065" - which works but leaves mattermost listening to the world directly on that port. To restrict this to the apache proxy, use: "ListenAddress": "localhost:8065". Of course, if your mattermost service is on a different server, that won't work. I'm not sure if you can use the IP address of your apache server in that case, but there's always iptables if you need to restrict access to a port selectively.

[jasonblais]

@gracion Thank you for the additional notes! Really appreciate it. Would you be open to help with a pull request to add them to the guide?

cc @justinegeffen

[gracion]

I'd love to! Good chance to experience my first github pull request (I still live happily in the subversion world). Also my first experience with reStructuredText. Trying to setup for this, but my BBEdit rst previewer script gives an error because Pygments cannot be found because openssl@1.1 is not working with python2.7 on Catalina, etc. Atom editor has an rst package, but I can't see a way to preview with it. Do you have a suggested way to preview .rst? Cheers,

…

--Paul

On Sep 7, 2020, at 7:18 AM, Jason Blais ***@***.***> wrote: @gracion <https://github.com/gracion> Thank you for the additional notes! Really appreciate it. Would you be open to help with a pull request to add them to the guide? cc @justinegeffen <https://github.com/justinegeffen> — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1295 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAHFWAOZMZN5D2CKLKZDTWLSETTTFANCNFSM4DQ7TMVA>.

[gracion]

Ok, pull request made! I used a javascript previewer at http://rst.ninjs.org <http://rst.ninjs.org/> - Should be no problems, but criticism is welcome. Thanks,

…

--Paul

On Sep 7, 2020, at 4:32 PM, Paul Collins ***@***.***> wrote: I'd love to! Good chance to experience my first github pull request (I still live happily in the subversion world). Also my first experience with reStructuredText. Trying to setup for this, but my BBEdit rst previewer script gives an error because Pygments cannot be found because ***@***.*** is not working with python2.7 on Catalina, etc. Atom editor has an rst package, but I can't see a way to preview with it. Do you have a suggested way to preview .rst? Cheers, --Paul > On Sep 7, 2020, at 7:18 AM, Jason Blais ***@***.*** ***@***.***>> wrote: > > > @gracion <https://github.com/gracion> Thank you for the additional notes! Really appreciate it. Would you be open to help with a pull request to add them to the guide? > > cc @justinegeffen <https://github.com/justinegeffen> > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub <#1295 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAHFWAOZMZN5D2CKLKZDTWLSETTTFANCNFSM4DQ7TMVA>. >

[justinegeffen]

Thank you, @gracion!

In terms of previewing the content, there is a preview server that spins up with every pull request but I see the deployment failed. I made some changes to your PR and committed them as that usually resolves the problem. In this case it didn't so I will have a chat with the DevOps team. It may be related to the PR being from a fork.

In terms of previewing the content locally, could you confirm how you've configured your local environment? In the interim, one way to take a look at the preview content is to display the rich diff in the PR. I've attached a screenshot of what the button looks like (apologies for my shaky circle!). Let me know if that helps. :)

[jasonblais]

Thank you @gracion! 🎉

[gracion]

Justine, I found the rich diff button in the PR, nice! Plus, I realized the file view shows the rendered document with changes applied. Thanks for asking. My local environment is adequate for docs updates. I'm configured for macOS and iOS application development, which is my actual job. Later, if I rise to the level of executable code PRs, I'll take another look at my environment, but no need to change anything at this point. Best regards from smoky Oregon, USA

…

--Paul Collins

On Sep 8, 2020, at 12:01 AM, Justine Geffen ***@***.***> wrote: Thank you, @gracion <https://github.com/gracion>! In terms of previewing the content, there is a preview server that spins up with every pull request but I see the deployment failed. I made some changes to your PR and committed them as that usually resolves the problem. In this case it didn't so I will have a chat with the DevOps team. It may be related to the PR being from a fork. In terms of previewing the content locally, could you confirm how you've configured your local environment? In the interim, one way to take a look at the preview content is to display the rich diff in the PR. I've attached a screenshot of what the button looks like (apologies for my shaky circle!). Let me know if that helps. :) <https://user-images.githubusercontent.com/4223530/92443490-d566a200-f1b1-11ea-8fdb-a4c1d334f864.png> — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1295 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAHFWAJINOGXSHLKHKNNJGTSEXJE7ANCNFSM4DQ7TMVA>.

[justinegeffen]

Oh I'm glad that worked! I mainly wanted to find out your local doc environment in case you were having an issue with make html. :D But definitely no need to change anything - especially if the rich diff works. :) Thanks again for the PR!

[tkyoon]

@thawn
I've been struggling with this problem all day.
I updated httpd to 2.4 and did everything possible, but it didn't work.
I solved it this way, thank you sooooooooooooooo~ much.

[justinegeffen]

@thawn I've been struggling with this problem all day. I updated httpd to 2.4 and did everything possible, but it didn't work. I solved it this way, thank you sooooooooooooooo~ much.

Thanks for letting us know, @tkyoon! I have reopened this issue so that we can ensure the docs are updated appropriately. Would you be open to updating the docs with this content?

[tkyoon]

@justinegeffen I just followed as https://docs.mattermost.com/configure/config-proxy-apache2.html.
thanks 😊

[rikrdo89]

I'm trying to install MM using docker-compose behind an apache proxy, but I dont find the config.json that I should change the "ListenAddress": "localhost:8065" as suggested above. Does that not apply to docker installation?

[rikrdo89]

Also, I am following the same instructions for the apache reverse proxy and created a .conf for port 80, but is there anything I need to change in the ssl.conf after I run certbot to force https connection (port 443)? I am not sure when I need to make the "ws" to "wss" change.
but I am a bit lost about when to change the "ws" to "wss".