Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Interaction with HTTP2 (mod_http2) #6

Closed
szbalint opened this issue Dec 1, 2016 · 16 comments · Fixed by #10
Closed

Interaction with HTTP2 (mod_http2) #6

szbalint opened this issue Dec 1, 2016 · 16 comments · Fixed by #10

Comments

@szbalint
Copy link

szbalint commented Dec 1, 2016

When attempting to use mod_proxy_protocol in combination with HTTP2 (mod_http2), that particular configuration fails with:

  1. The ip addresses not being set correctly
  2. Depending whether mod_http2's H2SerializeHeaders is set to on or not:
    a. On every request of a HTTP2 connection: [proxy_protocol:error] [pid 8565] [client 127.0.0.1:38181] ProxyProtocol: no valid header found
    b. On every subsequent request of a HTTP2 connection that doesn't match the original request type the error above. (If the original request type was GET then POSTs fail with the error etc)

HTTP2 is probably a pretty big change to what the different Apache layers do in terms of separating a connection from requests and responses and I'm not intimately familiar with how this is implemented therefor I'm opening an issue both here and over at mod_http2 about this incompatibility.
@szbalint szbalint mentioned this issue Dec 1, 2016
Closed
@icing
Copy link

icing commented Dec 1, 2016 •
edited

Cheers, as far as I can tell on a first glance at the code:

This seems to be an issue of correct input filter ordering. The proxy_protocol input filter must read before the h2 input filter.

And subsequently, for request patching, this will happen on a slave connection with h2, so the module needs to check conn_rec->master and retrieve its settings there, if available.

@mattkolb
Copy link

mattkolb commented Jan 3, 2017

To clarify, are the following statements correct?

  • The issue is in mod_proxy_protocol
  • The issue cannot be resolved by a user changing something in the Apache config
  • mod_proxy_protocol does not currently work with mod_http2

@roadrunner2
Copy link
Owner

To clarify, are the following statements correct?

  • The issue is in mod_proxy_protocol

At this point it's not clear to me where the issue is; icing's comment about this being an ordering issue sounds reasonable based on the reported behaviour, but a quick glance at mod_http2 indicates that it is registering its input-filter after the mod_proxy_protocol input filter (in fact, mod_proxy_protocol registers before mod_ssl, and mod_http2 after mod_ssl). So I'm not sure where the issue is.

  • The issue cannot be resolved by a user changing something in the Apache config

Correct.

  • mod_proxy_protocol does not currently work with mod_http2

Apparently.

@mattkolb
Copy link

mattkolb commented Jan 4, 2017

Thank you very much for your prompt response. It was extremely helpful. Keep up the good work. I'll monitor this issue to see if/when this gets resolved.

@smelchior
Copy link

Is there any progress on this issue yet? I ran into it recently and it halted my http2 rollout :-|

@roadrunner2
Copy link
Owner

@smelchior I have not had time to try and figure this out yet.

@linksilver
Copy link

experiencing the same issue

@pgodschalk
Copy link
Contributor

Experienced this as well.

OS and package details:

web001 :: ~ » uname -a
Linux web001 4.9.0-6-amd64 #1 SMP Debian 4.9.82-1+deb9u3 (2018-03-02) x86_64 GNU/Linux
web001 :: ~ » apt-cache policy apache2 | grep Installed
  Installed: 2.4.25-3+deb9u3
web001 :: ~ » apt-cache policy haproxy | grep Installed
  Installed: 1.7.5-2

haproxy.cfg:

(...)
frontend http
  bind 10.121.1.151:80
  default_backend nodes-http

frontend https
  bind 10.121.1.151:443
  default_backend nodes-https

(...)
                                                                  
backend nodes-http
  balance roundrobin
  mode tcp
  option tcp-check
      server web001 10.121.1.150:80 send-proxy-v2 maxconn 1000 check fall 3 rise 2 
      server web002 10.121.1.152:80 send-proxy-v2 maxconn 1000 check fall 3 rise 2 
      server web003 10.121.1.157:80 send-proxy-v2 maxconn 1000 check fall 3 rise 2  backup 
  
backend nodes-https
  balance roundrobin
  mode tcp
  option tcp-check
      server web001 10.121.1.150:443 send-proxy-v2 maxconn 1000 check fall 3 rise 2 
      server web002 10.121.1.152:443 send-proxy-v2 maxconn 1000 check fall 3 rise 2 
      server web003 10.121.1.157:443 send-proxy-v2 maxconn 1000 check fall 3 rise 2  backup 
(...)

ProxyProtocol is enabled globally (an exception is made for an internal virtualhost running on a non-default port)

web001 :: /etc/apache2/mods-available » cat proxy_protocol.*
ProxyProtocol on
LoadModule proxy_protocol_module /usr/lib/apache2/modules/mod_proxy_protocol.so

[Sun Mar 18 12:47:30 2018] [notice] [pid 6357] mod_proxy_protocol.c(237): [client ProxyProtocol: enabled on 0.0.0.0:0

This works as expected for virtual hosts not running HTTP/2:
www.mozard.nl:443 10.121.1.152 213.126.108.162 59066 - - [18/Mar/2018:12:42:31 +0100] "GET /mozard/document/pub/6041,7684,, HTTP/1.1" 200 5846 "https://www.mozard.nl/mozard/!suite05.scherm1081?mNch=gu35yrimwx" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.13; rv:59.0) Gecko/20100101 Firefox/59.0"

The client IP and port is correctly logged as the fourth item as defined by the global LogFormat directive:

LogFormat "%v:%p %h %a %{remote}p %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" vhost_combined

Enabling HTTP/2 on a TLS-enabled virtual host. Other than enabling HTTP/2, this is identical to production, save the obvious differences in certificate, ServerName directive, etc.

<VirtualHost *:443>
  Protocols h2 http/1.1

  ServerName [redacted].mozard.nl

  ServerAlias [redacted].mozard.nl
  ServerAlias [redacted].mozard.nl
      
  ServerAdmin [redacted]@mozard.nl

  DocumentRoot /var/www/mzrdo-inter

  LogLevel info ssl:warn
  ErrorLog ${APACHE_LOG_DIR}/error.log

  SSLEngine on
  SSLCertificateFile [redacted]
  SSLCertificateKeyFile [redacted]
  SSLCompression off

  Header always set Strict-Transport-Security \
    "max-age=63072000; includeSubDomains; preload"

  <Directory /var/www/mzrdo-inter>
    AllowOverride all
  </Directory>

  <FilesMatch "\.(cgi|shtml|phtml|php)$">
    SSLOptions +StdEnvVars
  </FilesMatch>

  <Directory /usr/lib/cgi-bin>
    SSLOptions +StdEnvVars
  </Directory>

  SetEnvIfNoCase ^Authorization$ "(.+)" HTTP_AUTHORIZATION=$1

  <FilesMatch ".+\.ph(p[3457]?|t|tml)$">
    SetHandler "proxy:unix:/run/php/php7.0-fpm-mzrdo.sock|fcgi://localhost"
  </FilesMatch>

  CacheEnable disk /mozard/index.php/document/pub/
  CacheRoot /var/cache/apache2/mod_cache_disk/mzrdo-inter
  CacheDefaultExpire 3600
  CacheDirLevels 2
  CacheDirLength 2
  CacheStorePrivate on
  CacheIgnoreHeaders Set-Cookie
  CacheIgnoreHeaders Cookie
  CacheIgnoreHeaders X-Remote-User
</VirtualHost>

Response headers confirm HTTP/2 is being used:

HTTP/2.0 200 OK
cache-control: max-age=0, no-transform
content-encoding: gzip
content-length: 5953
content-security-policy: default-src 'self' https: data: blob: 'unsafe-eval' 'unsafe-inline'; report-uri https://[redacted].report-uri.com/r/d/csp/reportOnly
content-type: text/html; charset=UTF-8
date: Sun, 18 Mar 2018 11:50:15 GMT
expect-ct: max-age=0, report-uri="https://[redacted].report-uri.com/r/d/ct/reportOnly"
expect-staple: max-age=0, report-uri="https://[redacted].report-uri.com/r/d/staple/reportOnly"
expires: Sun, 18 Mar 2018 11:50:15 GMT
p3p: policyref="/w3c/p3p.xml", CP="IDC DSP COR ADM DEVi TAIi PSA PSD IVAi IVDi CONi HIS OUR IND CNT"
referrer-policy: no-referrer-when-downgrade
server: Apache
strict-transport-security: max-age=63072000; includeSubDomains; preload
timing-allow-origin: *
x-content-type-options: nosniff
X-Firefox-Spdy: h2
x-frame-options: sameorigin
x-moz-username: -
x-moz-webserver: web002
x-ua-compatible: IE=edge
x-xss-protection: 1; mode=block

Access log shows the IP and client port of the load balancer, rather than the initiator. Also reconfirms that HTTP/2 is being used.

[redacted].mozard.nl:443 10.121.1.152 10.121.1.152 56734 - - [18/Mar/2018:12:47:55 +0100] "GET /mozard/!suite05.scherm1081?mNch=ubdg5xbtn3 HTTP/2.0" 200 5949 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.13; rv:59.0) Gecko/20100101 Firefox/59.0"

This GET request is succesful.

Sending a POST is not:

[redacted].mozard.nl:443 10.121.1.152 10.121.1.152 34182 - - [18/Mar/2018:12:55:37 +0100] "POST /mozard/!suite09.scherm1089Mut HTTP/2.0" 200 0 "https://[redacted].mozard.nl/mozard/!suite09.scherm1089?mWfrs=19990&mNch=1ho8aem19s"

"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.13; rv:59.0) Gecko/20100101 Firefox/59.0" [Sun Mar 18 12:55:37 2018] [error] [pid 6384] mod_proxy_protocol.c(587): [client 10.121.1.152:34182] ProxyProtocol: no valid header found

@icing
Copy link

icing commented Mar 19, 2018 •
edited

Me confused here. Is this the module that has become mod_remoteip in the Apache httpd releases? If yes, does the same issue apply for that one?

I have no setup where I can verify this module.

@pgodschalk
Copy link
Contributor

This module looks to be merged into mod_remoteip in revision 1824221 with further changes in revision 1824248.

Compiling httpd from 2.4.x trunk confirms the issue exists there as well, at least, the %a variable is not correctly set to the initiator when enabling HTTP/2. While it's reasonable to assume the issue when changing request type is also present in trunk, I've not had the time to test this yet. When that's done, this issue should probably be crossposted to mod_remoteip's issue tracker as well either way.

root@haproxy-test:/usr/local/apache2/bin# uname -a
Linux haproxy-test 4.9.0-6-amd64 #1 SMP Debian 4.9.82-1+deb9u3 (2018-03-02) x86_64 GNU/Linux
root@haproxy-test:/usr/local/apache2/bin# ./apachectl -v
Server version: Apache/2.4.34-dev (Unix)
Server built:   Mar 19 2018 10:53:14

HAProxy configured similarly to production as above:

(...)
frontend https
  bind 167.99.44.48:443
  mode tcp
  option tcplog
  default_backend nodes-https

backend nodes-https
  balance roundrobin
  mode tcp
  option tcp-check
  option forwardfor
  server localhost 10.18.0.8:4433 send-proxy-v2
(...)

HTTPD configured with project defaults, ProxyProtocol enabled on 4433 and HTTP/2 enabled on the default SSL virtual host and the %a variable added to the LogFormat confirms that the initiator IP is correctly set when using HTTP/1.1, but not when using HTTP/2:

[19/Mar/2018:11:14:55 +0000] 10.18.0.8 213.126.108.162 TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384 "GET / HTTP/1.1" -
[19/Mar/2018:11:15:58 +0000] 10.18.0.8 10.18.0.8 TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384 "GET / HTTP/2.0" -

@icing
Copy link

icing commented Mar 19, 2018

Thanks, could you try the following patch in your 2.4.x build?

Index: modules/metadata/mod_remoteip.c
===================================================================
--- modules/metadata/mod_remoteip.c	(revision 1827187)
+++ modules/metadata/mod_remoteip.c	(working copy)
@@ -852,8 +852,13 @@
     remoteip_conn_config_t *conn_conf;
     int i;
 
-    /* Do not attempt to manipulate slave connections */
+    /* Establish master config in slave connections, so that request processing
+     * finds it. */
     if (c->master != NULL) {
+        conn_conf = ap_get_module_config(c->master->conn_config, &remoteip_module);
+        if (conn_conf) {
+            ap_set_module_config(c->conn_config, &remoteip_module, conn_conf);
+        }
         return DECLINED;
     }

Thanks!

@pgodschalk
Copy link
Contributor

pgodschalk commented Mar 19, 2018 •
edited

Compiling with the above patch works as expected:

[19/Mar/2018:12:49:03 +0000] 10.18.0.8 213.126.108.162 TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384 "GET / HTTP/2.0" -

Switching request type from GET to POST also appears to function correctly, over proxy_http at least.

[19/Mar/2018:12:49:03 +0000] 10.18.0.8 213.126.108.162 TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384 "GET / HTTP/2.0" -
[19/Mar/2018:12:49:03 +0000] 10.18.0.8 213.126.108.162 TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384 "GET /mozard/!scherm0927 HTTP/2.0" 7861
[19/Mar/2018:12:49:03 +0000] 10.18.0.8 213.126.108.162 TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384 "GET /mozard/!suite86.mozard_pagina_lib?n=184160.js HTTP/2.0" 149
[19/Mar/2018:12:49:03 +0000] 10.18.0.8 213.126.108.162 TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384 "GET /mozard/!suite07.logx?res=1895x1243 HTTP/2.0" 23
[19/Mar/2018:12:49:29 +0000] 10.18.0.8 213.126.108.162 TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384 "GET /mozard/!suite74.scherm0249 HTTP/2.0" 9566
[19/Mar/2018:12:49:29 +0000] 10.18.0.8 213.126.108.162 TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384 "GET /mozard/!suite86.mozard_pagina_lib?n=184161.js HTTP/2.0" 333
[19/Mar/2018:12:49:29 +0000] 10.18.0.8 213.126.108.162 TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384 "GET /mozard/document/pub/css_print_249-_opt_x_style.css HTTP/2.0" 30288
[19/Mar/2018:12:49:29 +0000] 10.18.0.8 213.126.108.162 TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384 "GET /mozard/document/pub/css_screen_249-_opt_xl_style.css HTTP/2.0" 53293
[19/Mar/2018:12:49:30 +0000] 10.18.0.8 213.126.108.162 TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384 "GET /mozard/!suite74.scherm0249 HTTP/2.0" 9572
[19/Mar/2018:12:49:31 +0000] 10.18.0.8 213.126.108.162 TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384 "GET /mozard/!suite86.mozard_pagina_lib?n=184162.js HTTP/2.0" 332
[19/Mar/2018:12:49:31 +0000] 10.18.0.8 213.126.108.162 TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384 "GET /mozard/!suite74.lovKiesFunctieverband?mId=mMpi&mKcc=&mInit=1&mCv=1840&mNch=KdemuLJ0ZN&_=1521463769958 HTTP/2.0" 1016
[19/Mar/2018:12:49:32 +0000] 10.18.0.8 213.126.108.162 TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384 "GET /mozard/!suite74.lovKiesFunctieverband?mId=mMpi&mKcc=&mInit=1&mCv=1840&mNch=LtXSrpqfwI&_=1521463771168 HTTP/2.0" 1016
[19/Mar/2018:12:49:32 +0000] 10.18.0.8 213.126.108.162 TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384 "POST /mozard/!suite74.scherm0248 HTTP/2.0" 21

Since mod_remoteip with PROXY support is not yet released as stable, I can't test this as easily in our staging environment, but I'll try to backport this to mod_proxy_protocol and test there later today.

@icing
Copy link

icing commented Mar 19, 2018 •
edited

Wonderful, thanks! I applied this to Apache subversion trunk as r1827196 and proposed for backport into 2.4.x.

asfgit pushed a commit to apache/httpd that referenced this issue Mar 19, 2018
@icing
mod_remoteip: make proxy-protocol work on slave connections, e.g. in …
f1d5f5d 
…HTTP/2

     requests. See also roadrunner2/mod-proxy-protocol#6


git-svn-id: https://svn.apache.org/repos/asf/httpd/httpd/trunk@1827196 13f79535-47bb-0310-9956-ffa450edef68
pgodschalk added a commit to pgodschalk/mod-proxy-protocol that referenced this issue Mar 19, 2018
@pgodschalk
Fix behaviour when using PROXY and HTTP/2. Backported from @icing at …
5474ae7 
…mod_remoteip f1d5f5d6b8577b2428082050cb03531d7614dba4 - fixes roadrunner2#6
@pgodschalk pgodschalk mentioned this issue Mar 19, 2018
Merged
@pgodschalk
Copy link
Contributor

pgodschalk commented Mar 19, 2018 •
edited

Verified working in our staging environment:

root@web003:/var/log/apache2# apachectl -v
Server version: Apache/2.4.25 (Debian)
Server built:   2017-09-19T18:58:57
root@web003:/var/log/apache2# apachectl -M
Loaded Modules:
 core_module (static)
 so_module (static)
 watchdog_module (static)
 http_module (static)
 log_config_module (static)
 logio_module (static)
 version_module (static)
 unixd_module (static)
 alias_module (shared)
 auth_basic_module (shared)
 authn_core_module (shared)
 authn_file_module (shared)
 authz_core_module (shared)
 authz_host_module (shared)
 authz_user_module (shared)
 cache_module (shared)
 cache_disk_module (shared)
 deflate_module (shared)
 dir_module (shared)
 env_module (shared)
 expires_module (shared)
 filter_module (shared)
 headers_module (shared)
 http2_module (shared)
 mime_module (shared)
 mpm_event_module (shared)
 negotiation_module (shared)
 proxy_module (shared)
 proxy_fcgi_module (shared)
 proxy_http_module (shared)
 proxy_protocol_module (shared)
 reqtimeout_module (shared)
 rewrite_module (shared)
 setenvif_module (shared)
 socache_memcache_module (shared)
 socache_shmcb_module (shared)
 ssl_module (shared)
 status_module (shared)

ontwikkel-intranet.mozard.nl:443 10.121.1.152 213.126.108.162 61815 - - [19/Mar/2018:21:25:23 +0100] "GET /mozard/!scherm0927 HTTP/2.0" 200 7848 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_3) AppleWebKit/604.5.6 (KHTML, like Gecko) Version/11.0.3 Safari/604.5.6"
ontwikkel-intranet.mozard.nl:443 10.121.1.152 213.126.108.162 61815 - - [19/Mar/2018:21:25:23 +0100] "GET /mozard/!suite07.logx?res=2220x1278 HTTP/2.0" 200 23 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_3) AppleWebKit/604.5.6 (KHTML, like Gecko) Version/11.0.3 Safari/604.5.6"
ontwikkel-intranet.mozard.nl:443 10.121.1.152 213.126.108.162 61815 - - [19/Mar/2018:21:25:23 +0100] "GET /mozard/!suite86.mozard_pagina_lib?n=184346.js HTTP/2.0" 200 149 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_3) AppleWebKit/604.5.6 (KHTML, like Gecko) Version/11.0.3 Safari/604.5.6"
ontwikkel-intranet.mozard.nl:443 10.121.1.152 213.126.108.162 61815 - - [19/Mar/2018:21:25:23 +0100] "GET /mozard/document/pub/intranet-print.css.map HTTP/2.0" 404 224 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_3) AppleWebKit/604.5.6 (KHTML, like Gecko) Version/11.0.3 Safari/604.5.6"
ontwikkel-intranet.mozard.nl:443 10.121.1.152 213.126.108.162 61815 - - [19/Mar/2018:21:25:24 +0100] "GET /mozard/!suite74.scherm0249 HTTP/2.0" 200 9751 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_3) AppleWebKit/604.5.6 (KHTML, like Gecko) Version/11.0.3 Safari/604.5.6"
ontwikkel-intranet.mozard.nl:443 10.121.1.152 213.126.108.162 61815 - - [19/Mar/2018:21:25:24 +0100] "GET /mozard/!suite86.mozard_pagina_lib?n=184347.js HTTP/2.0" 200 358 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_3) AppleWebKit/604.5.6 (KHTML, like Gecko) Version/11.0.3 Safari/604.5.6"
ontwikkel-intranet.mozard.nl:443 10.121.1.152 213.126.108.162 61815 - - [19/Mar/2018:21:25:24 +0100] "GET /mozard/document/pub/intranet-print.css.map HTTP/2.0" 404 224 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_3) AppleWebKit/604.5.6 (KHTML, like Gecko) Version/11.0.3 Safari/604.5.6"
ontwikkel-intranet.mozard.nl:443 10.121.1.152 213.126.108.162 61815 - - [19/Mar/2018:21:25:25 +0100] "GET /mozard/!suite74.lovKiesFunctieverband?mId=mMpi&mKcc=&mInit=1&mCv=1839&mNch=mfc5OxGOH7&_=1521491124880 HTTP/2.0" 200 1015 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_3) AppleWebKit/604.5.6 (KHTML, like Gecko) Version/11.0.3 Safari/604.5.6"
ontwikkel-intranet.mozard.nl:443 10.121.1.152 213.126.108.162 61815 - - [19/Mar/2018:21:25:26 +0100] "POST /mozard/!suite74.scherm0248 HTTP/2.0" 302 21 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_3) AppleWebKit/604.5.6 (KHTML, like Gecko) Version/11.0.3 Safari/604.5.6"

@icing
Copy link

icing commented Mar 20, 2018

Nice!

@roadrunner2
Copy link
Owner

roadrunner2 commented Mar 29, 2018 •
edited

@icing Thanks for debugging this and creating the fix! @pgodschalk thanks for backporting the fix to here.

asfgit pushed a commit to apache/httpd that referenced this issue Oct 5, 2019
@icing
mod_remoteip: make proxy-protocol work on slave connections, e.g. in …
a1b7278 
…HTTP/2

     requests. See also roadrunner2/mod-proxy-protocol#6


git-svn-id: https://svn.apache.org/repos/asf/httpd/httpd/trunk@1827196 13f79535-47bb-0310-9956-ffa450edef68
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix behaviour when using PROXY and HTTP/2 pgodschalk/mod-proxy-protocol
7 participants
@icing @szbalint @roadrunner2 @smelchior @pgodschalk @mattkolb @linksilver