Reimplementation of remoteip-rpaf for 2.4.48 and up (updated to trunk)

[AlexAT]

This is related to #68 and is the reimplementation of it to 2.4.48, removing all the hacky parts.
Please tell if anything else needs to be adjusted. If someone is using my older variant (from #68), please test in production to make sure I did not introduce any new bugs (I did the initial testing myself though).

The feature description is:
mod_remoteip is extended by multiple options handy for frontend proxies moving traffic to Apache backends

• RemoteAllowOnlyInternalProxies - only allows 'internal' trusted proxies to connect to the Apache server, so no untrusted requests can be done (such are refused with Forbidden)
• RemoteHostHeader - accepted only from 'internal' trusted proxies, designates name of HTTP header to use as the supplementary Host header used to re-select virtual host inside request processing - keeping the original supplied Host header (if any) normally visible to i.e. user scripts so they don't know internal virtual host ID is different
• RemotePortHeader - accepted only from 'internal' trusted proxies, designates name of HTTP header to use as the port number originally used to connect which is used as 'default port' in the request and its subrequests
• RemoteProtoHeader - accepted only from 'internal' trusted proxies, designates name of HTTP header to supply for HTTPS 'simulation' (only sets the HTTPS used flag internally in the request and its subrequests, so i.e. rewrite conditions work like HTTPS is enabled when the header is present, even if the access to Apache from the proxy is done via HTTP, PHP shows HTTPS enabled, and so on, request HTTP schema is also adjusted)
• RemoteHTTPSEnableProto - accepted only from 'internal' trusted proxies, supplements RemoteProtoHeader, if set, additionally compares RemoteProtoHeader header contents to the value set and only sets HTTPS emulation if header contents match

---

Changes to 2.4.41-46 version in #68:

• No more obscure hacks of SSL module and internal metadata for HTTPS simulation handling and port alteration, rely fully on existing Apache hooks (including new 2.4.48 ssl_conn_is_ssl() hook)
• Use apr_table_nset() for connection annotation and header changes, we strdup() anyways
• RemoteHTTPSEnableProto option omitted now means setting HTTPS mode simulation on any RemoteProtoHeader header presence, without verifying header contents, for compatibility with the previous default "RemoteHTTPSEnableProto https" needs to be used

[AlexAT]

Added 'static' declaration missing from the initial commit, build passes now.

[AlexAT]

I think this also needs some justification about how it's usable. This one was made with shared hosting clusters and frontend-to-backend splits in mind.

• RemoteAllowOnlyInternalProxies is rather trivial, it allows only your trusted frontends to connect, denying all direct requests to backend servers, a safety measure.
• RemoteHostHeader and RemotePortHeader apply respective header contents after virtual host selection and allow to define a single virtual host for backend, performing real hostname and port mapping on frontends. All .htaccess and other configuration subrequests like rewrite are done to the original virtual host defined in the backend, while i.e. backend scripts see the original Host/Port that were accessed, thinking they are actually running with the specified hostname/port and generating proper URLs from this data. So you can i.e. call your virtual host 'cmr1234.mydomain.com' and map 'yourcustomerhost.com' to 'cmr1234.mydomain.com' on the proxying frontend, sending Host of 'cmr1234.mydomain.com' in the request to backend, and specific headers for the original host/port accessed.
• RemoteProtoHeader / RemoteHTTPSEnableProto is a widely used setup supplement. It allows to perform all SSL on frontend balancers, while running backend over HTTP only in a trusted network, removing SSL overhead from backends. This allows to create a header to inform backend scripts and other dynamic stuff the original request came via SSL to make it generate HTTPS URLs while the backend will always serve unencrypted. This also affects rewrite conditions so i.e. SSL=on/off dependent rules may be used. This feature removes the need of tricky SSL/non-SSL state handling on backends in such setups, faking SSL state and allowing third party stuff to run unmodified in SSL proxy setups.

[tomsommer]

Should the options not be prefixed RemoteIP*? So RemoteIPHostHeader etc.

Would love to see this added to master 👍

[AlexAT]

Should the options not be prefixed RemoteIP*? So RemoteIPHostHeader etc.

Well, they are technically part of remoteip module, but as they do not exactly affect/specify remote IP but remote Host and Port, remote protocol flag or everything at once (in case of RemoteAllowOnlyInternalProxies), I've named them accordingly.

[tomsommer]

I tried this patch. It works, but there is something with RemoteAllowOnlyInternalProxies On, it returns 403 for requests that run through .htaccess - i can't really figure out any other pattern of why it happens.

But excellent patch otherwise, thank you :)

[AlexAT]

I tried this patch. It works, but there is something with RemoteAllowOnlyInternalProxies On, it returns 403 for requests that run through .htaccess - i can't really figure out any other pattern of why it happens.

If you please can share a bit of configuration that allows to reproduce the issue, I'll definitely look into it as I want this to be as clean as possible.

If it's not exactly publicly sharable but possible to send in private, I can be reached by mail at alex [meow] alex-at.net

[tomsommer]

It's really hard to debug, but after some time (many reloads) it's like things leak from one request to another. Suddenly %HTTPS in mod_rewrite doesn't return the correct value, where on the previous request it did. After a httpd restart it works again for a little while.

I'm using

  RemoteIPInternalProxy 127.0.0.1/32
  RemoteIPHeader X-Forwarded-For
  RemoteHostHeader X-Forwarded-Host
  RemotePortHeader X-Forwarded-Port
  RemoteProtoHeader X-Forwarded-Proto
  RemoteHTTPSEnableProto https
  RemoteAllowOnlyInternalProxies Off

[AlexAT]

Hello, @tomsommer

I have been using this on my massive shared hostings for years, especially with mod_rewrite for customers, and am not seeing this issue. HTTPS state adjustment relies on the internal Apache hook, so I wonder.

Nevertheless, if there is any possibility of this happening, I want to investigate it as it potentially may shoot us in the leg as well. First things first, please state the MPM and list of loaded modules you are using, also the nature of the request (to static files, to dynamic code). Having full mod_rewrite rules (without sensitive data) you are able to reproduce this on also would be very helpful. That's because I suspect this could be mpm_event or other threaded MPM, then I would have to thoroughly doublecheck the local flags scope in the code and dig in that direction (we are using mpm_event at places for it as well, but not for shared hostings, still had never seen the issue, but yes, anyways I want to thoroughly check that).

[AlexAT]

@tomsommer

Okay, knowing it's mpm_event already points me to the test set creation. Not promising it would be fast to diagnose, but I'll surely post if I find (or do not) something.

[tomsommer]

There is definitely some leakage between requests on mpm_event, the HTTPS variable is switching between on and off when hitting the same non-https page, even though HTTP_X_FORWARDED_PROTO=http and HTTP_X_FORWARDED_PORT=80 remain.

Don't know if gnif/mod_rpaf#42 is relevant

[tomsommer]

I turned off keep-alive on the upstream in nginx and that solved the problem, but of course I would like to use keep-alive :)

[AlexAT]

@tomsommer

Yes, I see where it may happen now. It is always spurious 'https on' state that should not be there, am I correct?
Not related to MPM, but to keepalive connections indeed. This appeared due to RemoteHTTPSEnableProto handling changes.
Thanks!!!

[AlexAT]

@tomsommer

Please test with the latest commit added. Should fix the leakage of HTTPS flag under the conditions stated in the commit description.

[tomsommer]

@AlexAT Seems to be fixed now 👍

[ylavic]

Please test with the latest commit added. Should fix the leakage of HTTPS flag under the conditions stated in the commit description.

IIUC the scenario fixed here is httpd with Keepalive on (listening on an http or https port), a first request arrives at the frontend on an https port so it sets e.g. X-Forwarded-Proto: https which causes HTTPS=on in httpd, then a second request arrives at the frontend on an http port so it now sets X-Forwarded-Proto: http and forwards the request on the same httpd kept alive connection leading to HTTPS=off (or unset) in httpd.

So RemoteHTTPSEnableProto is supposed to force the HTTPS env var according to what the frontend says, regardless of whether httpd is listening on https or http, right?
I suppose that there are cases where the communications between the frontend and httpd can be considered secure/private even if it's not TLS (or the other way around), and that controlling HTTPS at the frontend can be useful for e.g. generating the right URL schemes in scripts/redirects, but that'll probably need a big warning in the docs stating that HTTPS=on can be set for non-TLS httpd connections (or the other way around, though maybe less critical).

Btw, it seems that HTTPS=off (or unsetting) is missing in remoteip_hook_fixups(), and that remoteip_hook_http_scheme() is registered APR_HOOK_LAST thus runs after mod_ssl's ssl_hook_http_scheme() at APR_HOOK_MIDDLE (so if SSLEngine is on, the scheme will always be "https" regardless of the frontend, which may not be consistent with the above?).

[AlexAT]

Hello, @ylavic

Thanks for the review, I'll do the changes accordingly today or tomorrow.

[AlexAT]

For the hook order and emulating off mode when it's SSL module handling requests, I assume I will have to use APR_HOOK_REALLY_FIRST given the nature of the function then if and then only if config->proto_header_name is set, return DONE instead of DECLINED so the process is complete, but the result is not OK (as it's compared with OK in ap_ssl_conn_is_ssl() @ ssl.c).

Will first check what adverse effects it may have. [glancing through the code only shows mod_nw_ssl may be not compatible]
There's a fallback to ssl_is_https() inside ap_ssl_conn_is_ssl() and so it will probably not bode well with mod_nw_ssl.

[ylavic]

For the hook order and emulating off mode...

I was talking about the remoteip_hook_http_scheme() actually, I don't think mod_remote_ip should hook into ssl_conn_is_ssl, that one should stay local IMO (we need a way in httpd to know for sure whether the local connection is TLS or not).

I didn't look closely at all the ap_ssl_conn_is_ssl() usages, but the two local and remote indicators shouldn't be mixed..

[ylavic]

This may mean adding a new ssl_conn_remote_is_ssl hook and using ap_ssl_conn_remote_is_ssl() where ap_ssl_conn_is_ssl() is currently misused if/when these changes to mod_remote_ip are in.

It could be an APR_OPTIONAL_FN too, provided by mod_remote_ip and uses by e.g. ap_conn_is_remote_ssl() if it exists.

[AlexAT]

Just to add to it, 'HTTPS' and stuff variants without 'local' or 'remote' still remain, but they're technically just aliases to 'remote', whatever it is.

[AlexAT]

Not in scope of the pull request itself but for those who use it, the variant for 2.4.56 (lacks APLOGNO(), lacks some docs changes).

httpd-2.4-remoteip-2.4.56-20230327.patch

Take care this can still contain bugs I do not know about. I've put this on a single production cluster and it has no issues atm, but anyways, take care, test before using.
Functional change from the initial 2.4.48 variant: RemoteHTTPSEnableProto now defaults back to 'https', to check just for header presence, use "*" for the option. All other changes are based on the review and are described above & in the commit series.
Technically should now work properly with HTTPS connection from frontend to backend as well.

[tomsommer]

Is this working in 2.4.57?

Maybe change the title of the pull-request :)

[AlexAT]

Alas, maintaining this onwards for next versions has become harder as it contains more changes after reviewing :)

Here we go:

httpd-2.4-remoteip-2.4.57-20220531.patch

Totally the same as 2.4.56 one above, adjusted for changed ap_mmn.h. Builds well, should work all the same as there are no related changes to the core it seems.

EDIT @ 31/05/2023: geez, uploaded wrong file 30/05/2023. Please use the one I placed here 31/05/2023 for builds.

[AlexAT]

I think this can already be merged actually - but we need maintainers to somehow check and re-review it beforehand.

[AlexAT]

I have by mistake uploaded adjusted older version of the patch in variant for 2.4.57. Terribly sorry for that.
The file above has been replaced and I'm duplicating new one here: httpd-2.4-remoteip-2.4.57-20220531.patch

[AlexAT]

This post is just to confirm latest patch version is running in my production environment @ version 2.4.57 without any new or noticeable issues for a month already.

[shaneshort]

I too would love to see this merged, this would be exceptionally handy for our platform.

[covener]

Maybe this has been debunked already, but is the new hook really needed vs existing scheme hook -- and trusting the scheme for things like %{HTTPS}e?

[AlexAT]

@covener Yes, the point was remote scheme should not affect local modules behavior and be separate, as it indeed can be HTTP/HTTPS flag over HTTP or HTTPS or whatever.

[shaneshort]

@AlexAT I don't suppose you've got some binaries built for this? I'm currently stuck between a rock and a hard place with RPAF not liking keepalive and another bug that goes away with keepalive enabled.

[AlexAT]

@shaneshort
Well, I can potentially provide "alternative module" RPM set for "CentOS 9" based platforms (RH9, OL9, etc.) with latest Apache, but as I won't do regular updates and check internal platform compatibility too deeply, its usage would as usual be on your own risk :)

Upd. It seems OL 9.3 would come with Apache 2.4.57 out of the box, for this I can build the replacement module rather seamlessly I think. I don't know if 'CentOS compatible' would be ok for you or you can wait for it, but once it's out, I'll post links to the 64-bit RPM with mod_remoteip_alexat (with a different name so it can be used along but instead of original module). 2.4.57/58 is what I actually run the modified module with, so should be more or less safe to use.

[AlexAT]

@shaneshort Okay, here we go, there is a build for OL 9.3 release where native httpd is altered and rebuilt. Few points beforehand:

• Any use is at your own risk, totally no guarantees whatever
• OL 9.3 only, as it's the first version coming with native Apache 2.4.57, may be usable on CentOS or compatibles, but not tested
• This build replaces native httpd RPMs and stays over them (Epoch is increased) even if OS native RPMs update
• I can't guarantee I'll provide updated builds timely or at all when native source RPM changes
• Compared to the patch in this branch, I skipped increase in Apache internal module version number so we don't cause conflicts with existing modules
• I tested at least mod_perl to work with this build, so existing modules should work, but some may require rebuild (although I can't imagine why) - did not test too extensively

Two additional minor and totally non-intrusive / non-default-behavior-changing patches are included as they are in my 'hosting' patch subset which I based the build on (I'm actually going to use this build myself). Both patches are of course public and can obviously be found in the source RPM.

• One adds useful %a and %A specifiers to RequestHeader directive. %a returns local client IP of the request. %A returns local bound IP of the request. Both are intended to be used with 'early' modifier and are mostly to set connecting frontend IP and serving backend IP into headers. Sample use case: RequestHeader set X-Frontend-IP %a early , and now you can have your connecting frontend IP tracked along with X-Real-IP retrieved and substituted by mod_remoteip as request client IP
• Another adds 'host' option to UseCanonicalName directive. It allows to use contents of incoming Host header as 'canonical server name' for mod_rewrite and other processing. Intended to only be used in very specific scenarios where canonic server name in mod_rewrite and elsewhere needs to be different from the hostname passed by frontend for virtual host selection, is to be used only under Host-filtering frontend as it may have certain security implications when Host header contents are not checked and cannot be trusted. Sample use case: user accesses host.domainA. Frontend passes host.domainA as X-Real-Host for virtualhost selection, but the backend is domainB and so the accessed host is host.domainB. If we want canonical server name to still stay host.domainA for the request (so hosted code, mod_rewrite and whatever is to be made 100% unaware it runs as host.domainB virtualhost), we put UseCanonicalName host to the virtualhost and that's it.

Related repository for OL9.3 x86_64 can be browsed at: https://yum.alex-at.net/ol9/native/

If .repo file is desired, it is at https://yum.alex-at.net/alex-at-ol9-native.repo , but take care I don't gurantee uptime and overall availability of the repository

Source RPM is provided so you can check what is inside and build yourself if you (pretty much obviously) do not trust the binaries built, at https://yum.alex-at.net/ol9/native/httpd-2.4.57-5.0.1.alex.1.el9.x86_64.rpm

[AlexAT]

To maintainers: please please check if there are any chances this pull request can be re-reviewed and possibly accepted.
It's really useful for i.e. split HTTPS(2/3) frontend and internal HTTP/HTTPS backend scenarios ultimately common nowadays.

[tomsommer]

@AlexAT Thanks for all this. Could you make a patch for 2.4.58?

[AlexAT]

Hello @tomsommer , sure, I'll look into that this weekend as I have to build it for myself anyways.

[AlexAT]

Hello @tomsommer
Here we go: httpd-2.4-remoteip-2.4.58-20231211.patch
Please take care I have not tested in in production yet, but the changes are minor.

[tomsommer]

@AlexAT and 2.4.59? :)

Thanks for all the work on this, I hope this gets merged ASAP. otherwise maybe you could create a new mod_remoteip so it's not just a patch?

[AlexAT]

@tomsommer

Here we go:
httpd-2.4-remoteip-2.4.59-20240603.patch

(only a minor 'magic number' change since the previous patch, it is running for me for a good while, no issues it seems)