Regex rewriting of upstream/downstream headers in proxy #2144

comp500 · 2018-04-28T01:04:26Z

1. What does this change do, exactly?

Adds an extra parameter to header_upstream and header_downstream that allows regex-based replacement of headers. They are in the format header_upstream [header] [regex] [replacement]

This allows unwanted values from the server and client (e.g. redirects, cookies) to be modified by Caddy. This therefore allows search (and mobile) to be fixed on wikipedia.matt.life, and wikimirror.

2. Please link to the relevant issues.

#442 - An existing (but not merged) implementation of Location rewriting
#606 - Nginx proxy_redirect feature request
This change may be obsoleted by #1639, the proxy middleware rewrite.

3. Which documentation changes (if any) need to be made because of this PR?

Add the extra parameter to header_upstream and header_downstream, and describe it. Give examples of it being used to change the Location header, possibly referencing proxy_redirect.

4. Checklist

I have written tests and verified that they fail without my change
I have squashed any insignificant commits
This change has comments for package types, values, functions, and non-obvious lines of code
I am willing to help maintain this change if there are issues with it later

CLAassistant · 2018-04-28T01:04:33Z

All committers have signed the CLA.

tobya · 2018-04-30T21:56:41Z

Hi @comp500
Thank you for the PR. Can you give a specific example of how you would write the header_upstream directive with your new functionality and what it would be used for?
Thanks.

comp500 · 2018-04-30T22:08:13Z

It could be used for modifying redirects for a mirrored site. For example, Wikipedia uses absolute redirects to en.wikipedia.org for the homepage and search.

The directive could be used like this:

wikipedia.yoursite.com

proxy / https://en.wikipedia.org {
    header_downstream Location en.wikipedia.org wikipedia.yoursite.com
}

This would ensure that all redirects are for the correct host, rather than just the main page as in Matt's tweet.
It works for any header sent to or from the proxied server, in the format header_upstream [header] [regex] [replacement] or header_downstream [header] [regex] [replacement].

mholt · 2018-06-17T15:16:01Z

This is cool. Hopefully I or someone else gets a chance to review this soon!

lukenowak · 2018-08-10T13:29:23Z

Are there any news regarding this pull request? Can I help somehow (eg. giving it a try in my environments and sharing the results)?

dyrkin · 2018-08-16T09:29:25Z

caddyhttp/proxy/upstream_test.go

+		{"proxy / localhost:8080 {\n transparent \nheader_upstream X-Test Tester \nheader_upstream X-Test Test Host \n}"},
+
+		// Test #3: transparent preset with multiple params
+		{"proxy / localhost:8080 {\n transparent \nheader_upstream X-Test Tester \nheader_upstream X-Test Test Host \nheader_upstream X-Test er ing \n}"},


@comp500
As I understand it, the first arg is a name of a header, the second is the regular expression of what we want to replace, and the last argument is the value for the replacement.
Will it work if the last argument contains white spaces?

You are correct. It will work if any argument contains white spaces, because you can surround each argument in quotes, so the white spaces within are part of the argument.
e.g. header_downstream Location https://en.wikipedia.org "http://local host"

tobya · 2018-08-16T10:22:48Z

@lukenowak it would be great if you could test in your environment.,

dyrkin · 2018-08-16T11:36:10Z

UPD

I'm trying to run it locally, but I get an error:
2018/08/16 13:28:52 /etc/caddy/Caddyfile:39 - Error during parsing: unknown property '443'

This is what I did:

dyrkin@nas:~/go/src/github.com/comp500/caddy/caddy$ git branch
  master
* rewriteheader

dyrkin@nas:~/go/src/github.com/comp500/caddy/caddy$ go run build.go

dyrkin@nas:~/go/src/github.com/comp500/caddy/caddy$ ./caddy -log stdout -agree=true -conf=/etc/caddy/Caddyfile -root=/var/tmp
    2018/08/16 13:28:52 /etc/caddy/Caddyfile:39 - Error during parsing: unknown property '443'

My Caddyfile

webmin.example.com {
    import wildcard_cert

    proxy / http://localhost:10000/ {
        transparent
        header_downstream Location "10000" "443"
    }
}

Did I miss something?

Update
I figured out the root cause of the issue. To fix it I removed original github.com/mholt and renamed github.com/comp500 folder to github.com/mholt, after that I ran the build again.
Sorry, I'm new in go.

I verified this functionality and it works excellent.
Directive header_downstream Location http://webmin.example.com:10000/ https://webmin.example.com/ works for me.

comp500 · 2018-08-16T14:27:47Z

Your caddyfile seems to work fine for me. Try checking out the branch in the ~/go/src/github.com/mholt/caddy folder instead, as the build script might be using the wrong package.

lukenowak · 2018-08-28T10:05:23Z

@tobya I was able to remove specific cookies from the browser sent to the upstream by using:

http://luke.example.com:11080 {
  bind 127.0.10.1
  proxy / http://127.0.10.1:8888 {
    header_upstream Cookie "(.*)(^Chocolate=[^;]*; |; Chocolate=[^;]*|^Chocolate=[^;]*$)(.*)" "$1 $3"
    header_upstream Cookie "(.*)(^Cacao=[^;]*; |; Cacao=[^;]*|^Cacao=[^;]*$)(.*)" "$1 $3"
  }
}

It is super cool, that one header can be rewritten many times.

So for me this feature is super useful and flexible.

Not released yet functionality for regular expression cookie rewriting is available: caddyserver/caddy#2144

mholt

Thanks, this is a useful change and definitely useful. I've only made a brief pass, but it's looking mostly good so far.

Add the extra parameter to header_upstream and header_downstream, and describe it. Give examples of it being used to change the Location header, possibly referencing proxy_redirect.

Can you write up the specific docs for this? This is a complex feature and it needs to be documented correctly. For example, are placeholders (captures) supported? To what extent can people use Go's regular expressions here?

caddyhttp/proxy/upstream.go

comp500 · 2018-09-08T21:05:13Z

I've written some docs in this gist, to be added to https://caddyserver.com/docs/proxy. If anything is unclear in it, please comment on the gist.

Not released yet functionality for regular expression cookie rewriting is available: caddyserver/caddy#2144

LaurensBosscher · 2018-11-02T12:40:57Z

@mholt , this PR would be very useful for us and seems almost complete.

Is there any way I can contribute to getting this merged?

Thanks!

francislavoie · 2018-11-02T14:18:40Z

@LaurensBosscher You could help by testing the changes. Check out the latest master and merge this PR locally, compiling from source. You can also review the code and point out any potentially questionable behaviour if any.

Not released yet functionality for regular expression cookie rewriting is available: caddyserver/caddy#2144 Caddy v0.11.1 fixes QUIC issues. Zenexer fixes for regerssion in v0.11.1 Note: Since Caddy release v0.11.0 certificates has to match sites served by Caddy. This will result in lack of response for sites served on HTTPS for which certificate names (subject COMMON_NAME, subjectAltName DNS or IP) does not match site name.

Caddy v0.11.1 fixes QUIC issues. Not released yet functionality for regular expression cookie rewriting is available: caddyserver/caddy#2144 Not released yet functionality for ca_certifices in proxy: caddyserver/caddy#2380 Zenexer fixes for regerssion in v0.11.1 Note: Since Caddy release v0.11.0 certificates has to match sites served by Caddy. This will result in lack of response for sites served on HTTPS for which certificate names (subject COMMON_NAME, subjectAltName DNS or IP) does not match site name.

Ramshackles · 2019-02-27T08:23:33Z

Very much interested in this PR 👍

tobya · 2019-03-04T21:22:45Z

@Ramshackles Any chance you would be able to compile locally and test this change?

lukenowak · 2019-03-06T09:54:30Z

@comp500 are you willing to continue working on this? Rebase on top of master (trivial change)?

@tobya I tried to run the test suite after rebasing this work on master, but my go environment seems "damaged" (caddytls/config.go:410:31: undefined: tls.VersionTLS13 and FAIL github.com/mholt/caddy/caddytls [build failed] during go test ./...), for now I have no time to investigate it further.

mholt · 2019-03-06T12:45:33Z

@lukenowak Based on that I bet you just need to update to Go 1.12 to fix the error.

Ramshackles · 2019-03-06T13:36:34Z

@Ramshackles Any chance you would be able to compile locally and test this change?

Not sure how I would go about doing it, but guess I could try if needed.

comp500 · 2019-03-06T19:38:26Z

@lukenowak I am certainly willing to continue working on this. I've rebased my PR on top of master, although the AppVeyor build seems to still be using Go 1.11 for some reason.

francislavoie · 2019-03-06T19:40:41Z

Yeah, just ignore the AppVeyor CI failures for now. AppVeyor still doesn't support Go 1.12 appveyor/ci#2875

mholt

Thanks for the work. :) Let's give this a shot.

ghost · 2019-04-02T15:45:24Z

Any chance to ship this one into a release soon? I found out that many back-ends behave badly and don't honor the X-Scheme or X-Forwarded-Proto headers and send 302 redirects to an http scheme even when the original request is https, and this is often blocked by the browser...
The only workaround is downstream Location header rewriting, so I'm really hoping to get this PR released...

mholt · 2019-04-02T15:46:09Z

It'll go out in our next release, which should be 1.0beta1.

This reverts commit 6d2019b, as new caddy has issues with tls certificate configuration: caddyserver/caddy#2588 About nxd-v0.11.5-4-g9d3151db: * not released yet functionality for regular expression cookie rewriting is available: caddyserver/caddy#2144 * not released yet functionality for ca_certifices in proxy: caddyserver/caddy#2380 * support for builtin log rotation disabling

CristianCantoro · 2019-09-12T12:48:18Z

@comp500 said:

This allows unwanted values from the server and client (e.g. redirects, cookies) to be modified by Caddy. This therefore allows search (and mobile) to be fixed on wikipedia.matt.life, and wikimirror.

Oh my goodness, more than one year after I discover this that in fact fixes a problem that I reported just recently on Wikimedia Phabricator (T232213) and here (#1011).

This is so awesome! It's hearthwarming knowing that there is somebody out there fixing my bugs even without me knowing it!

I have already pushed it on vikiansiklopedi.org and wikiproxy (formerly wikimirror, because - well - it's a proxy ont a mirror)

mholt · 2019-09-14T19:40:31Z

FWIW, I just got this implemented into Caddy 2: https://github.com/caddyserver/caddy/wiki/v2:-Documentation#httphandlersreverse_proxy

It reuses the existing http.handlers.headers module, which also has the same ability to mutate headers by adding, deleting, setting, or replacing substrings in header values. It's pretty powerful. Try it out and let me know what you think!

dpastoor mentioned this pull request Jun 15, 2018

updating rstudio caddy configuration boettiger-lab/servers#1

Merged

dyrkin reviewed Aug 16, 2018

View reviewed changes

NexediGitlab pushed a commit to SlapOS/slapos that referenced this pull request Sep 3, 2018

component/caddy: Upgrade to nxd-v0.11.0-3-g12438f6

5eb76fa

Not released yet functionality for regular expression cookie rewriting is available: caddyserver/caddy#2144

mholt requested changes Sep 7, 2018

View reviewed changes

caddyhttp/proxy/upstream.go Outdated Show resolved Hide resolved

caddyhttp/proxy/upstream.go Outdated Show resolved Hide resolved

NexediGitlab pushed a commit to SlapOS/slapos that referenced this pull request Sep 11, 2018

component/caddy: Upgrade to nxd-v0.11.0-3-g12438f6

365f446

Not released yet functionality for regular expression cookie rewriting is available: caddyserver/caddy#2144

francislavoie mentioned this pull request Dec 7, 2018

Proxy support for rewriting cookie path #2381

Closed

mkrensing mentioned this pull request Jan 14, 2019

http 502 - caddy proxies to svn server #2018

Closed

whitestrake requested a review from mholt February 26, 2019 05:41

comp500 added 3 commits March 6, 2019 19:21

Add upstream header replacements (TODO: tests, docs)

fb678cf

Add tests, fix a few bugs

579c15f

Add more tests and comments

b955819

Refactor header_upstream to use a fallthrough; return regex errors

80c5c15

comp500 force-pushed the rewriteheader branch from e33be14 to 80c5c15 Compare March 6, 2019 19:21

mholt approved these changes Mar 6, 2019

View reviewed changes

mholt merged commit 47b7871 into caddyserver:master Mar 6, 2019

mholt mentioned this pull request Mar 6, 2019

Implement the equivalent of nginx proxy_redirect directive #606

Closed

ghost mentioned this pull request Mar 29, 2019

Wrong scheme in login redirection pyload/pyload#3352

Closed

mholt mentioned this pull request Sep 13, 2019

header_upstream Host {host} cause "Domain not configured" error #2745

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Regex rewriting of upstream/downstream headers in proxy #2144

Regex rewriting of upstream/downstream headers in proxy #2144

comp500 commented Apr 28, 2018 •

edited

Loading

CLAassistant commented Apr 28, 2018 •

edited

Loading

tobya commented Apr 30, 2018

comp500 commented Apr 30, 2018

mholt commented Jun 17, 2018

lukenowak commented Aug 10, 2018

dyrkin Aug 16, 2018 •

edited

Loading

comp500 Aug 16, 2018

tobya commented Aug 16, 2018

dyrkin commented Aug 16, 2018 •

edited

Loading

comp500 commented Aug 16, 2018 •

edited

Loading

lukenowak commented Aug 28, 2018

mholt left a comment

comp500 commented Sep 8, 2018

LaurensBosscher commented Nov 2, 2018

francislavoie commented Nov 2, 2018

Ramshackles commented Feb 27, 2019

tobya commented Mar 4, 2019

lukenowak commented Mar 6, 2019

mholt commented Mar 6, 2019

Ramshackles commented Mar 6, 2019

comp500 commented Mar 6, 2019

francislavoie commented Mar 6, 2019

mholt left a comment

ghost commented Apr 2, 2019

mholt commented Apr 2, 2019

CristianCantoro commented Sep 12, 2019

mholt commented Sep 14, 2019

Regex rewriting of upstream/downstream headers in proxy #2144

Regex rewriting of upstream/downstream headers in proxy #2144

Conversation

comp500 commented Apr 28, 2018 • edited Loading

1. What does this change do, exactly?

2. Please link to the relevant issues.

3. Which documentation changes (if any) need to be made because of this PR?

4. Checklist

CLAassistant commented Apr 28, 2018 • edited Loading

tobya commented Apr 30, 2018

comp500 commented Apr 30, 2018

mholt commented Jun 17, 2018

lukenowak commented Aug 10, 2018

dyrkin Aug 16, 2018 • edited Loading

Choose a reason for hiding this comment

comp500 Aug 16, 2018

Choose a reason for hiding this comment

tobya commented Aug 16, 2018

dyrkin commented Aug 16, 2018 • edited Loading

comp500 commented Aug 16, 2018 • edited Loading

lukenowak commented Aug 28, 2018

mholt left a comment

Choose a reason for hiding this comment

comp500 commented Sep 8, 2018

LaurensBosscher commented Nov 2, 2018

francislavoie commented Nov 2, 2018

Ramshackles commented Feb 27, 2019

tobya commented Mar 4, 2019

lukenowak commented Mar 6, 2019

mholt commented Mar 6, 2019

Ramshackles commented Mar 6, 2019

comp500 commented Mar 6, 2019

francislavoie commented Mar 6, 2019

mholt left a comment

Choose a reason for hiding this comment

ghost commented Apr 2, 2019

mholt commented Apr 2, 2019

CristianCantoro commented Sep 12, 2019

mholt commented Sep 14, 2019

comp500 commented Apr 28, 2018 •

edited

Loading

CLAassistant commented Apr 28, 2018 •

edited

Loading

dyrkin Aug 16, 2018 •

edited

Loading

dyrkin commented Aug 16, 2018 •

edited

Loading

comp500 commented Aug 16, 2018 •

edited

Loading