Disable URL decoding #1615

gayan415 · 2017-10-28T02:29:37Z

Is this a BUG REPORT or FEATURE REQUEST?
Maybe a bug or a question.

NGINX Ingress controller version:
gcr.io/google_containers/nginx-ingress-controller:0.8.3

Kubernetes version (use kubectl version):
1.7.2

Environment:
Kubernetes cluster on AWS. I have an ELB in front of the cluster

What happened:
I deployed scormengine inside my k8s cluster and I have an url-encoded curl request to scormengine service.

curl -u user:pwd http://xxxxxx.com/scormengine/api/v1/default/registrations/learning-activity%3A12721%3Buser%3A91701.

I can see the correct log from nginx ingress controller. Here is the logs.
GET /scormengine/api/v1/default/registrations/learning-activity%3A12721%3Buser%3A91761/launchLink?forceReview=false. You can see the encorded values in the in the url (learning-activity%3A12721%3Buser%3A91761).

Then nginx ingress controller proxy into scormengine service accoring to the below rule.

apiVersion: extensions/v1beta1
    kind: Ingress
    metadata:
      name: qc-scormengine-ingress
      labels:
        app: qc-scormengine
      annotations:
        kubernetes.io/ingress.class: nginx
        ingress.kubernetes.io/rewrite-target: /
    spec:
      rules:
      - host: "kube-qc.bluedrop360.com"
        http:
          paths:
          - path: /scormengine
            backend:
              serviceName: qc-scormengine-svc
              servicePort: 80
          - path: /scormengine/*
            backend:
              serviceName: qc-scormengine-svc
              servicePort: 80

when it comes to scormengine pod logs, what I can see is

httpMethod":"GET","url":"http://xxxxxx.com/api/v1/default/registrations/learning-activity:12721;user:91761"

So the values are decoded here. Im suspecting that nginx ingress did url decoding.

How can I disable URL decoding on nginx ingress controller ?

What you expected to happen:
I need to send the same url to scormengine without decoding it, which means I need to hit

httpMethod":"GET","url":"http://xxxxxx.com/api/v1/default/registrations/learning-activity%3A12721%3Buser%3A91761" to scormengine pod.

Anything else we need to know:
Few sections from nginx ingress controller config file

location ~* /scormengine/* {
  	
  	proxy_set_header Host                   $host;
  
  	# Pass Real IP
  	proxy_set_header X-Real-IP              $remote_addr;
  
  	# Allow websocket connections
  	proxy_set_header                        Upgrade           $http_upgrade;
  	proxy_set_header                        Connection        $connection_upgrade;
  
  	proxy_set_header X-Forwarded-For        $proxy_add_x_forwarded_for;
  	proxy_set_header X-Forwarded-Host       $host;
  	proxy_set_header X-Forwarded-Port       $server_port;
  	proxy_set_header X-Forwarded-Proto      $pass_access_scheme;
  
  	# mitigate HTTPoxy Vulnerability
  	# https://www.nginx.com/blog/mitigating-the-httpoxy-vulnerability-with-nginx/
  	proxy_set_header Proxy                  "";
  
  	proxy_connect_timeout                   5s;
  	proxy_send_timeout                      60s;
  	proxy_read_timeout                      60s;
  
  	proxy_redirect                          off;
  	proxy_buffering                         off;
  
  	proxy_http_version                      1.1;
  
  rewrite /scormengine/*/(.*) /$1 break;
  rewrite /scormengine/* / break;
  proxy_pass http://qc-qc-scormengine-svc-80;
  
  }
  	
  	
  location ~* /scormengine {
  	proxy_set_header Host                   $host;
  
  	# Pass Real IP
  	proxy_set_header X-Real-IP              $remote_addr;
  
  	# Allow websocket connections
  	proxy_set_header                        Upgrade           $http_upgrade;
  	proxy_set_header                        Connection        $connection_upgrade;
  
  	proxy_set_header X-Forwarded-For        $proxy_add_x_forwarded_for;
  	proxy_set_header X-Forwarded-Host       $host;
  	proxy_set_header X-Forwarded-Port       $server_port;
  	proxy_set_header X-Forwarded-Proto      $pass_access_scheme;
  
  	# mitigate HTTPoxy Vulnerability
  	# https://www.nginx.com/blog/mitigating-the-httpoxy-vulnerability-with-nginx/
  	proxy_set_header Proxy                  "";
  
  	proxy_connect_timeout                   5s;
  	proxy_send_timeout                      60s;
  	proxy_read_timeout                      60s;
  
  	proxy_redirect                          off;
  	proxy_buffering                         off;
  
  	proxy_http_version                      1.1;
  
  rewrite /scormengine/(.*) /$1 break;
  rewrite /scormengine / break;
  proxy_pass http://qc-qc-scormengine-svc-80;
  
  }

  upstream qc-qc-scormengine-svc-80 {
      least_conn;
      server XXX.XX.X.XXX:8880 max_fails=0 fail_timeout=0;
      
  }

The text was updated successfully, but these errors were encountered:

aledbf · 2017-10-28T02:37:18Z

@thegayanj please update to the latest 0.9 beta version.
Also please remove the /scormengine/* path. That is not required. In nginx only when you use regex in the mappings (not supported here)

gayan415 · 2017-10-28T02:38:36Z

Hey @aledbf - Let me try and let you know. Thank you.

gayan415 · 2017-10-28T02:56:58Z

Hey @aledbf.
nginx-ingress controller image changed to gcr.io/google_containers/nginx-ingress-controller:0.9.0-beta.15.

Still experiencing the same issue.

New nginx ingress config

location ~* ^/scormengine\/?(?<baseuri>.*) {

	set $proxy_upstream_name "qc-qc-scormengine-svc-80";

	set $namespace      "qc";
	set $ingress_name   "qc-scormengine-ingress";
	set $service_name   "";

	port_in_redirect off;

	client_max_body_size                    "1m";

	proxy_set_header Host                   $best_http_host;
	# Pass the extracted client certificate to the backend

	# Allow websocket connections
	proxy_set_header                        Upgrade           $http_upgrade;
	proxy_set_header                        Connection        $connection_upgrade;

	proxy_set_header X-Real-IP              $the_real_ip;
	proxy_set_header X-Forwarded-For        $the_real_ip;
	proxy_set_header X-Forwarded-Host       $best_http_host;
	proxy_set_header X-Forwarded-Port       $pass_port;
	proxy_set_header X-Forwarded-Proto      $pass_access_scheme;
	proxy_set_header X-Original-URI         $request_uri;
	proxy_set_header X-Scheme               $pass_access_scheme;

	# Pass the original X-Forwarded-For
	proxy_set_header X-Original-Forwarded-For $http_x_forwarded_for;

	# mitigate HTTPoxy Vulnerability
	# https://www.nginx.com/blog/mitigating-the-httpoxy-vulnerability-with-nginx/
	proxy_set_header Proxy                  "";

	# Custom headers to proxied server

	proxy_connect_timeout                   5s;
	proxy_send_timeout                      60s;
	proxy_read_timeout                      60s;

	proxy_redirect                          off;
	proxy_buffering                         off;
	proxy_buffer_size                       "4k";
	proxy_buffers                           4 "4k";
	proxy_request_buffering                 "on";

	proxy_http_version                      1.1;

	proxy_cookie_domain                     off;
	proxy_cookie_path                       off;

	# In case of errors try the next upstream server before returning an error
	proxy_next_upstream                     error timeout invalid_header http_502 http_503 http_504;

rewrite /scormengine/(.*) /$1 break;
rewrite /scormengine / break;
proxy_pass http://qc-qc-scormengine-svc-80;
}

upstream qc-qc-scormengine-svc-80 {
	# Load balance algorithm; empty for round robin, which is the default
	least_conn;

	keepalive 32;

	server XXX.XX.X.XXX:8880 max_fails=0 fail_timeout=0;
}

any other suggestions?

gayan415 · 2017-10-28T12:29:23Z

How do I know that this is an ingress issue

Get the scormengine pod internal IP by running kubectl exec -it qc-scormengine-pod -n qc cat /etc/hosts.
exec into ingress controller - kubectl exec -it ingress_ctr_pod /bin/sh
curl to scormengine with encoded URL from ingress controller - curl -u user:pwd http://XXX.XX.X.XXX:8880/api/v1/default/registrations/learning-activity%3A12721%3Buser%3A91701/launchLink?forceReview=false.
Its working as expected and I can see the url encoded curl request from scormengine pod and scormengine returned 200.

Which means ingress controller decode the URL before forwarding to scormengine pod.

aledbf · 2017-10-28T12:34:16Z

@thegayanj to avoid the decoding you can:

remove the rewrite annotation (this is the reason why the uri is decoded)
create a custom template and use something like:

map '' $seed_uri {
        default $request_uri;
}

proxy_pass  http://qc-qc-scormengine-svc-80$seed_uri;

gayan415 · 2017-10-28T21:55:25Z

Thank you for your suggestion @aledbf. I created a custom template and I referred this example.

map '' $seed_uri {
        default $request_uri;
}

Now I have a new map (as above) on ingres controller config file (/etc/nginx/nginx.conf).
I'm not sure how can I config my ingress to get proxy_pass http://qc-qc-scormengine-svc-80$seed_uri; Because proxy_pass created in pkg/nginx/template/template.go . I tried few ways and no luck, couldn't find a good example either.
Here is my ingress manifest.

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: qc-scormengine-ingress
  labels:
    app: qc-scormengine
spec:
  rules:
  - host: "xxxxxxxx.com"
    http:
      paths:
      - path: /scormengine
        backend:
          serviceName: qc-scormengine-svc
          servicePort: 80

Can you instruct/guide me to do that please, thank you and I really appreciate that.

gayan415 · 2017-10-30T21:07:51Z

@aledbf , Okay.

Here is my nginx ingress controller config file now.

map '' $seed_uri {
	default          $request_uri;
}

upstream qc-qc-scormengine-svc-80 {
	# Load balance algorithm; empty for round robin, which is the default
	least_conn;

	keepalive 32;

	server xxx.xx.x.xxx:8880 max_fails=0 fail_timeout=0;
}

server {
	server_name xxxxxxx.com;
	listen 80;
	listen [::]:80;
	set $proxy_upstream_name "-";
	location /scormengine {
		set $proxy_upstream_name "qc-qc-scormengine-svc-80";

		set $namespace      "qc";
		set $ingress_name   "qc-scormengine-ingress";
		set $service_name   "qc-scormengine-svc";

		port_in_redirect off;

		client_max_body_size                    "1m";

		proxy_set_header Host                   $best_http_host;

		# Pass the extracted client certificate to the backend

		# Allow websocket connections
		proxy_set_header                        Upgrade           $http_upgrade;
		proxy_set_header                        Connection        $connection_upgrade;

		proxy_set_header X-Real-IP              $the_real_ip;
		proxy_set_header X-Forwarded-For        $the_real_ip;
		proxy_set_header X-Forwarded-Host       $best_http_host;
		proxy_set_header X-Forwarded-Port       $pass_port;
		proxy_set_header X-Forwarded-Proto      $pass_access_scheme;
		proxy_set_header X-Original-URI         $request_uri;
		proxy_set_header X-Scheme               $pass_access_scheme;

		# mitigate HTTPoxy Vulnerability
		# https://www.nginx.com/blog/mitigating-the-httpoxy-vulnerability-with-nginx/
		proxy_set_header Proxy                  "";

		# Custom headers to proxied server

		proxy_connect_timeout                   5s;
		proxy_send_timeout                      60s;
		proxy_read_timeout                      60s;

		proxy_redirect                          off;
		proxy_buffering                         off;
		proxy_buffer_size                       "4k";
		proxy_buffers                           4 "4k";

		proxy_http_version                      1.1;

		proxy_cookie_domain                     off;
		proxy_cookie_path                       off;

		# In case of errors try the next upstream server before returning an error
		proxy_next_upstream                     error timeout invalid_header http_502 http_503 http_504;
		# proxy_pass
		proxy_pass http://qc-qc-scormengine-svc-80$seed_uri;
	}
}

And my ingress rule is

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: qc-scormengine-ingress
  labels:
    app: qc-scormengine
spec:
  rules:
  - host: "xxxxxxx.com"
    http:
      paths:
      - path: /scormengine
        backend:
          serviceName: qc-scormengine-svc
          servicePort: 80

Then I ran curl - curl -u user:pwd https://xxxxxxx.com/scormengine/api/v1/default/ping.

This curl command is not hitting scormengine pod, any suggestions why?

Platzii · 2017-11-13T11:23:17Z

Any updates on this?
Or a workaround of some kind?

gayan415 · 2017-11-13T11:42:40Z

Nope 😐

aledbf · 2017-11-13T16:11:36Z

@thegayanj URL decoding is disabled if you remove the rewrite annotation. There's no workaround for that (NGINX restriction)

Platzii · 2017-11-14T09:48:54Z

@aledbf You mentioned this could be fixed by having a custom template. Could you give some more info on that? What's the impact on other (already existing) ingresses?

I guess with every update from the ingress controller, the custom template should be updated too then?

aledbf · 2018-01-19T00:10:47Z

I guess with every update from the ingress controller, the custom template should be updated too then?

Yes

aledbf · 2018-01-19T00:11:41Z

Closing. As I pointed here #1615 (comment) is not possible to do this with the rewrite annotation.

aledbf closed this as completed Jan 19, 2018

kpsuperplane mentioned this issue Apr 10, 2019

[APM] Transaction names ending in a "/" shows no transactions details elastic/kibana#32640

Closed

camallen mentioned this issue Aug 20, 2020

some www.galaxyzoo.org urls contain '+' chars which don't resolve via s3 server zooniverse/Talk-archiver#82

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Disable URL decoding #1615

Disable URL decoding #1615

gayan415 commented Oct 28, 2017 •

edited

aledbf commented Oct 28, 2017

gayan415 commented Oct 28, 2017

gayan415 commented Oct 28, 2017 •

edited

gayan415 commented Oct 28, 2017

aledbf commented Oct 28, 2017

gayan415 commented Oct 28, 2017 •

edited

gayan415 commented Oct 30, 2017 •

edited

Platzii commented Nov 13, 2017

gayan415 commented Nov 13, 2017

aledbf commented Nov 13, 2017

Platzii commented Nov 14, 2017

aledbf commented Jan 19, 2018

aledbf commented Jan 19, 2018

Disable URL decoding #1615

Disable URL decoding #1615

Comments

gayan415 commented Oct 28, 2017 • edited

aledbf commented Oct 28, 2017

gayan415 commented Oct 28, 2017

gayan415 commented Oct 28, 2017 • edited

gayan415 commented Oct 28, 2017

aledbf commented Oct 28, 2017

gayan415 commented Oct 28, 2017 • edited

gayan415 commented Oct 30, 2017 • edited

Platzii commented Nov 13, 2017

gayan415 commented Nov 13, 2017

aledbf commented Nov 13, 2017

Platzii commented Nov 14, 2017

aledbf commented Jan 19, 2018

aledbf commented Jan 19, 2018

gayan415 commented Oct 28, 2017 •

edited

gayan415 commented Oct 28, 2017 •

edited

gayan415 commented Oct 28, 2017 •

edited

gayan415 commented Oct 30, 2017 •

edited