Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: fix path-traversal bug #229

Merged
merged 5 commits into from Sep 8, 2022
Merged

Conversation

ruokeqx
Copy link
Contributor

@ruokeqx ruokeqx commented Sep 4, 2022

What type of PR is this?

fix: A bug fix

Check the PR title.

  • This PR title match the format: (optional scope):

  • The description of this PR title is user-oriented and clear enough for others to understand.

(Optional) Translate the PR title into Chinese.

修复目录穿越漏洞

(Optional) More detail description for this PR(en: English/zh: Chinese).

en: fix by simply replace all backslashes with forward slashes
zh(optional): 将反斜杠全部改成正斜杠

Which issue(s) this PR fixes:

Fixes #228

Currently hertz has no restriction on backslash in normalizePath function.

func normalizePath(dst, src []byte) []byte {
	dst = dst[:0]
	dst = addLeadingSlash(dst, src)
	dst = decodeArgAppendNoPlus(dst, src)

	// remove duplicate slashes
	b := dst
	bSize := len(b)
	for {
		n := bytes.Index(b, bytestr.StrSlashSlash)
		if n < 0 {
			break
		}
		b = b[n:]
		copy(b, b[1:])
		b = b[:len(b)-1]
		bSize--
	}
	dst = dst[:bSize]

	// remove /./ parts
	b = dst
	for {
		n := bytes.Index(b, bytestr.StrSlashDotSlash)
		if n < 0 {
			break
		}
		nn := n + len(bytestr.StrSlashDotSlash) - 1
		copy(b[n:], b[nn:])
		b = b[:len(b)-nn+n]
	}

	// remove /foo/../ parts
	for {
		n := bytes.Index(b, bytestr.StrSlashDotDotSlash)
		if n < 0 {
			break
		}
		nn := bytes.LastIndexByte(b[:n], '/')
		if nn < 0 {
			nn = 0
		}
		n += len(bytestr.StrSlashDotDotSlash) - 1
		copy(b[nn:], b[n:])
		b = b[:len(b)-n+nn]
	}

	// remove trailing /foo/..
	n := bytes.LastIndex(b, bytestr.StrSlashDotDot)
	if n >= 0 && n+len(bytestr.StrSlashDotDot) == len(b) {
		nn := bytes.LastIndexByte(b[:n], '/')
		if nn < 0 {
			return bytestr.StrSlash
		}
		b = b[:nn+1]
	}

	return b
}

After:

func normalizePath(dst, src []byte) []byte {
	// replace all backslashes with forward slashes
	for {
		n := bytes.Index(src, bytestr.StrBackSlash)
		if n < 0 {
			break
		}
		src[n] = '/'
	}
	...
}

@li-jin-gou
Copy link
Member

li-jin-gou commented Sep 4, 2022

@ruokeqx 按照 pr 模版来填写哈
参考 #223

@codecov
Copy link

codecov bot commented Sep 4, 2022

Codecov Report

Merging #229 (4ecd30e) into develop (8751608) will decrease coverage by 0.02%.
The diff coverage is 0.00%.

@@             Coverage Diff             @@
##           develop     #229      +/-   ##
===========================================
- Coverage    59.19%   59.17%   -0.03%     
===========================================
  Files           79       79              
  Lines         8105     8110       +5     
===========================================
+ Hits          4798     4799       +1     
- Misses        2953     2957       +4     
  Partials       354      354              
Impacted Files Coverage Δ
pkg/protocol/uri.go 70.34% <0.00%> (-0.72%) ⬇️
pkg/network/standard/buffer.go 81.08% <0.00%> (-0.50%) ⬇️

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

@ruokeqx
Copy link
Contributor Author

ruokeqx commented Sep 4, 2022

When checking other http framework source code, I find that fasthttp community fix the same bug six months ago.
Maybe we can borrow some code.

valyala/fasthttp#1226
valyala/fasthttp@6b5bc7b

@ruokeqx
Copy link
Contributor Author

ruokeqx commented Sep 4, 2022

When checking other http framework source code, I find that fasthttp community fix the same bug six months ago. Maybe we can borrow some code.

valyala/fasthttp#1226 valyala/fasthttp@6b5bc7b

They just repeat the logic, there is no need to write duplicate code.
The point is that backslash also work in windows. So, essentially, we just need to replace the backslash with forward slash, by which we, to some extent, limit the separator.
In my limited point of view, my fix is better.

@@ -480,6 +480,15 @@ func splitHostURI(host, uri []byte) ([]byte, []byte, []byte) {
}

func normalizePath(dst, src []byte) []byte {
// replace all backslashes with forward slashes
for {
n := bytes.Index(src, bytestr.StrBackSlash)
Copy link
Contributor

@a631807682 a631807682 Sep 5, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. Seems it will double check for bytes that have been replaced, is this an performance issue on long paths?
  2. I'm not sure it's a good idea to replace src in place, maybe it's hard to understand that if normalizePath has a return value and also modifies the input parameters.

Copy link
Contributor Author

@ruokeqx ruokeqx Sep 5, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. Seems it will double check for bytes that have been replaced, is this an performance issue on long paths?
  2. I'm not sure it's a good idea to replace src in place, maybe it's hard to understand that if normalizePath has a return value and also modifies the input parameters.

maybe this one would be better

func normalizePath(dst, src []byte) []byte {
	dst = dst[:0]
	dst = addLeadingSlash(dst, src)
	dst = decodeArgAppendNoPlus(dst, src)

	// replace all backslashes with forward slashes
	if filepath.Separator == '\\' {
		for i := range dst {
			if dst[i] == '\\' {
				dst[i] = '/'
			}
		}
	}
	...
}

BTW: echo use function filepath.ToSlash

https://github.com/labstack/echo/blob/master/context_fs.go#L33

func ToSlash(path string) string {
	if Separator == '/' {
		return path
	}
	return strings.ReplaceAll(path, string(Separator), "/")
}

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bytes.IndexByte will do a SIMD accelerate in some platforms. Not sure which one is better.

And since the bug only happens in Windows? Maybe add a if filepath.Separator == '\\' { to filter the platform?

@welkeyever
Copy link
Member

welkeyever commented Sep 5, 2022

The example/standard demo seems still have the problem in Windows if the url is : 127.0.0.1:8888/..%5c..%5cgo.sum

@ruokeqx
Copy link
Contributor Author

ruokeqx commented Sep 5, 2022

The example/standard demo seems still have the problem in Windows if the url is : 127.0.0.1:8888/..%5c..%5cgo.sum

this version solve the problem

func normalizePath(dst, src []byte) []byte {
	dst = dst[:0]
	dst = addLeadingSlash(dst, src)
	dst = decodeArgAppendNoPlus(dst, src)

	// replace all backslashes with forward slashes
	if filepath.Separator == '\\' {
		for i := range dst {
			if dst[i] == '\\' {
				dst[i] = '/'
			}
		}
	}
	...
}

before the if judgment
src = "/..%5c..%5cgo.sum"
dst = "//..\..\go.sum"

curl -v 127.0.0.1:8888/..%5c..%5cgo.sum
*   Trying 127.0.0.1:8888...
* Connected to 127.0.0.1 (127.0.0.1) port 8888 (#0)
> GET /..%5c..%5cgo.sum HTTP/1.1
> Host: 127.0.0.1:8888
> User-Agent: curl/7.83.1
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 404 404 Page not found
< Date: Mon, 05 Sep 2022 09:10:07 GMT
< Content-Type: text/plain; charset=utf-8
< Content-Length: 26
<
Cannot open requested path* Connection #0 to host 127.0.0.1 left intact

@welkeyever
Copy link
Member

The example/standard demo seems still have the problem in Windows if the url is : 127.0.0.1:8888/..%5c..%5cgo.sum

this version solve the problem

func normalizePath(dst, src []byte) []byte {
	dst = dst[:0]
	dst = addLeadingSlash(dst, src)
	dst = decodeArgAppendNoPlus(dst, src)

	// replace all backslashes with forward slashes
	if filepath.Separator == '\\' {
		for i := range dst {
			if dst[i] == '\\' {
				dst[i] = '/'
			}
		}
	}
	...
}

before the if judgment src = "/..%5c..%5cgo.sum" dst = "//....\go.sum"

curl -v 127.0.0.1:8888/..%5c..%5cgo.sum
*   Trying 127.0.0.1:8888...
* Connected to 127.0.0.1 (127.0.0.1) port 8888 (#0)
> GET /..%5c..%5cgo.sum HTTP/1.1
> Host: 127.0.0.1:8888
> User-Agent: curl/7.83.1
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 404 404 Page not found
< Date: Mon, 05 Sep 2022 09:10:07 GMT
< Content-Type: text/plain; charset=utf-8
< Content-Length: 26
<
Cannot open requested path* Connection #0 to host 127.0.0.1 left intact

Good job! As for the performance part, maybe do a bench test to see the answer.

@ruokeqx
Copy link
Contributor Author

ruokeqx commented Sep 5, 2022

The example/standard demo seems still have the problem in Windows if the url is : 127.0.0.1:8888/..%5c..%5cgo.sum

this version solve the problem

func normalizePath(dst, src []byte) []byte {
	dst = dst[:0]
	dst = addLeadingSlash(dst, src)
	dst = decodeArgAppendNoPlus(dst, src)

	// replace all backslashes with forward slashes
	if filepath.Separator == '\\' {
		for i := range dst {
			if dst[i] == '\\' {
				dst[i] = '/'
			}
		}
	}
	...
}

before the if judgment src = "/..%5c..%5cgo.sum" dst = "//....\go.sum"

curl -v 127.0.0.1:8888/..%5c..%5cgo.sum
*   Trying 127.0.0.1:8888...
* Connected to 127.0.0.1 (127.0.0.1) port 8888 (#0)
> GET /..%5c..%5cgo.sum HTTP/1.1
> Host: 127.0.0.1:8888
> User-Agent: curl/7.83.1
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 404 404 Page not found
< Date: Mon, 05 Sep 2022 09:10:07 GMT
< Content-Type: text/plain; charset=utf-8
< Content-Length: 26
<
Cannot open requested path* Connection #0 to host 127.0.0.1 left intact

Good job! As for the performance part, maybe do a bench test to see the answer.

I did some simple benchmarks, when there are backslashes in path, for-range option is faster, when there is no backslash in path, IndexByte option is faster.

Choose IndexByte maybe more reasonable since most normal URLs don't have backslashes in them.

var benchDstBackslashShort = []byte("/..\\foo")
var benchDstBackslashLong = []byte("/..\\..\\..\\..\\..\\..\\..\\..\\..\\..\\foo")
var benchDstNoBackslashShort = []byte("/../foo")
var benchDstNoBackslashLong = []byte("/../../../../../../../../../../foo")

func BenchmarkForrange(b *testing.B) {
	for i := 0; i < b.N; i++ {
		for i := range benchDstNoBackslashShort {
			if benchDstNoBackslashShort[i] == '\\' {
				benchDstNoBackslashShort[i] = '/'
			}
		}
		benchDstNoBackslashShort = []byte("/../foo")
	}
}

func BenchmarkIndexByte(b *testing.B) {
	for i := 0; i < b.N; i++ {
		for {
			n := bytes.IndexByte(benchDstNoBackslashShort, '\\')
			if n < 0 {
				break
			}
			benchDstNoBackslashShort[n] = '/'
		}
		benchDstNoBackslashShort = []byte("/../foo")
	}
}

environment

goos: windows
goarch: amd64
pkg: github.com/cloudwego/hertz/pkg/protocol
cpu: Intel(R) Core(TM) i7-9750H CPU @ 2.60GHz

benchDstBackslashShort

BenchmarkForrange-12     	57650732	        19.92 ns/op	       8 B/op	       1 allocs/op
BenchmarkIndexByte-12    	43823608	        27.97 ns/op	       8 B/op	       1 allocs/op
PASS
ok  	github.com/cloudwego/hertz/pkg/protocol	2.521s

benchDstBackslashLong

BenchmarkForrange-12     	21818498	        52.70 ns/op	      48 B/op	       1 allocs/op
BenchmarkIndexByte-12    	 9185786	       130.1 ns/op	      48 B/op	       1 allocs/op
PASS
ok  	github.com/cloudwego/hertz/pkg/protocol	2.941s

benchDstNoBackslashShort

BenchmarkForrange-12     	53754288	        19.66 ns/op	       8 B/op	       1 allocs/op
BenchmarkIndexByte-12    	61503766	        19.34 ns/op	       8 B/op	       1 allocs/op
PASS
ok  	github.com/cloudwego/hertz/pkg/protocol	2.379s

benchDstNoBackslashLong

BenchmarkForrange-12     	21096711	        49.30 ns/op	      48 B/op	       1 allocs/op
BenchmarkIndexByte-12    	32403181	        37.07 ns/op	      48 B/op	       1 allocs/op
PASS
ok  	github.com/cloudwego/hertz/pkg/protocol	2.724s

@welkeyever
Copy link
Member

PTAL. There may have a side effect which lead to a failure of existing UT

@ruokeqx
Copy link
Contributor Author

ruokeqx commented Sep 6, 2022

My fault. Since we add if filepath.Separator == '\\' {, backslashes were replaced when running UT in my windows laptop. However, linux would not replace them.
We can pass the the UT by modifying the code like below.

if filepath.Separator == '\\' && string(parsedPath) != expectedPath {
    t.Fatalf("Unexpected Path: %q. Expected %q", parsedPath, expectedPath)
}

Also, github action can not check windows path.

Copy link
Member

@welkeyever welkeyever left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@welkeyever welkeyever merged commit dcb0b5a into cloudwego:develop Sep 8, 2022
4 of 6 checks passed
@welkeyever welkeyever mentioned this pull request Jan 31, 2023
2 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging this pull request may close these issues.

hertz path-traversal bug in windows server
4 participants