Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add ReadAllLimitSize to io/ioutil #28788

Closed
fwhezfwhez opened this issue Nov 14, 2018 · 5 comments

Comments

@fwhezfwhez
Copy link

@fwhezfwhez fwhezfwhez commented Nov 14, 2018

What do I do?

Add new function for io/ioutil package.

What does it work for?

Read a reader with a size limit.When using ioutil.ReadAll(), I'm afaid the reader contains too much to fullfill the memory.

What is it?

func ReadAllLimitSize(r io.Reader, maxSize int64) ([]byte, error) {
	// when maxSize <= 0, read all
	if maxSize <= 0 {
		return ioutil.ReadAll(r)
	}
	const (
		B  = 1
		KB = 1024
		MB = 1024 * 1024
		GB = 1024 * 1024 * 1024
	)
	result := make([]byte, 0, 512)
	step,n :=0,0
	buf := make([]byte,512)
	var er error
	for {
		step ,er =r.Read(buf)
		n += step
		if int64(n) >= maxSize {
			return nil, errors.New(fmt.Sprintf("read  %d,but limit %d", n, maxSize))
		}

		result = append(result, buf[:step]...)
		if er!=nil && er == io.EOF{
			break
		}
		if er !=nil {
			return nil,er
		}
	}
	return result, nil
}

Testcase

func TestReadAllLimitSize(t *testing.T) {
	buf,e :=ReadAllLimitSize(strings.NewReader("hello world"), 122)
	if e!=nil {
		fmt.Println(e.Error())
		t.Fail()
	}
	fmt.Println(string(buf))
}

// 5000000	       214 ns/op
func BenchmarkReadAllLimitSize(b *testing.B) {
	r := strings.NewReader("benchmark for readAllLimitSize")
	const maxSize = int64(100)
	var buf []byte
	var e error
	for i:=0;i<b.N;i++{
		buf,e =ReadAllLimitSize(r, maxSize)
		if e!=nil{
			b.Fatal(e.Error())
		}
	}
	fmt.Println(string(buf))
}
@agnivade

This comment has been minimized.

Copy link
Contributor

@agnivade agnivade commented Nov 14, 2018

I believe you are looking for - https://golang.org/pkg/io/#LimitReader

According to your code, it should be ioutil.ReadAll((io.LimitReader(r, 100)).

I am going to close this as I see that the functionality is already present in the standard library. Adding another function which does the exact same thing will lead to unnecessary duplication. Please feel free to comment if you still want it, or had something else in mind.

Thank you.

@agnivade agnivade closed this Nov 14, 2018
@fwhezfwhez

This comment has been minimized.

Copy link
Author

@fwhezfwhez fwhezfwhez commented Nov 14, 2018

3q

@fwhezfwhez

This comment has been minimized.

Copy link
Author

@fwhezfwhez fwhezfwhez commented Nov 14, 2018

@agnivade hello, ioutil.ReadAll((io.LimitReader(r, 100)) only read the first 100 byte throwing no error.How can I make an error to warn the caller.

@mvdan

This comment has been minimized.

Copy link
Member

@mvdan mvdan commented Nov 14, 2018

@fwhezfwhez for questions about Go, see https://golang.org/wiki/Questions.

@fwhezfwhez

This comment has been minimized.

Copy link
Author

@fwhezfwhez fwhezfwhez commented Nov 15, 2018

well

@golang golang locked and limited conversation to collaborators Nov 15, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
4 participants
You can’t perform that action at this time.