Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

net: /etc/hosts parsing truncates to 4KB (os.GetPagesize) #21674

Closed
alexzorin opened this issue Aug 28, 2017 · 5 comments

Comments

Projects
None yet
4 participants
@alexzorin
Copy link

commented Aug 28, 2017

Due to the way net/file,open is implemented, pure-Go DNS resolution cannot handle /etc/hosts files that are larger than the OS page size, which is 4KB by default on Linux platforms.

Go silently truncates the data after 4KB and does not parse lines on the threshold, since EOL is never reached for them.

The cgo equivalents for this functionality handle these files properly, and so should Go.

We often see these large hosts files in the web hosting environment (e.g. cPanel), when servers are behind NAT and require large hosts files.

Env: Go1.9, linux/amd64

I have also attached a Dockerfile tarball that reproduces the issue fully - go-4kb-hosts.tar.gz

@odeke-em

This comment has been minimized.

Copy link
Member

commented Aug 29, 2017

Interesting! Thank you @alexzorin for reporting this issue. Just for easier lookup in the future, here is the permalink to the code you referred to

go/src/net/parse.go

Lines 67 to 72 in 93471a1

func open(name string) (*file, error) {
fd, err := os.Open(name)
if err != nil {
return nil, err
}
return &file{fd, make([]byte, 0, os.Getpagesize()), false}, nil
.

@odeke-em odeke-em added this to the Go1.10 milestone Aug 29, 2017

@odeke-em

This comment has been minimized.

Copy link
Member

commented Aug 29, 2017

@rsc I suppose that the seemingly arbitrary os.Getpagesize() limit was set because we wanted a performance hit from reads on page boundaries or that we thought that size was sufficient? I don't see a file limit for a hosts file in either of the below:

@gopherbot

This comment has been minimized.

Copy link

commented Sep 24, 2017

Change https://golang.org/cl/65671 mentions this issue: net: parse file lines longer than one page

@alexzorin

This comment has been minimized.

Copy link
Author

commented Nov 17, 2017

Hello,

I've been tracking the CL and I'm not sure how to add my input, sorry if I'm doing it wrong.

I saw a comment by @rsc that worries me a little:

Or at least I'm OK with not seeing lines > 4kB

I originally ran into this issue with customers exactly because of entries on specific lines that exceeded 4kB (big NAT-ted cPanel servers). If the system resolver deals with it without a problem I would not expect Go to introduce such a surprising and arbitrary limitation here.

I apologise if the original issue made it look like the only concern was the remainder of the file - properly parsing long lines is equally important to my application.

@gopherbot

This comment has been minimized.

Copy link

commented Nov 22, 2017

Change https://golang.org/cl/79516 mentions this issue: net: accept 64 kB lines in /etc/hosts

@gopherbot gopherbot closed this in 13f45d0 Nov 28, 2017

@golang golang locked and limited conversation to collaborators Nov 28, 2018

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
You can’t perform that action at this time.