go/printer, cmd/gofmt: panic on large Go file of length >= 1<<30 #30121
Comments
|
I can reproduce on Here is the file after truncating all lines to ~120 bytes, so that we can see what kind of file it is without trying to load 1.1GB: https://play.golang.org/p/i6HApVeq41v I can't spot an obvious bug or cause, so I guess this is just That first length gets close to the maximum value of an @tsenart This panic aside, I'd generally suggest to refrain from humongous Go source files, and particularly those that have most of the bytes in a single line. Even if gofmt was fixed, I'm sure other tools could also have issues with these extremely large line lengths. And of course, If you really do want to include the file as part of the binary, perhaps you could use an assets code generator which set a limit on line lengths, such as 8MiB per line. |
mvdan
added
the
NeedsInvestigation
|
/cc @griesemer |
Thanks for investigating so quickly and for the feedback. It makes total sense.
Indeed this is an assets file. Do you know of any asset generator project that limits line lengths to 8MiB? |
|
Found the cause, and it is indeed a kind of overflow; the longest line is I can see a few possible short-term solutions:
For now, I'll mail 1. I personally believe that 2 should also be done, since
As it turns out, it's the overall length of the file that causes this panic, and not line length. As long as you have position offsets (from the file's first byte) which are close to |
Actually, |
|
(coworker of @tsenart here) FWIW I definitely don't expect most Go tooling (and probably not even the compiler) to support 1GB+ Go source files, that was definitely a mistake / bug on our end. |
|
Thanks everyone for the reports, responses and root cause on this bug!
@mvdan, it might be tricky in deed! Perhaps we could negotiate with the compiler team about I think given that we are this close to the end of the Go1.13 cycle, but also as I believe I've seen the issue about very large source files causing problems, perhaps we should document limits on parsing source files. Maybe later we can revisit using |
odeke-em
changed the title
|
I think the simplest course of action would be for If the issue comes up again in the future, we can always consider better ways of dealing with large files, like doubling the limit or using 64-bit integers. |
|
I'm not surprised to see problems with 1GB files - it goes to show that no matter how relaxed a restriction is, at some point somebody will run into the restriction. I think for now the best course of action might be to error for files over a certain size, per @mvdan's suggestion. This may have to happen when the |
What version of Go are you using (
go version)?Does this issue reproduce with the latest release?
Yes.
What operating system and processor architecture are you using (
go env)?go envOutputWhat did you do?
What did you expect to see?
Not a panic.
What did you see instead?
A panic.
The text was updated successfully, but these errors were encountered: