Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

compress/flate: add support for deflate64 compression #33023

Closed
alex opened this issue Jul 10, 2019 · 6 comments
Closed

compress/flate: add support for deflate64 compression #33023

alex opened this issue Jul 10, 2019 · 6 comments

Comments

@alex
Copy link
Contributor

@alex alex commented Jul 10, 2019

What version of Go are you using (go version)?

$ go version
go version go1.12.5 linux/amd64

Does this issue reproduce with the latest release?

Yup

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/alex/.cache/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/alex/go"
GOPROXY=""
GORACE=""
GOROOT="/usr/local/go"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="clang"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build508250989=/tmp/go-build -gno-record-gcc-switches"

What did you do?

Attempt to use archive/zip.NewReader() to open a ZIP archive containing files compressed with deflate64.

What did you expect to see?

The archive is opened as seamlessly as if files were compressed with standard deflate.

What did you see instead?

zip: unsupported compression algorithm

I should note that deflate64 seems to be exceptionally poorly supported in the open source world, as a result of the scarcity of documentation from PKWARE (the originators of this algorithm), so I'd be understanding if Go also was not interested in supporting it. With that said, I'd find it exceptionally convenient if Go had native support for deflate64.

Wikipedia has a brief description of how deflate64 differs from deflate: https://en.wikipedia.org/wiki/DEFLATE#Deflate64/Enhanced_Deflate

@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Jul 10, 2019

CC @dsnet

@alex

This comment has been minimized.

Copy link
Contributor Author

@alex alex commented Jul 10, 2019

To amend my comment on the scarcity of open source implementations, it turns out zlib contains an implementation in their contrib directory: https://github.com/madler/zlib/blob/master/contrib/infback9/

@dsnet

This comment has been minimized.

Copy link
Member

@dsnet dsnet commented Jul 10, 2019

This is issue actually less about archive/zip and more about compress/flate since most of the logic would be there to support this.

However, I'm inclined to say that compress/flate should not support Deflate64 since it's a proprietary extension of RFC1951, and to my knowledge, not well-specified.

Given the fact that archive/zip already provides the means to register your own custom compressors, what's wrong with using a third party implementation of Deflate64 outside of standard library?

@alex

This comment has been minimized.

Copy link
Contributor Author

@alex alex commented Jul 10, 2019

You're right, I should have filed this against compress/flate.

Yes, it could absolutely live outside of the standard library, unfortunately right now there is no deflate64 implementation for Go, so it's not simply a matter of wiring one up to archive/zip. My argument for having it in the standard library are: a) the possibility that it could share code with compress/flate, b) it'd give archive/zip broader compatibility the standard tools like unzip(1), c) and this is the least persuasive -- the underspecification of the algorithm puts a premium on rigorous testing and other techniques for having confidence, being in the standard library is generally a good marker of such quality.

@dsnet

This comment has been minimized.

Copy link
Member

@dsnet dsnet commented Jul 10, 2019

To be honest, I'm not sure deflate64 meets the bar for inclusion the standard library. The shared code argument alone is not sufficient. See https://golang.org/doc/faq#x_in_std

@dsnet dsnet changed the title archive/zip: Add support for deflate64 compression compress/flate: add support for deflate64 compression Jul 10, 2019
@dsnet dsnet added the NeedsDecision label Jul 10, 2019
@dsnet dsnet added the FeatureRequest label Aug 3, 2019
@ALTree

This comment has been minimized.

Copy link
Member

@ALTree ALTree commented Jan 28, 2020

It seems like there isn't a strong argument for deflate64 inclusion in the standard library, considering it's 1) proprietary and underspecified 2) not commonly used; also the owner of the flate package seems unconvinced, and no one else has spoken up, so I think we can close this "needsDecision" issue as WONTFIX.

@ALTree ALTree closed this Jan 28, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
4 participants
You can’t perform that action at this time.