New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: Support pgzip as an alternative (de)compression implementation #5108
Conversation
"k8s.io/utils/env" | ||
) | ||
|
||
var gzipImpl = env.GetString(GZipImplEnvVarKey, PGZIP) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I did not know about this library - neat
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yea it's super fast and works great for large workflows.
util/file/fileutil.go
Outdated
@@ -63,23 +73,52 @@ func DecodeDecompressString(content string) (string, error) { | |||
return string(dBuf), nil | |||
} | |||
|
|||
type GZipWriter interface { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
isn't this the same interface as io.Writer
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's not since io.Writer
does not implement Close()
. However I just found io.WriteCloser
which is exactly what I needed. Just switched.
Signed-off-by: terrytangyuan <terrytangyuan@gmail.com>
Signed-off-by: terrytangyuan <terrytangyuan@gmail.com>
Signed-off-by: terrytangyuan <terrytangyuan@gmail.com>
util/file/fileutil.go
Outdated
case GZIP: | ||
gzipReader, err = gzip.NewReader(buf) | ||
default: | ||
log.Infof("%s implementation for decompression is not supported. Fallback to PGZip.", gzipImpl) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
remove this logging - not needed
util/file/fileutil.go
Outdated
gzipReader, err = pgzip.NewReader(buf) | ||
} | ||
if err != nil { | ||
log.Warnf("Error in decompressing: %v", err) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
remove this logging
util/file/fileutil.go
Outdated
} | ||
if err != nil { | ||
log.Warnf("Error in decompressing: %v", err) | ||
return nil, err |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
fmt.Errorf("failed to decompress: %w", err)
util/file/fileutil.go
Outdated
case GZIP: | ||
gzipWriter = gzip.NewWriter(&buf) | ||
default: | ||
log.Infof("%s implementation for compression is not supported. Fallback to PGZip.", gzipImpl) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
remove this logging
Signed-off-by: terrytangyuan <terrytangyuan@gmail.com>
For relatively large workflows, using pgzip to parallelize the (de)compression would greatly speed up the (de)hydration process, especially when the size overhead during compression can be neglected.
Signed-off-by: terrytangyuan terrytangyuan@gmail.com
Checklist: