Implement HTTP Output Compression #4807

mihaitodor · 2018-10-04T01:51:22Z

Fixes #4472.

TODO:

Should there be some refactoring done to avoid duplicating compressWithGzip here, since it's already present in influxdb and in influxdb_v2?
Would you be interested in caching gzip writers similar to here for better performance?
Do you really want to support "deflate" compression besides "gzip"?
If "br" stands for "brotli", it's not supported natively yet: golang support google/brotli#182 Do you want to bring in a 3rd party package for this if support for it is required?

Required for all PRs:

Signed CLA.
Associated README.md updated.
Has appropriate unit tests.

glinton · 2018-10-04T18:53:41Z

I think that's a great idea, though obviously a bit more involved
Also likely worth implementing some form of writer caching
Personally, I'm fine with just gzip
I don't think it's worth adding, but I think having some generic compressor/writer pool that can be extended to add deflate/br sometime in the future wouldn't be a bad idea.

I'm curious to know, and would wait to find out, what @danielnelson has to say on the matter though.

danielnelson · 2018-10-04T19:40:34Z

We could move the shared code under internal, it sounds like it would be a good idea.

I would rather avoid a pool for gzip.Writer. If it is a performance bottleneck, it may make sense to reuse the Writer between calls to Write, within a plugin instance, but I would want to see some benchmarks showing the gzip.Writer as a significant portion of write time first.

On brotli, I don't think it has enough adoption to warrant, I'm just hearing of it for the first time now.

mihaitodor · 2018-10-05T00:46:47Z

@glinton @danielnelson Thanks for answering my questions! I have implemented 1 in a53e591.

Studying this CompressWithGzip function in more detail, I think its design is not safe. io.Copy(gzipWriter, data) is executed in the background, which means that the internal goroutine will be still running when CompressWithGzip returns, so the caller will never know if io.Copy(gzipWriter, data) returns an error eventually. Do you have any insight into why it was designed this way? I'd probably consider returning an error channel to the caller instead and have it wait for the channel to receive the error. WDYT?

Regarding 2, I'd rather leave the pool for gzip.Writer for a future PR, if you're happy with the current performance. I'm not too keen to spend time on coming up with benchmarks for this right now.

Regarding 3 and 4, I asked because #4472 asks explicitly for "gzip, deflate, br, etc", so wasn't sure what your expectations are. Glad to hear that you're OK with having only gzip. I think it should be enough for most situations.

danielnelson

The setting of the error in CompressWithGzip is definitely wrong, but I don't think an error channel would be the right approach here as we really want any reader errors passed though the Pipe, this way the user of the pipe (http package in this case) gets the error. This would give consistent behavior with other errors in the Reader.

I'm not immediately sure how to implement this, but we shouldn't return the error from CompressWithGzip since we can't detect an error at this point. Also, fixing the error handling is optional for this PR, since it wouldn't be a step backwards.

danielnelson · 2018-10-05T17:43:39Z

plugins/outputs/http/http.go

+	var reqBodyBuffer io.Reader = bytes.NewBuffer(reqBody)
+	var err error
+
+	if h.ContentEncoding == "gzip" && h.Method == http.MethodPost {


Why only for POST method?

That's what was requested in #4472. It will add a bit of overhead to requests which don't have a body, but I wouldn't normally consider such small optimisations. Maybe @madisonleavo can offer additional insight. If not, I can delete that condition. Just let me know.

All of the requests currently have a body in this plugin, and it is hard to imagine a new data format that wouldn't produce it, perhaps a null data format? At any rate we currently do POST and PUT and I think they should both use the encoding if it is requested.

Cool! Done in 26ee478.

danielnelson · 2018-10-05T17:53:00Z

plugins/outputs/http/README.md

+
+  ## HTTP Content-Encoding for write request body, can be set to "gzip" to
+  ## compress body or empty string to not apply any encoding.
+  # content_encoding = ""


This is minor but can you document like the influxdb option with the identity setting as default, no code needs to change:

## HTTP Content-Encoding for write request body, can be set to "gzip" to ## compress body or "identity" to apply no encoding. # content_encoding = "identity"

I left it out on purpose as requested in #4472. Changed it in c5502a0.

mihaitodor · 2018-10-05T21:34:28Z

@danielnelson

The setting of the error in CompressWithGzip is definitely wrong, but I don't think an error channel would be the right approach here as we really want any reader errors passed though the Pipe, this way the user of the pipe (http package in this case) gets the error. This would give consistent behavior with other errors in the Reader.

Indeed, you are right! I got carried away thinking that it would be enough to return an error in the write method of the HTTP output plugin.

How about calling CloseWithError on the PipeWriter? I can simply pass into it the error returned by io.Copy, because:

subsequent reads from the read half of the pipe will return no bytes and the error err, or EOF if err is nil.

Also, fixing the error handling is optional for this PR, since it wouldn't be a step backwards.

I agree. Will open a new issue.

danielnelson · 2018-10-05T21:35:37Z

That sounds perfect

mihaitodor · 2018-10-05T21:47:09Z

@danielnelson OK, I believe I have addressed all the required changes in this PR. Please let me know if I missed anything.

danielnelson · 2018-10-05T21:51:28Z

Did you do the CloseWithError idea, I'm not seeing it?

Propagate io.Copy errors through the io.Pipe

mihaitodor · 2018-10-05T21:55:58Z

Oops, communication issue. I thought you preferred to have a new PR for it. Added it in 11cdf77.

danielnelson · 2018-10-05T22:08:28Z

Thanks!

glinton added hacktoberfest feat Improvement on an existing feature such as adding a new setting/mode to an existing plugin labels Oct 4, 2018

mihaitodor force-pushed the http-output-compression branch 2 times, most recently from 8954aff to a53e591 Compare October 5, 2018 00:34

glinton approved these changes Oct 5, 2018

View reviewed changes

danielnelson suggested changes Oct 5, 2018

View reviewed changes

mihaitodor force-pushed the http-output-compression branch from a53e591 to 4a49fee Compare October 5, 2018 21:26

mihaitodor added 2 commits October 5, 2018 22:39

Implement HTTP Output Compression

26ee478

Extract CompressWithGzip into the internal package

1e0a10a

mihaitodor force-pushed the http-output-compression branch from 4a49fee to 1e0a10a Compare October 5, 2018 21:43

Fix bug in CompressWithGzip

11cdf77

Propagate io.Copy errors through the io.Pipe

danielnelson approved these changes Oct 5, 2018

View reviewed changes

danielnelson merged commit f3da717 into influxdata:master Oct 5, 2018

danielnelson added this to the 1.9.0 milestone Oct 5, 2018

rgitzel pushed a commit to rgitzel/telegraf that referenced this pull request Oct 17, 2018

Add entity-body compression to http output (influxdata#4807)

3ee6548

otherpirate pushed a commit to otherpirate/telegraf that referenced this pull request Mar 15, 2019

Add entity-body compression to http output (influxdata#4807)

c1057d3

otherpirate pushed a commit to otherpirate/telegraf that referenced this pull request Mar 15, 2019

Add entity-body compression to http output (influxdata#4807)

237776c

dupondje pushed a commit to dupondje/telegraf that referenced this pull request Apr 22, 2019

Add entity-body compression to http output (influxdata#4807)

8b6ad06

athoune pushed a commit to bearstech/telegraf that referenced this pull request Apr 17, 2020

Add entity-body compression to http output (influxdata#4807)

5fef4e3

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement HTTP Output Compression #4807

Implement HTTP Output Compression #4807

mihaitodor commented Oct 4, 2018 •

edited

Loading

glinton commented Oct 4, 2018

danielnelson commented Oct 4, 2018

mihaitodor commented Oct 5, 2018 •

edited

Loading

danielnelson left a comment

danielnelson Oct 5, 2018

mihaitodor Oct 5, 2018 •

edited

Loading

danielnelson Oct 5, 2018

mihaitodor Oct 5, 2018

danielnelson Oct 5, 2018

mihaitodor Oct 5, 2018

mihaitodor commented Oct 5, 2018 •

edited

Loading

danielnelson commented Oct 5, 2018

mihaitodor commented Oct 5, 2018

danielnelson commented Oct 5, 2018

mihaitodor commented Oct 5, 2018 •

edited

Loading

danielnelson commented Oct 5, 2018

Implement HTTP Output Compression #4807

Implement HTTP Output Compression #4807

Conversation

mihaitodor commented Oct 4, 2018 • edited Loading

Required for all PRs:

glinton commented Oct 4, 2018

danielnelson commented Oct 4, 2018

mihaitodor commented Oct 5, 2018 • edited Loading

danielnelson left a comment

Choose a reason for hiding this comment

danielnelson Oct 5, 2018

Choose a reason for hiding this comment

mihaitodor Oct 5, 2018 • edited Loading

Choose a reason for hiding this comment

danielnelson Oct 5, 2018

Choose a reason for hiding this comment

mihaitodor Oct 5, 2018

Choose a reason for hiding this comment

danielnelson Oct 5, 2018

Choose a reason for hiding this comment

mihaitodor Oct 5, 2018

Choose a reason for hiding this comment

mihaitodor commented Oct 5, 2018 • edited Loading

danielnelson commented Oct 5, 2018

mihaitodor commented Oct 5, 2018

danielnelson commented Oct 5, 2018

mihaitodor commented Oct 5, 2018 • edited Loading

danielnelson commented Oct 5, 2018

mihaitodor commented Oct 4, 2018 •

edited

Loading

mihaitodor commented Oct 5, 2018 •

edited

Loading

mihaitodor Oct 5, 2018 •

edited

Loading

mihaitodor commented Oct 5, 2018 •

edited

Loading

mihaitodor commented Oct 5, 2018 •

edited

Loading