Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add gzip support to DC sync #140

Closed
5 tasks done
e-orz opened this issue Sep 6, 2017 · 3 comments
Closed
5 tasks done

Add gzip support to DC sync #140

e-orz opened this issue Sep 6, 2017 · 3 comments
Assignees
Projects

Comments

@e-orz
Copy link
Contributor

e-orz commented Sep 6, 2017

Accept-Encoding:gzip to:

  • tsvGetter(consume)
  • _out
  • _ow

Checks:

  • accept-encoding header forces the server to send gzip or just allows it? If it's not a must on all the places there should put a check that the response is gzipped and only then deflate it.
  • check akka-http library, maybe it will unzip it automatically.
@e-orz e-orz self-assigned this Sep 6, 2017
@e-orz e-orz added this to Backlog in CM-Well Sep 6, 2017
@YoniMataraso YoniMataraso moved this from Backlog to Focus Set in CM-Well Sep 12, 2017
@e-orz e-orz moved this from Focus Set to Doing in CM-Well Dec 5, 2017
@e-orz
Copy link
Contributor Author

e-orz commented Dec 27, 2017

What to check:

  1. on direct access: make sure that the content is actually gzipped (tcpdump and http headers). Also make sure in the logs that the tsvs are streamed from the beginning and not the whole bunch is got after some delay.
  2. with no direct (the intermediate proxy might remove the request header) access check the same as the above:
    1. with only HAproxy
    2. with HAproxy and F5
    3. with api-garden
  3. with Dc-Sync satandalone _ow should also be gzipped (the request)

@AnnaGrebnev
Copy link

@yanpom please talk to me before start testing, I have idea how to test it

@AnnaGrebnev AnnaGrebnev assigned Ivan-Shestakov and unassigned yanpom Jan 2, 2018
@Ivan-Shestakov
Copy link

Together with @e-orz we have captured tcpdump traffic on the machine running dc sync process, and observed that traffic was gzipped both for bulk-consume (get tsvs) and _out (get actual infoton data in chunks of 25).
For _ow (ingest to target) - it is not gzipped (as expected), since the dc sync process is local to environment and it would be a waste of cpu cycles.

DcSync standalone tool encodes this traffic as well - but it wasn't tested due to time constraints.

@Ivan-Shestakov Ivan-Shestakov moved this from Testing to Done in CM-Well Jan 4, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
CM-Well
  
Done
Development

No branches or pull requests

4 participants