Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

it is not clear to be it was ever correct for BRO to be using GZIP -9 #614

Closed
initconf opened this issue Oct 2, 2019 · 3 comments

Comments

@initconf
Copy link

commented Oct 2, 2019

broctl log archiving SHOULD NOT BE USING gzip -9

default : zeekctl-config.sh:compresscmd="gzip -9"

should be please please changed to

zeekctl-config.sh:compresscmd="gzip"

Reference: https://changelog.complete.org/archives/931-how-to-think-about-compression-part-2

OR talk to PSB

@initconf

This comment has been minimized.

Copy link
Author

commented Oct 2, 2019

gzip -9 compresses file smaller but takes upto 60% more time to do that. Its highly unoptimal esp in times when logs are huge and storage is cheap.

some PBS numbers to convince you to Please remove -9

GZIP-DEFAULT vs DATA-BRO:
1092860102 - 1067044215 = 25815887 bytes larger
1092860102/1067044215 = 1.024 times larger

GZIP9 vs DATA-BRO
1067044259 - 1067044259 = 0bytes

Also note run time of 3min vs 5min: GZIP-DEFAULT takes 60% the time

and this is for 1G file - our files are generally 80-90Gb.

@jsiwek

This comment has been minimized.

Copy link
Member

commented Oct 2, 2019

Not sure the history behind it, but also a bit surprised that our default is to use highest compression level instead of just gzip's own default, so changing this sounds fine to me. See this as a low-priority (but easy) task for 3.1.0 since it's a simple option for users to change themselves in the meantime.

@jsiwek jsiwek added this to Unassigned / Todo in Release 3.1.0 via automation Oct 2, 2019
@jsiwek jsiwek added this to the 3.1.0 milestone Oct 2, 2019
@stevesmoot

This comment has been minimized.

Copy link

commented Oct 2, 2019

jcconnell added a commit to jcconnell/zeek that referenced this issue Oct 16, 2019
@0xxon 0xxon assigned 0xxon and unassigned 0xxon Oct 16, 2019
@0xxon 0xxon closed this in af76637 Oct 17, 2019
Release 3.1.0 automation moved this from Unassigned / Todo to Done Oct 17, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Release 3.1.0
  
Done
4 participants
You can’t perform that action at this time.