Skip to content

Commit

Permalink
qubesbuilder.BuildLog: compress logs bigger than 10MB
Browse files Browse the repository at this point in the history
Github has a file size limit of 100MB. Most of the time it isn't an
issue, but for example gentoo template build log has over 200MB. Since
build-log repository is quite bit already, take a more aggressive
approach - compress with xz every log bigger than 10MB. In case of said
gentoo build log the effect is amazing: down to 8MB.
  • Loading branch information
marmarek committed Sep 19, 2020
1 parent 08f6029 commit 9dc43fa
Showing 1 changed file with 12 additions and 0 deletions.
12 changes: 12 additions & 0 deletions rpc-services/qubesbuilder.BuildLog
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ import string
import subprocess
from datetime import datetime

XZ_THRESHOLD = 10 * 2 ** 20

def sanitize_line(untrusted_line):
line = bytearray(untrusted_line)
Expand Down Expand Up @@ -88,17 +89,28 @@ while True:
file_name = '{}.{}'.format(file_name_base, try_no)

try:
# check both compressed and uncompressed files availability, continue
# only if both are unused at the same time
fd = os.open(file_name, os.O_CREAT | os.O_EXCL, 0o664)
fd2 = os.open(file_name + '.xz', os.O_CREAT | os.O_EXCL, 0o664)
except OSError as err:
if err.errno == errno.EEXIST:
try_no += 1
continue
raise

os.close(fd)
os.close(fd2)
break

shutil.move(tmp_log.name, file_name)
if os.path.getsize(file_name) > XZ_THRESHOLD:
subprocess.check_call(['xz', '--force', file_name],
stdout=subprocess.DEVNULL,
stdin=subprocess.DEVNULL)
file_name += '.xz'
else:
os.unlink(file_name + '.xz')

# report actually used file name to the build domain
print(os.path.relpath(
Expand Down

1 comment on commit 9dc43fa

@fepitre
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please sign in to comment.