Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create .tar.gz package deterministically using external commands #12108

Closed
wants to merge 1 commit into from
Closed
Changes from all commits
Commits
File filter...
Filter file types
Jump to…
Jump to file
Failed to load files.

Always

Just for now

Create .tar.gz package deterministically

Use the system 'tar' binary instead of the 'tarfile' Python module.
Fixes #11981.
  • Loading branch information
tonyo committed Jul 2, 2016
commit f55342d138aa0b7f9e073af7208d7139efbd2815
@@ -13,7 +13,6 @@
import os.path as path
import shutil
import subprocess
import tarfile

from mach.registrar import Registrar
from datetime import datetime
@@ -183,9 +182,29 @@ def package(self, release=False, dev=False, android=None, debug=False, debugger=
tar_path = '/'.join(dir_to_package.split('/')[:-1]) + '/'
tar_path += datetime.utcnow().replace(microsecond=0).isoformat()
tar_path += "-servo-tech-demo.tar.gz"
with tarfile.open(tar_path, "w:gz") as tar:
# arcname is to add by relative rather than absolute path
tar.add(dir_to_package, arcname='servo/')

# Archive deterministically
# See https://reproducible-builds.org/docs/archives/
archive_root_dir = 'servo'
with cd(dir_to_package):

This comment has been minimized.

@aneeshusa

aneeshusa Jul 2, 2016

Member

I think it would be a good idea to hoist this into a separate function to create a tarball, deterministically. I think python/servo/command_base.py is a good place for this.

p1 = subprocess.Popen(['find', '.', '-print0'],
stdout=subprocess.PIPE)
p2 = subprocess.Popen(['sort', '-z'], env=dict(os.environ, LC_ALL='C'),
stdin=p1.stdout, stdout=subprocess.PIPE)
p3 = subprocess.Popen(['tar', '--null', '-T', '-', '--no-recursion',
'--mtime', '1970-01-01 00:00Z',

This comment has been minimized.

@aneeshusa

aneeshusa Jul 2, 2016

Member

Let's connect these into one string using = and use the @epoch form for the argument: '--mtime=@0'. I think this is more clear than the "magic" 1970 start of epoch date string.

'--transform', 's,^./,{}/,'.format(archive_root_dir),

This comment has been minimized.

@aneeshusa

aneeshusa Jul 3, 2016

Member

Let's also connect these into one string with =.

'--owner=root', '--group=root', '--numeric-owner',
'-cf', '-'],

This comment has been minimized.

@aneeshusa

aneeshusa Jul 3, 2016

Member

Do the '-f', '-' args mean write to stdout? I think the --to-stdout option would be more clear.

stdin=p2.stdout, stdout=subprocess.PIPE)
with open(tar_path, 'wb') as tar_fh:
archive_proc = subprocess.Popen(['gzip', '-n'],
stdin=p3.stdout, stdout=tar_fh)
p1.stdout.close()
p2.stdout.close()
p3.stdout.close()
archive_proc.communicate()

print("Packaged Servo into " + tar_path)

@Command('install',
ProTip! Use n and p to navigate between commits in a pull request.
You can’t perform that action at this time.