Skip to content
Permalink
Browse files
[webkitscmpy] Generate Commit object from local repository
https://bugs.webkit.org/show_bug.cgi?id=216404
<rdar://problem/68702897>

Reviewed by Dewei Zhu.

* Scripts/libraries/webkitcorepy/webkitcorepy/__init__.py: Bump version.
* Scripts/libraries/webkitcorepy/webkitcorepy/decorators.py:
(Memoize.__call__.decorator): Handle case where function arguments are different.
* Scripts/libraries/webkitcorepy/webkitcorepy/mocks/popen.py:
(PopenBase.poll): Reset stdout and stderr after calling the completion handler.
* Scripts/libraries/webkitcorepy/webkitcorepy/tests/decorators_unittest.py:
(TestMemoize.test_conflicting_args):
* Scripts/libraries/webkitscmpy/webkitscmpy/__init__.py: Bump version.
* Scripts/libraries/webkitscmpy/webkitscmpy/contributor.py:
(Contributor): Number of lines may be singular.
* Scripts/libraries/webkitscmpy/webkitscmpy/local/git.py:
(Git):
(Git.__init__): Caller may wish to specify the patterns for production and development branches.
(Git.branches): Call the more general _branches_for to unify remote stripping logic.
(Git._commit_count): Given a parameter (hash, branch, difference between default branch and commit), compute
the number of commits.
(Git._branches_for): Given a hash, return all branches the commit is part of.
(Git.commit): Construct a commit given a commit hash, revision, identifier or branch.
* Scripts/libraries/webkitscmpy/webkitscmpy/local/scm.py:
(Scm):
(Scm.__init__): Caller may wish to specify the patterns for production and development branches.
(Scm.commit): Construct a commit given a commit hash, revision, identifier or branch.
(Scm.prioritize_branches): Given a set of branches, pick the highest priority one.
(Scm.log): Log error to a configured logger or stderr.
* Scripts/libraries/webkitscmpy/webkitscmpy/local/svn.py:
(Svn):
(Svn.__init__): Caller may wish to specify the patterns for production and development branches.
(Svn.info): Allow info for a specific commit to be queried.
(Svn._cache_path): Path to json cache of branch-commit mapping.
(Svn._cache_revisions): Query the remote for a specific branch to get the list of all commits on that branch.
(Svn._commit_count): Given a revision or branch, compute the number of commits.
(Svn._branch_for): Given a revision, use local data to determine the most likely branch before asking the remote which branch
a commit is on.
(Svn.commit): Construct a commit given a commit hash, revision, identifier or branch.
* Scripts/libraries/webkitscmpy/webkitscmpy/mocks/local/git.py:
(Git): Add commands needed to construct an identifier, add mock commits.
* Scripts/libraries/webkitscmpy/webkitscmpy/mocks/local/svn.py:
(Svn): Add commands needed to construct an identifier, add mock commits.
* Scripts/libraries/webkitscmpy/webkitscmpy/test/contributor_unittest.py:
(TestContributor.test_short_svn_log):
* Scripts/libraries/webkitscmpy/webkitscmpy/test/git_unittest.py:
(TestGit):
* Scripts/libraries/webkitscmpy/webkitscmpy/test/scm_unittest.py:
(TestScm.test_remote):
(TestScm):
* Scripts/libraries/webkitscmpy/webkitscmpy/test/svn_unittest.py:
(TestSvn):


Canonical link: https://commits.webkit.org/230169@main
git-svn-id: https://svn.webkit.org/repository/webkit/trunk@268080 268f45cc-cd09-0410-ab3c-d52691b4dbfc
  • Loading branch information
JonWBedard committed Oct 6, 2020
1 parent c48031c commit c7df73ed8c93fd002e742b384ffae2c5bedf753c
Show file tree
Hide file tree
Showing 17 changed files with 1,190 additions and 64 deletions.
@@ -1,3 +1,59 @@
2020-10-06 Jonathan Bedard <jbedard@apple.com>

[webkitscmpy] Generate Commit object from local repository
https://bugs.webkit.org/show_bug.cgi?id=216404
<rdar://problem/68702897>

Reviewed by Dewei Zhu.

* Scripts/libraries/webkitcorepy/webkitcorepy/__init__.py: Bump version.
* Scripts/libraries/webkitcorepy/webkitcorepy/decorators.py:
(Memoize.__call__.decorator): Handle case where function arguments are different.
* Scripts/libraries/webkitcorepy/webkitcorepy/mocks/popen.py:
(PopenBase.poll): Reset stdout and stderr after calling the completion handler.
* Scripts/libraries/webkitcorepy/webkitcorepy/tests/decorators_unittest.py:
(TestMemoize.test_conflicting_args):
* Scripts/libraries/webkitscmpy/webkitscmpy/__init__.py: Bump version.
* Scripts/libraries/webkitscmpy/webkitscmpy/contributor.py:
(Contributor): Number of lines may be singular.
* Scripts/libraries/webkitscmpy/webkitscmpy/local/git.py:
(Git):
(Git.__init__): Caller may wish to specify the patterns for production and development branches.
(Git.branches): Call the more general _branches_for to unify remote stripping logic.
(Git._commit_count): Given a parameter (hash, branch, difference between default branch and commit), compute
the number of commits.
(Git._branches_for): Given a hash, return all branches the commit is part of.
(Git.commit): Construct a commit given a commit hash, revision, identifier or branch.
* Scripts/libraries/webkitscmpy/webkitscmpy/local/scm.py:
(Scm):
(Scm.__init__): Caller may wish to specify the patterns for production and development branches.
(Scm.commit): Construct a commit given a commit hash, revision, identifier or branch.
(Scm.prioritize_branches): Given a set of branches, pick the highest priority one.
(Scm.log): Log error to a configured logger or stderr.
* Scripts/libraries/webkitscmpy/webkitscmpy/local/svn.py:
(Svn):
(Svn.__init__): Caller may wish to specify the patterns for production and development branches.
(Svn.info): Allow info for a specific commit to be queried.
(Svn._cache_path): Path to json cache of branch-commit mapping.
(Svn._cache_revisions): Query the remote for a specific branch to get the list of all commits on that branch.
(Svn._commit_count): Given a revision or branch, compute the number of commits.
(Svn._branch_for): Given a revision, use local data to determine the most likely branch before asking the remote which branch
a commit is on.
(Svn.commit): Construct a commit given a commit hash, revision, identifier or branch.
* Scripts/libraries/webkitscmpy/webkitscmpy/mocks/local/git.py:
(Git): Add commands needed to construct an identifier, add mock commits.
* Scripts/libraries/webkitscmpy/webkitscmpy/mocks/local/svn.py:
(Svn): Add commands needed to construct an identifier, add mock commits.
* Scripts/libraries/webkitscmpy/webkitscmpy/test/contributor_unittest.py:
(TestContributor.test_short_svn_log):
* Scripts/libraries/webkitscmpy/webkitscmpy/test/git_unittest.py:
(TestGit):
* Scripts/libraries/webkitscmpy/webkitscmpy/test/scm_unittest.py:
(TestScm.test_remote):
(TestScm):
* Scripts/libraries/webkitscmpy/webkitscmpy/test/svn_unittest.py:
(TestSvn):

2020-10-06 Aakash Jain <aakash_jain@apple.com>

[build.webkit.org] Ensure that steps names are valid Buildbot identifiers
@@ -35,7 +35,7 @@
from webkitcorepy.subprocess_utils import TimeoutExpired, CompletedProcess, run
from webkitcorepy.output_capture import LoggerCapture, OutputCapture, OutputDuplicate

version = Version(0, 4, 11)
version = Version(0, 4, 12)

from webkitcorepy.autoinstall import Package, AutoInstall
if sys.version_info > (3, 0):
@@ -42,8 +42,9 @@ def decorator(*args, **kwargs):
if 'cached' not in inspect.getargspec(function).args:
cached = kwargs.pop('cached', cached)

last_called = self._last_called[function].get(args, 0)
is_cached = args in self._cache[function]
keyargs = args + tuple(sorted([(key, value) for key, value in kwargs.items()]))
last_called = self._last_called[function].get(keyargs, 0)
is_cached = keyargs in self._cache[function]
if not cached:
is_cached = False
if timeout and timeout < time.time() - last_called:
@@ -52,8 +53,8 @@ def decorator(*args, **kwargs):
return self._cache[function].get(args, None)

value = function(*args, **kwargs)
self._last_called[function][args] = time.time()
self._cache[function][args] = value
self._last_called[function][keyargs] = time.time()
self._cache[function][keyargs] = value
return value

decorator.clear = self.clear
@@ -88,13 +88,14 @@ def poll(self):
string_utils.decode(self._completion.stderr, target_type=self._stderr_type))
(self.stderr or sys.stderr).flush()

if self.returncode is not None and time.time() >= self._start_time + self._completion.elapsed:
self.returncode = self._completion.returncode
if self.stdout:
self.stdout.seek(0)
if self.stderr:
self.stderr.seek(0)

if self.returncode is not None and time.time() >= self._start_time + self._completion.elapsed:
self.returncode = self._completion.returncode

return self.returncode

def send_signal(self, sig):
@@ -106,4 +106,4 @@ def test_conflicting_args(self):
self.assertEqual(TestMemoize.increment_with_args(
timeout='timeout-new',
cached='cached-new',
), ('timeout', 'cached'))
), ('timeout-new', 'cached-new'))
@@ -55,7 +55,7 @@ def readme():
'webkitscmpy.mocks.local',
'webkitscmpy.test',
],
install_requires=['webkitcorepy'],
install_requires=['python-dateutil', 'webkitcorepy'],
include_package_data=True,
zip_safe=False,
)
@@ -38,15 +38,17 @@ def _maybe_add_webkitcorepy_path():
_maybe_add_webkitcorepy_path()

try:
from webkitcorepy.version import Version
from webkitcorepy import AutoInstall, Package, Version
except ImportError:
raise ImportError(
"'webkitcorepy' could not be found on your Python path.\n" +
"You are not running from a WebKit checkout.\n" +
"Please install webkitcorepy with `pip install webkitcorepy --extra-index-url <package index URL>`"
)

version = Version(0, 0, 8)
version = Version(0, 1, 0)

AutoInstall.register(Package('dateutil', Version(2, 8, 1), pypi_name='python-dateutil'))

from webkitscmpy.contributor import Contributor
from webkitscmpy.commit import Commit
@@ -26,7 +26,7 @@
class Contributor(object):
GIT_AUTHOR_RE = re.compile(r'Author: (?P<author>.*) <(?P<email>[^@]+@[^@]+)(@.*)?>')
AUTOMATED_CHECKIN_RE = re.compile(r'Author: (?P<author>.*) <devnull>')
SVN_AUTHOR_RE = re.compile(r'r\d+ \| (?P<email>.*) \| (?P<date>.*) \| \d+ lines')
SVN_AUTHOR_RE = re.compile(r'r\d+ \| (?P<email>.*) \| (?P<date>.*) \| \d+ lines?')
SVN_PATCH_FROM_RE = re.compile(r'Patch by (?P<author>.*) <(?P<email>.*)> on \d+-\d+-\d+')

by_email = dict()
@@ -21,19 +21,24 @@
# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.


import re

from webkitcorepy import run, decorators, TimeoutExpired
from webkitscmpy.local import Scm
from webkitscmpy import Commit, Contributor


class Git(Scm):
executable = '/usr/bin/git'
GIT_COMMIT = re.compile(r'commit (?P<hash>[0-9a-f]+)')
GIT_SVN_REVISION = re.compile(r'git-svn-id: \S+:\/\/.+@(?P<revision>\d+) .+-.+-.+-.+')

@classmethod
def is_checkout(cls, path):
return run([cls.executable, 'rev-parse', '--show-toplevel'], cwd=path, capture_output=True).returncode == 0

def __init__(self, path):
super(Git, self).__init__(path)
def __init__(self, path, dev_branches=None, prod_branches=None):
super(Git, self).__init__(path, dev_branches=dev_branches, prod_branches=prod_branches)
if not self.root_path:
raise OSError('Provided path {} is not a git repository'.format(path))

@@ -100,11 +105,7 @@ def branch(self):

@property
def branches(self):
branch = run([self.executable, 'branch', '-a'], cwd=self.root_path, capture_output=True, encoding='utf-8')
if branch.returncode:
raise self.Exception('Failed to retrieve branch list for {}'.format(self.root_path))
result = [branch.lstrip(' *') for branch in filter(lambda branch: '->' not in branch, branch.stdout.splitlines())]
return ['/'.join(branch.split('/')[2:]) if branch.startswith('remotes/origin/') else branch for branch in result]
return self._branches_for()

@property
def tags(self):
@@ -118,3 +119,134 @@ def remote(self, name=None):
if result.returncode:
raise self.Exception('Failed to retrieve remote for {}'.format(self.root_path))
return result.stdout.rstrip()

def _commit_count(self, native_parameter):
revision_count = run(
[self.executable, 'rev-list', '--count', '--no-merges', native_parameter],
cwd=self.root_path, capture_output=True, encoding='utf-8',
)
if revision_count.returncode:
raise self.Exception('Failed to retrieve revision count for {}'.format(native_parameter))
return int(revision_count.stdout)

def _branches_for(self, hash=None):
branch = run(
[self.executable, 'branch', '-a'] + (['--contains', hash] if hash else []),
cwd=self.root_path,
capture_output=True,
encoding='utf-8',
)
if branch.returncode:
raise self.Exception('Failed to retrieve branch list for {}'.format(self.root_path))
result = [branch.lstrip(' *') for branch in filter(lambda branch: '->' not in branch, branch.stdout.splitlines())]
return sorted(set(['/'.join(branch.split('/')[2:]) if branch.startswith('remotes/origin/') else branch for branch in result]))

def commit(self, hash=None, revision=None, identifier=None, branch=None):
if revision and not self.is_svn:
raise self.Exception('This git checkout does not support SVN revisions')
elif revision:
revision = Commit._parse_revision(revision, do_assert=True)
revision_log = run(
[self.executable, 'svn', 'find-rev', 'r{}'.format(revision)],
cwd=self.root_path, capture_output=True, encoding='utf-8',
timeout=3,
)
if revision_log.returncode:
raise self.Exception("Failed to retrieve commit information for 'r{}'".format(revision))
hash = revision_log.stdout.rstrip()
if not hash:
raise self.Exception("Failed to find 'r{}'".format(revision))

default_branch = self.default_branch
parsed_branch_point = None

if identifier is not None:
if revision:
raise ValueError('Cannot define both revision and identifier')
if hash:
raise ValueError('Cannot define both hash and identifier')

parsed_branch_point, identifier, parsed_branch = Commit._parse_identifier(identifier, do_assert=True)
if parsed_branch:
if branch and branch != parsed_branch:
raise ValueError(
"Caller passed both 'branch' and 'identifier', but specified different branches ({} and {})".format(
branch, parsed_branch,
),
)
branch = parsed_branch

baseline = branch or 'HEAD'
is_default = baseline == default_branch
if baseline == 'HEAD':
is_default = default_branch in self._branches_for(baseline)

if is_default and parsed_branch_point:
raise self.Exception('Cannot provide a branch point for a commit on the default branch')

base_count = self._commit_count(baseline if is_default else '{}..{}'.format(default_branch, baseline))

if identifier > base_count:
raise self.Exception('Identifier {} cannot be found on the specified branch in the current checkout'.format(identifier))
log = run(
[self.executable, 'log', '{}~{}'.format(branch or 'HEAD', base_count - identifier), '-1'],
cwd=self.root_path,
capture_output=True,
encoding='utf-8',
)
if log.returncode:
raise self.Exception("Failed to retrieve commit information for 'i{}@{}'".format(identifier, branch or 'HEAD'))

# Negative identifiers are actually commits on the default branch, we will need to re-compute the identifier
if identifier < 0 and is_default:
raise self.Exception('Illegal negative identifier on the default branch')
if identifier < 0:
identifier = None

elif branch:
if hash:
raise ValueError('Cannot define both branch and hash')
log = run([self.executable, 'log', branch, '-1'], cwd=self.root_path, capture_output=True, encoding='utf-8')
if log.returncode:
raise self.Exception("Failed to retrieve commit information for '{}'".format(branch))

else:
hash = Commit._parse_hash(hash, do_assert=True)
log = run([self.executable, 'log', hash or 'HEAD', '-1'], cwd=self.root_path, capture_output=True, encoding='utf-8')
if log.returncode:
raise self.Exception("Failed to retrieve commit information for '{}'".format(hash or 'HEAD'))

match = self.GIT_COMMIT.match(log.stdout.splitlines()[0])
if not match:
raise self.Exception('Invalid commit hash in git log')
hash = match.group('hash')

branch = self.prioritize_branches(self._branches_for(hash))

if not identifier:
identifier = self._commit_count(hash if branch == default_branch else '{}..{}'.format(default_branch, hash))
branch_point = None if branch == default_branch else self._commit_count(hash)
if branch_point and parsed_branch_point and branch_point != parsed_branch_point:
raise ValueError("Provided 'branch_point' does not match branch point of specified branch")

if self.is_svn:
match = self.GIT_SVN_REVISION.match(log.stdout.splitlines()[-1].lstrip())
revision = int(match.group('revision')) if match else None

commit_time = run(
[self.executable, 'show', '-s', '--format=%ct', hash],
cwd=self.root_path, capture_output=True, encoding='utf-8',
)
if commit_time.returncode:
raise self.Exception('Failed to retrieve commit time for {}'.format(hash))

return Commit(
hash=hash,
revision=revision,
identifier=identifier,
branch_point=branch_point,
branch=branch,
timestamp=int(commit_time.stdout.lstrip()),
author=Contributor.from_scm_log(log.stdout.splitlines()[1]),
message='\n'.join(line[4:] for line in log.stdout.splitlines()[4:]),
)

0 comments on commit c7df73e

Please sign in to comment.