SVN completed #179

Merged
merged 1 commit into from Jul 27, 2011

Projects

None yet

3 participants

@in3xes

Please take closer look than usual, especially purge and url stuff.

@djmitche djmitche commented on an outdated diff Jul 20, 2011
master/docs/cfg-buildsteps.texinfo
+ @{'name':'test-ppc', 'slavename':'bot-ppc', 'builddir':'test-ppc',
+ 'factory':f @},
+ ]
+@end example
+
+In this example, when a change arrives with a @code{branch} attribute of
+``trunk'', the resulting build will have a SVN step that concatenates
+``svn://svn.example.org/MyProject/'' (the baseURL) with ``trunk'' (the branch
+name) to get the correct svn command. If the ``newthing'' branch has a change
+to ``src/foo.c'', then the SVN step will concatenate
+``svn://svn.example.org/MyProject/'' with ``features/newthing'' to get the
+svnurl for checkout.
+
+For added flexibility, @code{baseURL} may contain a @code{%%BRANCH%%}
+placeholder, which will be replaced either by the branch in the SourceStamp or
+the default specified in @code{defaultBranch}.
@djmitche
djmitche Jul 20, 2011

This should be described in the @item for baseURL, above!

@djmitche djmitche commented on an outdated diff Jul 20, 2011
master/docs/cfg-buildsteps.texinfo
+@item depth
+(optional): Specify depth argument to achieve sparse checkout. Only available if slave has Subversion 1.5 or higher.
+
+If set to "empty" updates will not pull in any files or subdirectories not already present. If set to "files", updates will pull in any files not already present, but not directories. If set to "immediates", updates will pull in any files or subdirectories not already present, the new subdirectories will have depth: empty. If set to "infinity", updates will pull in any files or subdirectories not already present; the new subdirectories will have depth-infinity. Infinity is equivalent to SVN default update behavior, without specifying any depth argument.
+
+@item mode
+@item method
+
+SVN's incremental mode does not require a method. The full mode has
+four methods defined:
+
+@table @code
+@item clobber
+It removes the working directory for each build then makes full checkout.
+@item fresh
+This is alternative for incremental (old update) mode with @code{always_purge}
@djmitche
djmitche Jul 20, 2011

Better phrased as "This is equivalent to the old update mode with @code{always_purge}", and only mentioned as the last sentence in this @item. That is:

This always always purges local changes before updating. This deletes
unversioned files and reverts everything that would appear in a @code{svn status
--no-ignore}. This is equivalent to the old update mode with @code{always_purge}.

@djmitche djmitche commented on the diff Jul 20, 2011
master/buildbot/test/unit/test_steps_source_svn.py
+ command=['svn', 'update', '--non-interactive',
+ '--no-auth-cache'])
+ + 0,
+ ExpectLogged('cpdir', {'fromdir': 'source',
+ 'todir': 'build'})
+ + 0,
+ ExpectShell(workdir='build',
+ command=['svnversion'])
+ + ExpectShell.log('stdio',
+ stdout='100')
+ + 0,
+ )
+ self.expectOutcome(result=SUCCESS, status_text=["update"])
+ return self.runStep()
+
+class TestGetUnversionedFiles(unittest.TestCase):
@djmitche
djmitche Jul 20, 2011

I'd like to see some more tests here - invalid XML, an XML document with no entries, an entry with no wc-status, a status with no item, and a wc-status with no path, at least -- just to see how those failures are handled.

@djmitche djmitche and 1 other commented on an outdated diff Jul 20, 2011
master/buildbot/steps/source/svn.py
+ return c
+
+ @staticmethod
+ def getUnversionedFiles(xmlStr, keep_on_purge):
+ """Delete everything that shown up on status."""
+ result_xml = parseString(xmlStr)
+ for entry in result_xml.getElementsByTagName('entry'):
+ (wc_status,) = entry.getElementsByTagName('wc-status')
+ if wc_status.getAttribute('item') == 'external':
+ continue
+ if wc_status.getAttribute('item') == 'missing':
+ continue
+ filename = entry.getAttribute('path')
+ if filename in keep_on_purge:
+ continue
+ yield filename
@djmitche
djmitche Jul 20, 2011

Nice use of a generator :)

@in3xes
in3xes Jul 22, 2011

I copied that from slaveside ;)

@djmitche djmitche commented on an outdated diff Jul 20, 2011
master/buildbot/steps/source/svn.py
+ cmd.useLog(self.stdio_log, False)
+ d = self.runCommand(cmd)
+ d.addCallback(lambda _: cmd.rc)
+ return d
+ c.addCallback(parseAndRemove)
+ def evaluateCommand(rc):
+ if rc != 0:
+ log.msg("Failed removing files")
+ raise failure.Failure(rc)
+ return rc
+ c.addCallback(evaluateCommand)
+ return c
+
+ @staticmethod
+ def getUnversionedFiles(xmlStr, keep_on_purge):
+ """Delete everything that shown up on status."""
@djmitche
djmitche Jul 20, 2011

This comment isn't quite right.. this method doesn't delete anything!

@djmitche djmitche commented on the diff Jul 20, 2011
master/buildbot/steps/source/svn.py
+ else:
+ cmd = buildstep.LoggedRemoteCommand('rmdir', {'dir': files})
+ cmd.useLog(self.stdio_log, False)
+ d = self.runCommand(cmd)
+ d.addCallback(lambda _: cmd.rc)
+ return d
+ c.addCallback(parseAndRemove)
+ def evaluateCommand(rc):
+ if rc != 0:
+ log.msg("Failed removing files")
+ raise failure.Failure(rc)
+ return rc
+ c.addCallback(evaluateCommand)
+ return c
+
+ @staticmethod
@in3xes
in3xes Jul 22, 2011

Only for tests, no other reason.

@djmitche djmitche commented on the diff Jul 20, 2011
master/buildbot/steps/source/svn.py
+ raise
+
+ log.msg("Got SVN revision %s" % (revision, ))
+ self.setProperty('got_revision', revision, 'Source')
+ return res
+ d.addCallback(lambda _: _setrev(cmd.rc))
+ return d
+
+ def purge(self, ignore_ignores):
+ command = ['status', '--xml']
+ if ignore_ignores:
+ command.append('--no-ignore')
+ c = self._dovccmd(command)
+ def parseAndRemove(_):
+ output = self.getLog('stdio').getText()
+ output = output[output.find('<'):]
@djmitche
djmitche Jul 20, 2011

I'm not sure what this is about - finding the first tag? Why?

@in3xes
in3xes Jul 22, 2011

When I do self.getLog('stdio').getText() it gives the output previous commands concatenated to one after other. So, I am find first '<' to find the starting of the xml data. Any other way?

@djmitche djmitche commented on an outdated diff Jul 20, 2011
master/buildbot/steps/source/svn.py
+ depth=depth,
+ )
+
+ assert self.mode in ['incremental', 'full']
+ assert self.method in ['clean', 'fresh', 'clobber', 'copy', None]
+
+ if svnurl and baseURL:
+ raise ValueError("you must provide exactly one of svnurl and"
+ " baseURL")
+
+ if svnurl is None and baseURL is None:
+ raise ValueError("you must privide at least one of svnurl and"
+ " baseURL")
+
+ self.svnurl = self.svnurl and _ComputeRepositoryURL(self.svnurl)
+ self.baseURL = self.baseURL and _ComputeRepositoryURL(self.baseURL)
@djmitche
djmitche Jul 20, 2011

Since you submitted this pull req, @tomprince has removed the _ComputeRepositoryURL invocations from new source steps, so please remove them here, too.

@djmitche djmitche commented on an outdated diff Jul 20, 2011
master/buildbot/steps/source/svn.py
+ @param username: username to pass to svn's --username
+
+ @type password: string
+ @param password: password to pass to svn's --password
+ """
+
+ self.svnurl = svnurl
+ self.baseURL = baseURL
+ self.branch = defaultBranch
+ self.username = username
+ self.password = password
+ self.extra_args = extra_args
+ self.keep_on_purge = keep_on_purge or []
+ self.depth = depth
+ self.method=method
+ Source.__init__(self, **kwargs)
@djmitche
djmitche Jul 20, 2011

This is an unusual place to call the parent constructor. It should either be called before the other attributes are set, or after all of them are set.

@djmitche djmitche commented on an outdated diff Jul 20, 2011
master/buildbot/steps/source/svn.py
+from buildbot.interfaces import BuildSlaveTooOldError
+
+
+class SVN(Source):
+ """I perform Subversion checkout/update operations."""
+
+ name = 'svn'
+ branch_placeholder = '%BRANCH%'
+
+ renderables = [ 'svnurl', 'baseURL' ]
+
+ def __init__(self, svnurl=None, baseURL=None, mode='incremental',
+ method=None, defaultBranch=None, username=None,
+ password=None, extra_args=None, keep_on_purge=None,
+ depth=None, **kwargs):
+ """
@djmitche
djmitche Jul 20, 2011

This docstring duplicates information available in the manual, and in fact doesn't even cover all of the constructor arguments. I think it should be removed.

@djmitche
Buildbot member

A few points:

baseURL vs svnurl

These names are confusing, and as I understand it, are just a vestige of backward compatibility. Since this step requires that users choose it explicitly, we have a chance to do it right. What do these two parameters accomplish, and how could we do that with one parameter, with lots of flexibility for the user? What about the common case when the repository in the SourceStamp is sufficient - do we need to make the user specify a URL to the SVN step in that case, too? You don't need to implement the full flexibility yet, but think ahead when deciding how to handle these URLs.

Tests on master

I pulled this onto master, and while it merged smoothly, the tests did not pass after the merge. This is probably "bitrot" after it took me so long to review - sorry.. Can you take a look?

@tomprince
Buildbot member
@in3xes

Updated, will fix URL stuff later.

@djmitche
Buildbot member

Merged (finally), yay!

@djmitche djmitche merged commit a31ef36 into buildbot:master Jul 27, 2011
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment