forked from buildbot/buildbot
/
cfg-buildsteps.texinfo
2392 lines (1889 loc) · 90.5 KB
/
cfg-buildsteps.texinfo
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
@c TODO: go over this file to organize into classes
@c and be sure all index entries are in place
@c TODO:
@c @bsindex buildbot.steps.python_twisted.HLint
@c @bsindex buildbot.steps.python_twisted.Trial
@c @bsindex buildbot.steps.python_twisted.ProcessDocs
@c @bsindex buildbot.steps.python_twisted.BuildDebs
@c @bsindex buildbot.steps.python_twisted.RemovePYCs
@code{BuildStep}s are usually specified in the buildmaster's
configuration file, in a list that goes into the @code{BuildFactory}.
The @code{BuildStep} instances in this list are used as templates to
construct new independent copies for each build (so that state can be
kept on the @code{BuildStep} in one build without affecting a later
build). Each @code{BuildFactory} can be created with a list of steps,
or the factory can be created empty and then steps added to it using
the @code{addStep} method:
@example
from buildbot.steps import source, shell
from buildbot.process import factory
f = factory.BuildFactory()
f.addStep(source.SVN(svnurl="http://svn.example.org/Trunk/"))
f.addStep(shell.ShellCommand(command=["make", "all"]))
f.addStep(shell.ShellCommand(command=["make", "test"]))
@end example
The rest of this section lists all the standard BuildStep objects
available for use in a Build, and the parameters which can be used to
control each.
@menu
* Common Parameters::
* Using Build Properties::
* Source Checkout::
* ShellCommand::
* Python BuildSteps::
* Transferring Files::
* Steps That Run on the Master::
* Triggering Schedulers::
* Writing New BuildSteps::
* Build Step Index::
@end menu
@node Common Parameters
@subsection Common Parameters
All BuildSteps accept some common parameters. Some of these control
how their individual status affects the overall build. Others are used
to specify which @code{Locks} (see @pxref{Interlocks}) should be
acquired before allowing the step to run.
Arguments common to all @code{BuildStep} subclasses:
@table @code
@item name
the name used to describe the step on the status display. It is also
used to give a name to any LogFiles created by this step.
@item haltOnFailure
if True, a FAILURE of this build step will cause the build to halt
immediately. Steps with @code{alwaysRun=True} are still run. Generally
speaking, haltOnFailure implies flunkOnFailure (the default for most
BuildSteps). In some cases, particularly series of tests, it makes sense
to haltOnFailure if something fails early on but not flunkOnFailure.
This can be achieved with haltOnFailure=True, flunkOnFailure=False.
@item flunkOnWarnings
when True, a WARNINGS or FAILURE of this build step will mark the
overall build as FAILURE. The remaining steps will still be executed.
@item flunkOnFailure
when True, a FAILURE of this build step will mark the overall build as
a FAILURE. The remaining steps will still be executed.
@item warnOnWarnings
when True, a WARNINGS or FAILURE of this build step will mark the
overall build as having WARNINGS. The remaining steps will still be
executed.
@item warnOnFailure
when True, a FAILURE of this build step will mark the overall build as
having WARNINGS. The remaining steps will still be executed.
@item alwaysRun
if True, this build step will always be run, even if a previous buildstep
with @code{haltOnFailure=True} has failed.
@item doStepIf
A step can be configured to only run under certain conditions. To do this, set
the step's @code{doStepIf} to a boolean value, or to a function that returns a
boolean value. If the value or function result is false, then the step will
return SKIPPED without doing anything. Oherwise, the step will be executed
normally. If you set @code{doStepIf} to a function, that function should
accept one parameter, which will be the @code{Step} object itself.
@item locks
a list of Locks (instances of @code{buildbot.locks.SlaveLock} or
@code{buildbot.locks.MasterLock}) that should be acquired before starting this
Step. The Locks will be released when the step is complete. Note that this is a
list of actual Lock instances, not names. Also note that all Locks must have
unique names. See @ref{Interlocks}.
@end table
@node Using Build Properties
@subsection Using Build Properties
@cindex Properties
Build properties are a generalized way to provide configuration
information to build steps; see @ref{Build Properties}.
Some build properties are inherited from external sources -- global
properties, schedulers, or buildslaves. Some build properties are
set when the build starts, such as the SourceStamp information. Other
properties can be set by BuildSteps as they run, for example the
various Source steps will set the @code{got_revision} property to the
source revision that was actually checked out (which can be useful
when the SourceStamp in use merely requested the ``latest revision'':
@code{got_revision} will tell you what was actually built).
In custom BuildSteps, you can get and set the build properties with
the @code{getProperty}/@code{setProperty} methods. Each takes a string
for the name of the property, and returns or accepts an
arbitrary@footnote{Build properties are serialized along with the
build results, so they must be serializable. For this reason, the
value of any build property should be simple inert data: strings,
numbers, lists, tuples, and dictionaries. They should not contain
class instances.} object. For example:
@example
class MakeTarball(ShellCommand):
def start(self):
if self.getProperty("os") == "win":
self.setCommand([ ... ]) # windows-only command
else:
self.setCommand([ ... ]) # equivalent for other systems
ShellCommand.start(self)
@end example
@menu
* WithProperties::
@end menu
@node WithProperties
@subsubsection WithProperties
@cindex WithProperties
You can use build properties in ShellCommands by using the
@code{WithProperties} wrapper when setting the arguments of
the ShellCommand. This interpolates the named build properties
into the generated shell command. Most step parameters accept
@code{WithProperties}. Please file bugs for any parameters which
do not.
@example
from buildbot.steps.shell import ShellCommand
from buildbot.process.properties import WithProperties
f.addStep(ShellCommand(
command=["tar", "czf",
WithProperties("build-%s.tar.gz", "revision"),
"source"]))
@end example
If this BuildStep were used in a tree obtained from Subversion, it
would create a tarball with a name like @file{build-1234.tar.gz}.
The @code{WithProperties} function does @code{printf}-style string
interpolation, using strings obtained by calling
@code{build.getProperty(propname)}. Note that for every @code{%s} (or
@code{%d}, etc), you must have exactly one additional argument to
indicate which build property you want to insert.
You can also use python dictionary-style string interpolation by using
the @code{%(propname)s} syntax. In this form, the property name goes
in the parentheses, and WithProperties takes @emph{no} additional
arguments:
@example
f.addStep(ShellCommand(
command=["tar", "czf",
WithProperties("build-%(revision)s.tar.gz"),
"source"]))
@end example
Don't forget the extra ``s'' after the closing parenthesis! This is
the cause of many confusing errors.
The dictionary-style interpolation supports a number of more advanced
syntaxes, too.
@table @code
@item propname:-replacement
If @code{propname} exists, substitute its value; otherwise,
substitute @code{replacement}. @code{replacement} may be empty
(@code{%(propname:-)s})
@item propname:+replacement
If @code{propname} exists, substitute @code{replacement}; otherwise,
substitute an empty string.
@end table
Although these are similar to shell substitutions, no other
substitutions are currently supported, and @code{replacement} in the
above cannot contain more substitutions.
Note: like python, you can either do positional-argument interpolation
@emph{or} keyword-argument interpolation, not both. Thus you cannot use
a string like @code{WithProperties("foo-%(revision)s-%s", "branch")}.
@heading Common Build Properties
The following build properties are set when the build is started, and
are available to all steps.
@table @code
@item branch
This comes from the build's SourceStamp, and describes which branch is
being checked out. This will be @code{None} (which interpolates into
@code{WithProperties} as an empty string) if the build is on the
default branch, which is generally the trunk. Otherwise it will be a
string like ``branches/beta1.4''. The exact syntax depends upon the VC
system being used.
@item revision
This also comes from the SourceStamp, and is the revision of the source code
tree that was requested from the VC system. When a build is requested of a
specific revision (as is generally the case when the build is triggered by
Changes), this will contain the revision specification. This is always a
string, although the syntax depends upon the VC system in use: for SVN it is an
integer, for Mercurial it is a short string, for Darcs it is a rather large
string, etc.
If the ``force build'' button was pressed, the revision will be @code{None},
which means to use the most recent revision available. This is a ``trunk
build''. This will be interpolated as an empty string.
@item got_revision
This is set when a Source step checks out the source tree, and
provides the revision that was actually obtained from the VC system.
In general this should be the same as @code{revision}, except for
trunk builds, where @code{got_revision} indicates what revision was
current when the checkout was performed. This can be used to rebuild
the same source code later.
Note that for some VC systems (Darcs in particular), the revision is a
large string containing newlines, and is not suitable for interpolation
into a filename.
@item buildername
This is a string that indicates which Builder the build was a part of.
The combination of buildername and buildnumber uniquely identify a
build.
@item buildnumber
Each build gets a number, scoped to the Builder (so the first build
performed on any given Builder will have a build number of 0). This
integer property contains the build's number.
@item slavename
This is a string which identifies which buildslave the build is
running on.
@item scheduler
If the build was started from a scheduler, then this property will
contain the name of that scheduler.
@item repository
The repository of the sourcestamp for this build
@item project
The project of the sourcestamp for this build
@end table
@node Source Checkout
@subsection Source Checkout
The first step of any build is typically to acquire the source code
from which the build will be performed. There are several classes to
handle this, one for each of the different source control system that
Buildbot knows about. For a description of how Buildbot treats source
control in general, see @ref{Version Control Systems}.
All source checkout steps accept some common parameters to control how
they get the sources and where they should be placed. The remaining
per-VC-system parameters are mostly to specify where exactly the
sources are coming from.
@table @code
@item mode
a string describing the kind of VC operation that is desired. Defaults
to @code{update}.
@table @code
@item update
specifies that the CVS checkout/update should be performed directly
into the workdir. Each build is performed in the same directory,
allowing for incremental builds. This minimizes disk space, bandwidth,
and CPU time. However, it may encounter problems if the build process
does not handle dependencies properly (sometimes you must do a ``clean
build'' to make sure everything gets compiled), or if source files are
deleted but generated files can influence test behavior (e.g. python's
.pyc files), or when source directories are deleted but generated
files prevent CVS from removing them. Builds ought to be correct
regardless of whether they are done ``from scratch'' or incrementally,
but it is useful to test both kinds: this mode exercises the
incremental-build style.
@item copy
specifies that the CVS workspace should be maintained in a separate
directory (called the 'copydir'), using checkout or update as
necessary. For each build, a new workdir is created with a copy of the
source tree (rm -rf workdir; cp -r copydir workdir). This doubles the
disk space required, but keeps the bandwidth low (update instead of a
full checkout). A full 'clean' build is performed each time. This
avoids any generated-file build problems, but is still occasionally
vulnerable to CVS problems such as a repository being manually
rearranged, causing CVS errors on update which are not an issue with a
full checkout.
@c TODO: something is screwy about this, revisit. Is it the source
@c directory or the working directory that is deleted each time?
@item clobber
specifes that the working directory should be deleted each time,
necessitating a full checkout for each build. This insures a clean
build off a complete checkout, avoiding any of the problems described
above. This mode exercises the ``from-scratch'' build style.
@item export
this is like @code{clobber}, except that the 'cvs export' command is
used to create the working directory. This command removes all CVS
metadata files (the CVS/ directories) from the tree, which is
sometimes useful for creating source tarballs (to avoid including the
metadata in the tar file).
@end table
@item workdir
like all Steps, this indicates the directory where the build will take
place. Source Steps are special in that they perform some operations
outside of the workdir (like creating the workdir itself).
@item alwaysUseLatest
if True, bypass the usual ``update to the last Change'' behavior, and
always update to the latest changes instead.
@item retry
If set, this specifies a tuple of @code{(delay, repeats)} which means
that when a full VC checkout fails, it should be retried up to
@var{repeats} times, waiting @var{delay} seconds between attempts. If
you don't provide this, it defaults to @code{None}, which means VC
operations should not be retried. This is provided to make life easier
for buildslaves which are stuck behind poor network connections.
@item repository
The name of this parameter might varies depending on the Source step you
are running. The concept explained here is common to all steps and
applies to @code{repourl} as well as for @code{baseURL} (when
aplicable). Buildbot, now being aware of the repository name via the
ChangeSource step might in some cases not need the repository url. There
are multiple way to pass it through to this step, those correspond to
the type of the parameter given to this step:
@table @code
@item None
In the case where no paraneter is specified, the repository url will be
taken exactly from the Change property. You are looking for that one if
your ChangeSource step has all informations about how to reach the
Change.
@item string
The parameter might be a string, in this case, this string will be taken
as the repository url, and nothing more. the value coming from the
ChangeSource step will be forgotten.
@item format string
If the parameter is a string containing @code{%s}, then this the
repository property from the Change will be place in place of the
@code{%s}. This is usefull when the ChangeSource step knows where the
repository resides locally, but don't know the scheme used to access
it. For instance @code{ssh://server/%s} makes sense if the the
repository property is the local path of the repository.
@item dict
In this case, the repository URL will be the value indexed by the
repository property in the dict given as parameter.
@item callable
The callable given as parameter will take the repository property from
the Change and its return value will be used as repository URL.
@end table
@end table
Note that this is quite similar to the mechanism used by the WebStatus
for the @code{changecommentlink}, @code{projects} or @code{repositories} parameter.
My habit as a developer is to do a @code{cvs update} and @code{make} each
morning. Problems can occur, either because of bad code being checked in, or
by incomplete dependencies causing a partial rebuild to fail where a
complete from-scratch build might succeed. A quick Builder which emulates
this incremental-build behavior would use the @code{mode='update'}
setting.
On the other hand, other kinds of dependency problems can cause a clean
build to fail where a partial build might succeed. This frequently results
from a link step that depends upon an object file that was removed from a
later version of the tree: in the partial tree, the object file is still
around (even though the Makefiles no longer know how to create it).
``official'' builds (traceable builds performed from a known set of
source revisions) are always done as clean builds, to make sure it is
not influenced by any uncontrolled factors (like leftover files from a
previous build). A ``full'' Builder which behaves this way would want
to use the @code{mode='clobber'} setting.
Each VC system has a corresponding source checkout class: their
arguments are described on the following pages.
@menu
* CVS::
* SVN::
* Darcs::
* Mercurial::
* Arch::
* Bazaar::
* Bzr::
* P4::
* Git::
* BitKeeper::
@end menu
@node CVS
@subsubsection CVS
@cindex CVS Checkout
@bsindex buildbot.steps.source.CVS
The @code{CVS} build step performs a @uref{http://www.nongnu.org/cvs/,
CVS} checkout or update. It takes the following arguments:
@table @code
@item cvsroot
(required): specify the CVSROOT value, which points to a CVS
repository, probably on a remote machine. For example, the cvsroot
value you would use to get a copy of the Buildbot source code is
@code{:pserver:anonymous@@cvs.sourceforge.net:/cvsroot/buildbot}
@item cvsmodule
(required): specify the cvs @code{module}, which is generally a
subdirectory of the CVSROOT. The cvsmodule for the Buildbot source
code is @code{buildbot}.
@item branch
a string which will be used in a @code{-r} argument. This is most
useful for specifying a branch to work on. Defaults to @code{HEAD}.
@item global_options
a list of flags to be put before the verb in the CVS command.
@item checkoutDelay
if set, the number of seconds to put between the timestamp of the last
known Change and the value used for the @code{-D} option. Defaults to
half of the parent Build's treeStableTimer.
@end table
@node SVN
@subsubsection SVN
@cindex SVN Checkout
@bsindex buildbot.steps.source.SVN
The @code{SVN} build step performs a
@uref{http://subversion.tigris.org, Subversion} checkout or update.
There are two basic ways of setting up the checkout step, depending
upon whether you are using multiple branches or not.
If all of your builds use the same branch, then you should create the
@code{SVN} step with the @code{svnurl} argument:
@table @code
@item svnurl
(required): this specifies the @code{URL} argument that will be given
to the @code{svn checkout} command. It dictates both where the
repository is located and which sub-tree should be extracted. In this
respect, it is like a combination of the CVS @code{cvsroot} and
@code{cvsmodule} arguments. For example, if you are using a remote
Subversion repository which is accessible through HTTP at a URL of
@code{http://svn.example.com/repos}, and you wanted to check out the
@code{trunk/calc} sub-tree, you would use
@code{svnurl="http://svn.example.com/repos/trunk/calc"} as an argument
to your @code{SVN} step.
@end table
If, on the other hand, you are building from multiple branches, then
you should create the @code{SVN} step with the @code{baseURL} and
@code{defaultBranch} arguments instead:
@table @code
@item baseURL
(required): this specifies the base repository URL, to which a branch
name will be appended. It should probably end in a slash.
@item defaultBranch
(optional): this specifies the name of the branch to use when a Build
does not provide one of its own. This will be appended to
@code{baseURL} to create the string that will be passed to the
@code{svn checkout} command.
@item username
(optional): if specified, this will be passed to the @code{svn}
binary with a @code{--username} option.
@item password
(optional): if specified, this will be passed to the @code{svn}
binary with a @code{--password} option. The password itself will be
suitably obfuscated in the logs.
@item extra_args
(optional): if specified, an array of strings that will be passed as
extra arguments to the @code{svn} binary.
@item keep_on_purge
(optional): specific files or directories to keep between purges,
like some build outputs that can be reused between builds.
@item ignore_ignores
(optional): when purging changes, don't use rules defined in
svn:ignore properties and global-ignores in subversion/config.
@item always_purge
(optional): if set to True, always purge local changes before updating. This deletes unversioned files and reverts everything that would appear in a @code{svn status}.
@item depth
(optional): Specify depth argument to achieve sparse checkout. Only available if slave has Subversion 1.5 or higher.
If set to "empty" updates will not pull in any files or subdirectories not already present. If set to "files", updates will pull in any files not already present, but not directories. If set to "immediates", updates willl pull in any files or subdirectories not already present, the new subdirectories will have depth: empty. If set to "infinity", updates will pull in any files or subdirectories not already present; the new subdirectories will have depth-infinity. Infinity is equivalent to SVN default update behavior, without specifying any depth argument.
@end table
If you are using branches, you must also make sure your
@code{ChangeSource} will report the correct branch names.
@heading branch example
Let's suppose that the ``MyProject'' repository uses branches for the
trunk, for various users' individual development efforts, and for
several new features that will require some amount of work (involving
multiple developers) before they are ready to merge onto the trunk.
Such a repository might be organized as follows:
@example
svn://svn.example.org/MyProject/trunk
svn://svn.example.org/MyProject/branches/User1/foo
svn://svn.example.org/MyProject/branches/User1/bar
svn://svn.example.org/MyProject/branches/User2/baz
svn://svn.example.org/MyProject/features/newthing
svn://svn.example.org/MyProject/features/otherthing
@end example
Further assume that we want the Buildbot to run tests against the
trunk and against all the feature branches (i.e., do a
checkout/compile/build of branch X when a file has been changed on
branch X, when X is in the set [trunk, features/newthing,
features/otherthing]). We do not want the Buildbot to automatically
build any of the user branches, but it should be willing to build a
user branch when explicitly requested (most likely by the user who
owns that branch).
There are three things that need to be set up to accomodate this
system. The first is a ChangeSource that is capable of identifying the
branch which owns any given file. This depends upon a user-supplied
function, in an external program that runs in the SVN commit hook and
connects to the buildmaster's @code{PBChangeSource} over a TCP
connection. (you can use the ``@code{buildbot sendchange}'' utility
for this purpose, but you will still need an external program to
decide what value should be passed to the @code{--branch=} argument).
For example, a change to a file with the SVN url of
``svn://svn.example.org/MyProject/features/newthing/src/foo.c'' should
be broken down into a Change instance with
@code{branch='features/newthing'} and @code{file='src/foo.c'}.
The second piece is an @code{AnyBranchScheduler} which will pay
attention to the desired branches. It will not pay attention to the
user branches, so it will not automatically start builds in response
to changes there. The AnyBranchScheduler class requires you to
explicitly list all the branches you want it to use, but it would not
be difficult to write a subclass which used
@code{branch.startswith('features/'} to remove the need for this
explicit list. Or, if you want to build user branches too, you can use
AnyBranchScheduler with @code{branches=None} to indicate that you want
it to pay attention to all branches.
The third piece is an @code{SVN} checkout step that is configured to
handle the branches correctly, with a @code{baseURL} value that
matches the way the ChangeSource splits each file's URL into base,
branch, and file.
@example
from buildbot.changes.pb import PBChangeSource
from buildbot.scheduler import AnyBranchScheduler
from buildbot.process import source, factory
from buildbot.steps import source, shell
c['change_source'] = PBChangeSource()
s1 = AnyBranchScheduler('main',
['trunk', 'features/newthing', 'features/otherthing'],
10*60, ['test-i386', 'test-ppc'])
c['schedulers'] = [s1]
f = factory.BuildFactory()
f.addStep(source.SVN(mode='update',
baseURL='svn://svn.example.org/MyProject/',
defaultBranch='trunk'))
f.addStep(shell.Compile(command="make all"))
f.addStep(shell.Test(command="make test"))
c['builders'] = [
@{'name':'test-i386', 'slavename':'bot-i386', 'builddir':'test-i386',
'factory':f @},
@{'name':'test-ppc', 'slavename':'bot-ppc', 'builddir':'test-ppc',
'factory':f @},
]
@end example
In this example, when a change arrives with a @code{branch} attribute
of ``trunk'', the resulting build will have an SVN step that
concatenates ``svn://svn.example.org/MyProject/'' (the baseURL) with
``trunk'' (the branch name) to get the correct svn command. If the
``newthing'' branch has a change to ``src/foo.c'', then the SVN step
will concatenate ``svn://svn.example.org/MyProject/'' with
``features/newthing'' to get the svnurl for checkout.
@node Darcs
@subsubsection Darcs
@cindex Darcs Checkout
@bsindex buildbot.steps.source.Darcs
The @code{Darcs} build step performs a
@uref{http://darcs.net/, Darcs} checkout or update.
Like @xref{SVN}, this step can either be configured to always check
out a specific tree, or set up to pull from a particular branch that
gets specified separately for each build. Also like SVN, the
repository URL given to Darcs is created by concatenating a
@code{baseURL} with the branch name, and if no particular branch is
requested, it uses a @code{defaultBranch}. The only difference in
usage is that each potential Darcs repository URL must point to a
fully-fledged repository, whereas SVN URLs usually point to sub-trees
of the main Subversion repository. In other words, doing an SVN
checkout of @code{baseURL} is legal, but silly, since you'd probably
wind up with a copy of every single branch in the whole repository.
Doing a Darcs checkout of @code{baseURL} is just plain wrong, since
the parent directory of a collection of Darcs repositories is not
itself a valid repository.
The Darcs step takes the following arguments:
@table @code
@item repourl
(required unless @code{baseURL} is provided): the URL at which the
Darcs source repository is available.
@item baseURL
(required unless @code{repourl} is provided): the base repository URL,
to which a branch name will be appended. It should probably end in a
slash.
@item defaultBranch
(allowed if and only if @code{baseURL} is provided): this specifies
the name of the branch to use when a Build does not provide one of its
own. This will be appended to @code{baseURL} to create the string that
will be passed to the @code{darcs get} command.
@end table
@node Mercurial
@subsubsection Mercurial
@cindex Mercurial Checkout
@bsindex buildbot.steps.source.Mercurial
The @code{Mercurial} build step performs a
@uref{http://selenic.com/mercurial, Mercurial} (aka ``hg'') checkout
or update.
Branches are available in two modes: ''dirname'' like @xref{Darcs}, or
''inrepo'', which uses the repository internal branches. Make sure this
setting matches your changehook, if you have that installed.
The Mercurial step takes the following arguments:
@table @code
@item repourl
(required unless @code{baseURL} is provided): the URL at which the
Mercurial source repository is available.
@item baseURL
(required unless @code{repourl} is provided): the base repository URL,
to which a branch name will be appended. It should probably end in a
slash.
@item defaultBranch
(allowed if and only if @code{baseURL} is provided): this specifies
the name of the branch to use when a Build does not provide one of its
own. This will be appended to @code{baseURL} to create the string that
will be passed to the @code{hg clone} command.
@item branchType
either 'dirname' (default) or 'inrepo' depending on whether
the branch name should be appended to the @code{baseURL}
or the branch is a mercurial named branch and can be
found within the @code{repourl}.
@item clobberOnBranchChange
boolean, defaults to True. If set and
using inrepos branches, clobber the tree
at each branch change. Otherwise, just
update to the branch.
@end table
@node Arch
@subsubsection Arch
@cindex Arch Checkout
@bsindex buildbot.steps.source.Arch
The @code{Arch} build step performs an @uref{http://gnuarch.org/,
Arch} checkout or update using the @code{tla} client. It takes the
following arguments:
@table @code
@item url
(required): this specifies the URL at which the Arch source archive is
available.
@item version
(required): this specifies which ``development line'' (like a branch)
should be used. This provides the default branch name, but individual
builds may specify a different one.
@item archive
(optional): Each repository knows its own archive name. If this
parameter is provided, it must match the repository's archive name.
The parameter is accepted for compatibility with the @code{Bazaar}
step, below.
@end table
@node Bazaar
@subsubsection Bazaar
@cindex Bazaar Checkout
@bsindex buildbot.steps.source.Bazaar
@code{Bazaar} is an alternate implementation of the Arch VC system,
which uses a client named @code{baz}. The checkout semantics are just
different enough from @code{tla} that there is a separate BuildStep for
it.
It takes exactly the same arguments as @code{Arch}, except that the
@code{archive=} parameter is required. (baz does not emit the archive
name when you do @code{baz register-archive}, so we must provide it
ourselves).
@node Bzr
@subsubsection Bzr
@cindex Bzr Checkout
@bsindex buildbot.steps.source.Bzr
@code{bzr} is a descendant of Arch/Baz, and is frequently referred to
as simply ``Bazaar''. The repository-vs-workspace model is similar to
Darcs, but it uses a strictly linear sequence of revisions (one
history per branch) like Arch. Branches are put in subdirectories.
This makes it look very much like Mercurial. It takes the following
arguments:
@table @code
@item repourl
(required unless @code{baseURL} is provided): the URL at which the
Bzr source repository is available.
@item baseURL
(required unless @code{repourl} is provided): the base repository URL,
to which a branch name will be appended. It should probably end in a
slash.
@item defaultBranch
(allowed if and only if @code{baseURL} is provided): this specifies
the name of the branch to use when a Build does not provide one of its
own. This will be appended to @code{baseURL} to create the string that
will be passed to the @code{bzr checkout} command.
@item forceSharedRepo
(boolean, optional, defaults to False): If set to True, the working directory
will be made into a bzr shared repository if it is not already. Shared
repository greatly reduces the amount of history data that needs to be
downloaded if not using update/copy mode, or if using update/copy mode with
multiple branches.
@end table
@node P4
@subsubsection P4
@cindex Perforce Update
@bsindex buildbot.steps.source.P4
@c TODO @bsindex buildbot.steps.source.P4Sync
The @code{P4} build step creates a @uref{http://www.perforce.com/,
Perforce} client specification and performs an update.
@table @code
@item p4base
A view into the Perforce depot without branch name or trailing "...".
Typically "//depot/proj/".
@item defaultBranch
A branch name to append on build requests if none is specified.
Typically "trunk".
@item p4port
(optional): the host:port string describing how to get to the P4 Depot
(repository), used as the -p argument for all p4 commands.
@item p4user
(optional): the Perforce user, used as the -u argument to all p4
commands.
@item p4passwd
(optional): the Perforce password, used as the -p argument to all p4
commands.
@item p4extra_views
(optional): a list of (depotpath, clientpath) tuples containing extra
views to be mapped into the client specification. Both will have
"/..." appended automatically. The client name and source directory
will be prepended to the client path.
@item p4client
(optional): The name of the client to use. In mode='copy' and
mode='update', it's particularly important that a unique name is used
for each checkout directory to avoid incorrect synchronization. For
this reason, Python percent substitution will be performed on this value
to replace %(slave)s with the slave name and %(builder)s with the
builder name. The default is "buildbot_%(slave)s_%(build)s".
@end table
@node Git
@subsubsection Git
@cindex Git Checkout
@bsindex buildbot.steps.source.Git
The @code{Git} build step clones or updates a @uref{http://git.or.cz/,
Git} repository and checks out the specified branch or revision. Note
that the buildbot supports Git version 1.2.0 and later: earlier
versions (such as the one shipped in Ubuntu 'Dapper') do not support
the @command{git init} command that the buildbot uses.
The Git step takes the following arguments:
@table @code
@item repourl
(required): the URL of the upstream Git repository.
@item branch
(optional): this specifies the name of the branch to use when a Build
does not provide one of its own. If this this parameter is not
specified, and the Build does not provide a branch, the ``master''
branch will be used.
@item ignore_ignores
(optional): when purging changes, don't use .gitignore and
.git/info/exclude.
@item submodules
(optional): when initializing/updating a Git repository, this decides whether
or not buildbot should consider git submodules. Default: False.
@item shallow
(optional): instructs git to attempt shallow clones (@code{--depth 1}). If the
user/scheduler asks for a specific revision, this parameter is ignored.
@end table
@node BitKeeper
@subsubsection BitKeeper
@cindex BitKeeper Checkout
@bsindex buildbot.steps.source.BK
The @code{BK} build step performs a @uref{http://www.bitkeeper.com/, BitKeeper}
checkout or update.
The BitKeeper step takes the following arguments:
@table @code
@item repourl
(required unless @code{baseURL} is provided): the URL at which the
BitKeeper source repository is available.
@item baseURL
(required unless @code{repourl} is provided): the base repository URL,
to which a branch name will be appended. It should probably end in a
slash.
@end table
@node ShellCommand
@subsection ShellCommand
Most interesting steps involve exectuing a process of some sort on the
buildslave. The @code{ShellCommand} class handles this activity.
Several subclasses of ShellCommand are provided as starting points for
common build steps.
@menu
* Using ShellCommands::
* Configure::
* Compile::
* Visual C++::
* Test::
* TreeSize::
* PerlModuleTest::
* Testing with mysql-test-run::
* SetProperty::
* SubunitShellCommand::
@end menu
@node Using ShellCommands
@subsubsection Using ShellCommands
@bsindex buildbot.steps.shell.ShellCommand
This is a useful base class for just about everything you might want
to do during a build (except for the initial source checkout). It runs
a single command in a child shell on the buildslave. All stdout/stderr
is recorded into a LogFile. The step finishes with a status of FAILURE
if the command's exit code is non-zero, otherwise it has a status of
SUCCESS.
The preferred way to specify the command is with a list of argv strings,
since this allows for spaces in filenames and avoids doing any fragile
shell-escaping. You can also specify the command with a single string, in
which case the string is given to '/bin/sh -c COMMAND' for parsing.
On Windows, commands are run via @code{cmd.exe /c} which works well. However,
if you're running a batch file, the error level does not get propagated
correctly unless you add 'call' before your batch file's name:
@code{cmd=['call', 'myfile.bat', ...]}.
@code{ShellCommand} arguments:
@table @code
@item command
a list of strings (preferred) or single string (discouraged) which
specifies the command to be run. A list of strings is preferred
because it can be used directly as an argv array. Using a single
string (with embedded spaces) requires the buildslave to pass the
string to /bin/sh for interpretation, which raises all sorts of
difficult questions about how to escape or interpret shell
metacharacters.
@item workdir
All ShellCommands are run by default in the ``workdir'', which
defaults to the ``@file{build}'' subdirectory of the slave builder's
base directory. The absolute path of the workdir will thus be the
slave's basedir (set as an option to @code{buildbot create-slave},
@pxref{Creating a buildslave}) plus the builder's basedir (set in the
builder's @code{c['builddir']} key in master.cfg) plus the workdir
itself (a class-level attribute of the BuildFactory, defaults to