forked from buildbot/buildbot
/
buildbot.texinfo
9513 lines (7592 loc) · 370 KB
/
buildbot.texinfo
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
\input texinfo @c -*-texinfo-*-
@c %**start of header
@setfilename buildbot.info
@settitle BuildBot Manual 0.7.12
@defcodeindex cs
@defcodeindex sl
@defcodeindex bf
@defcodeindex bs
@defcodeindex st
@defcodeindex bc
@c %**end of header
@c these indices are for classes useful in a master.cfg config file
@c @csindex : Change Sources
@c @slindex : Schedulers and Locks
@c @bfindex : Build Factories
@c @bsindex : Build Steps
@c @stindex : Status Targets
@c @bcindex : keys that make up BuildmasterConfig
@copying
This is the BuildBot manual.
Copyright (C) 2005, 2006, 2009, 2010 Brian Warner
Copying and distribution of this file, with or without
modification, are permitted in any medium without royalty
provided the copyright notice and this notice are preserved.
@end copying
@titlepage
@title BuildBot
@page
@vskip 0pt plus 1filll
@insertcopying
@end titlepage
@c Output the table of the contents at the beginning.
@contents
@ifnottex
@node Top, Introduction, (dir), (dir)
@top BuildBot
@insertcopying
@end ifnottex
@menu
* Introduction:: What the BuildBot does.
* Installation:: Creating a buildmaster and buildslaves,
running them.
* Concepts:: What goes on in the buildbot's little mind.
* Configuration:: Controlling the buildbot.
* Getting Source Code Changes:: Discovering when to run a build.
* Build Process:: Controlling how each build is run.
* Status Delivery:: Telling the world about the build's results.
* Command-line tool::
* Resources:: Getting help.
* Developer's Appendix::
* Index of Useful Classes::
* Index of master.cfg keys::
* Index:: Complete index.
@detailmenu
--- The Detailed Node Listing ---
Introduction
* History and Philosophy::
* System Architecture::
* Control Flow::
System Architecture
* BuildSlave Connections::
* Buildmaster Architecture::
* Status Delivery Architecture::
Installation
* Requirements::
* Installing the code::
* Creating a buildmaster::
* Upgrading an Existing Buildmaster::
* Creating a buildslave::
* Launching the daemons::
* Logfiles::
* Shutdown::
* Maintenance::
* Troubleshooting::
Creating a buildslave
* Buildslave Options::
Troubleshooting
* Starting the buildslave::
* Connecting to the buildmaster::
* Forcing Builds::
Concepts
* Version Control Systems::
* Schedulers::
* BuildSet::
* BuildRequest::
* Builder::
* Users::
* Build Properties::
Version Control Systems
* Generalizing VC Systems::
* Source Tree Specifications::
* How Different VC Systems Specify Sources::
* Attributes of Changes::
Users
* Doing Things With Users::
* Email Addresses::
* IRC Nicknames::
* Live Status Clients::
Configuration
* Config File Format::
* Loading the Config File::
* Testing the Config File::
* Defining the Project::
* Change Sources and Schedulers::
* Setting the slaveport::
* Buildslave Specifiers::
* On-Demand ("Latent") Buildslaves::
* Defining Global Properties::
* Defining Builders::
* Defining Status Targets::
* Debug options::
Change Sources and Schedulers
* Scheduler Scheduler::
* AnyBranchScheduler::
* Dependent Scheduler::
* Periodic Scheduler::
* Nightly Scheduler::
* Try Schedulers::
* Triggerable Scheduler::
Buildslave Specifiers
* When Buildslaves Go Missing::
On-Demand ("Latent") Buildslaves
* Amazon Web Services Elastic Compute Cloud ("AWS EC2")::
* Dangers with Latent Buildslaves::
* Writing New Latent Buildslaves::
Getting Source Code Changes
* Change Sources::
* Choosing ChangeSources::
* CVSToys - PBService::
* Mail-parsing ChangeSources::
* PBChangeSource::
* P4Source::
* BonsaiPoller::
* SVNPoller::
* MercurialHook::
* Bzr Hook::
* Bzr Poller::
Mail-parsing ChangeSources
* Subscribing the Buildmaster::
* Using Maildirs::
* Parsing Email Change Messages::
Parsing Email Change Messages
* FCMaildirSource::
* SyncmailMaildirSource::
* BonsaiMaildirSource::
* SVNCommitEmailMaildirSource::
Build Process
* Build Steps::
* Interlocks::
* Build Factories::
Build Steps
* Common Parameters::
* Using Build Properties::
* Source Checkout::
* ShellCommand::
* Simple ShellCommand Subclasses::
* Python BuildSteps::
* Transferring Files::
* Steps That Run on the Master::
* Triggering Schedulers::
* Writing New BuildSteps::
Source Checkout
* CVS::
* SVN::
* Darcs::
* Mercurial::
* Arch::
* Bazaar::
* Bzr::
* P4::
* Git::
Simple ShellCommand Subclasses
* Configure::
* Compile::
* Test::
* TreeSize::
* PerlModuleTest::
* Testing with mysql-test-run::
* SetProperty::
* SubunitShellCommand::
Python BuildSteps
* BuildEPYDoc::
* PyFlakes::
* PyLint::
Writing New BuildSteps
* BuildStep LogFiles::
* Reading Logfiles::
* Adding LogObservers::
* BuildStep URLs::
Build Factories
* BuildStep Objects::
* BuildFactory::
* Process-Specific build factories::
BuildStep Objects
* BuildFactory Attributes::
* Quick builds::
BuildFactory
* BuildFactory Attributes::
* Quick builds::
Process-Specific build factories
* GNUAutoconf::
* CPAN::
* Python distutils::
* Python/Twisted/trial projects::
Status Delivery
* WebStatus::
* MailNotifier::
* IRC Bot::
* PBListener::
* Writing New Status Plugins::
WebStatus
* WebStatus Configuration Parameters::
* Buildbot Web Resources::
* XMLRPC server::
* HTML Waterfall::
Command-line tool
* Administrator Tools::
* Developer Tools::
* Other Tools::
* .buildbot config directory::
Developer Tools
* statuslog::
* statusgui::
* try::
waiting for results
* try --diff::
Other Tools
* sendchange::
* debugclient::
@end detailmenu
@end menu
@node Introduction, Installation, Top, Top
@chapter Introduction
@cindex introduction
The BuildBot is a system to automate the compile/test cycle required by most
software projects to validate code changes. By automatically rebuilding and
testing the tree each time something has changed, build problems are
pinpointed quickly, before other developers are inconvenienced by the
failure. The guilty developer can be identified and harassed without human
intervention. By running the builds on a variety of platforms, developers
who do not have the facilities to test their changes everywhere before
checkin will at least know shortly afterwards whether they have broken the
build or not. Warning counts, lint checks, image size, compile time, and
other build parameters can be tracked over time, are more visible, and
are therefore easier to improve.
The overall goal is to reduce tree breakage and provide a platform to
run tests or code-quality checks that are too annoying or pedantic for
any human to waste their time with. Developers get immediate (and
potentially public) feedback about their changes, encouraging them to
be more careful about testing before checkin.
Features:
@itemize @bullet
@item
run builds on a variety of slave platforms
@item
arbitrary build process: handles projects using C, Python, whatever
@item
minimal host requirements: python and Twisted
@item
slaves can be behind a firewall if they can still do checkout
@item
status delivery through web page, email, IRC, other protocols
@item
track builds in progress, provide estimated completion time
@item
flexible configuration by subclassing generic build process classes
@item
debug tools to force a new build, submit fake Changes, query slave status
@item
released under the GPL
@end itemize
@menu
* History and Philosophy::
* System Architecture::
* Control Flow::
@end menu
@node History and Philosophy, System Architecture, Introduction, Introduction
@section History and Philosophy
@cindex Philosophy of operation
The Buildbot was inspired by a similar project built for a development
team writing a cross-platform embedded system. The various components
of the project were supposed to compile and run on several flavors of
unix (linux, solaris, BSD), but individual developers had their own
preferences and tended to stick to a single platform. From time to
time, incompatibilities would sneak in (some unix platforms want to
use @code{string.h}, some prefer @code{strings.h}), and then the tree
would compile for some developers but not others. The buildbot was
written to automate the human process of walking into the office,
updating a tree, compiling (and discovering the breakage), finding the
developer at fault, and complaining to them about the problem they had
introduced. With multiple platforms it was difficult for developers to
do the right thing (compile their potential change on all platforms);
the buildbot offered a way to help.
Another problem was when programmers would change the behavior of a
library without warning its users, or change internal aspects that
other code was (unfortunately) depending upon. Adding unit tests to
the codebase helps here: if an application's unit tests pass despite
changes in the libraries it uses, you can have more confidence that
the library changes haven't broken anything. Many developers
complained that the unit tests were inconvenient or took too long to
run: having the buildbot run them reduces the developer's workload to
a minimum.
In general, having more visibility into the project is always good,
and automation makes it easier for developers to do the right thing.
When everyone can see the status of the project, developers are
encouraged to keep the tree in good working order. Unit tests that
aren't run on a regular basis tend to suffer from bitrot just like
code does: exercising them on a regular basis helps to keep them
functioning and useful.
The current version of the Buildbot is additionally targeted at
distributed free-software projects, where resources and platforms are
only available when provided by interested volunteers. The buildslaves
are designed to require an absolute minimum of configuration, reducing
the effort a potential volunteer needs to expend to be able to
contribute a new test environment to the project. The goal is for
anyone who wishes that a given project would run on their favorite
platform should be able to offer that project a buildslave, running on
that platform, where they can verify that their portability code
works, and keeps working.
@node System Architecture, Control Flow, History and Philosophy, Introduction
@comment node-name, next, previous, up
@section System Architecture
The Buildbot consists of a single @code{buildmaster} and one or more
@code{buildslaves}, connected in a star topology. The buildmaster
makes all decisions about what, when, and how to build. It sends
commands to be run on the build slaves, which simply execute the
commands and return the results. (certain steps involve more local
decision making, where the overhead of sending a lot of commands back
and forth would be inappropriate, but in general the buildmaster is
responsible for everything).
The buildmaster is usually fed @code{Changes} by some sort of version control
system (@pxref{Change Sources}), which may cause builds to be run. As the
builds are performed, various status messages are produced, which are then sent
to any registered Status Targets (@pxref{Status Delivery}).
@c @image{FILENAME, WIDTH, HEIGHT, ALTTEXT, EXTENSION}
@image{images/overview,,,Overview Diagram,}
The buildmaster is configured and maintained by the ``buildmaster
admin'', who is generally the project team member responsible for
build process issues. Each buildslave is maintained by a ``buildslave
admin'', who do not need to be quite as involved. Generally slaves are
run by anyone who has an interest in seeing the project work well on
their favorite platform.
@menu
* BuildSlave Connections::
* Buildmaster Architecture::
* Status Delivery Architecture::
@end menu
@node BuildSlave Connections, Buildmaster Architecture, System Architecture, System Architecture
@subsection BuildSlave Connections
The buildslaves are typically run on a variety of separate machines,
at least one per platform of interest. These machines connect to the
buildmaster over a TCP connection to a publically-visible port. As a
result, the buildslaves can live behind a NAT box or similar
firewalls, as long as they can get to buildmaster. The TCP connections
are initiated by the buildslave and accepted by the buildmaster, but
commands and results travel both ways within this connection. The
buildmaster is always in charge, so all commands travel exclusively
from the buildmaster to the buildslave.
To perform builds, the buildslaves must typically obtain source code
from a CVS/SVN/etc repository. Therefore they must also be able to
reach the repository. The buildmaster provides instructions for
performing builds, but does not provide the source code itself.
@image{images/slaves,,,BuildSlave Connections,}
@node Buildmaster Architecture, Status Delivery Architecture, BuildSlave Connections, System Architecture
@subsection Buildmaster Architecture
The Buildmaster consists of several pieces:
@image{images/master,,,BuildMaster Architecture,}
@itemize @bullet
@item
Change Sources, which create a Change object each time something is
modified in the VC repository. Most ChangeSources listen for messages
from a hook script of some sort. Some sources actively poll the
repository on a regular basis. All Changes are fed to the Schedulers.
@item
Schedulers, which decide when builds should be performed. They collect
Changes into BuildRequests, which are then queued for delivery to
Builders until a buildslave is available.
@item
Builders, which control exactly @emph{how} each build is performed
(with a series of BuildSteps, configured in a BuildFactory). Each
Build is run on a single buildslave.
@item
Status plugins, which deliver information about the build results
through protocols like HTTP, mail, and IRC.
@end itemize
Each Builder is configured with a list of BuildSlaves that it will use
for its builds. These buildslaves are expected to behave identically:
the only reason to use multiple BuildSlaves for a single Builder is to
provide a measure of load-balancing.
Within a single BuildSlave, each Builder creates its own SlaveBuilder
instance. These SlaveBuilders operate independently from each other.
Each gets its own base directory to work in. It is quite common to
have many Builders sharing the same buildslave. For example, there
might be two buildslaves: one for i386, and a second for PowerPC.
There may then be a pair of Builders that do a full compile/test run,
one for each architecture, and a lone Builder that creates snapshot
source tarballs if the full builders complete successfully. The full
builders would each run on a single buildslave, whereas the tarball
creation step might run on either buildslave (since the platform
doesn't matter when creating source tarballs). In this case, the
mapping would look like:
@example
Builder(full-i386) -> BuildSlaves(slave-i386)
Builder(full-ppc) -> BuildSlaves(slave-ppc)
Builder(source-tarball) -> BuildSlaves(slave-i386, slave-ppc)
@end example
and each BuildSlave would have two SlaveBuilders inside it, one for a
full builder, and a second for the source-tarball builder.
Once a SlaveBuilder is available, the Builder pulls one or more
BuildRequests off its incoming queue. (It may pull more than one if it
determines that it can merge the requests together; for example, there
may be multiple requests to build the current HEAD revision). These
requests are merged into a single Build instance, which includes the
SourceStamp that describes what exact version of the source code
should be used for the build. The Build is then randomly assigned to a
free SlaveBuilder and the build begins.
The behaviour when BuildRequests are merged can be customized, @pxref{Merging
BuildRequests}.
@node Status Delivery Architecture, , Buildmaster Architecture, System Architecture
@subsection Status Delivery Architecture
The buildmaster maintains a central Status object, to which various
status plugins are connected. Through this Status object, a full
hierarchy of build status objects can be obtained.
@image{images/status,,,Status Delivery,}
The configuration file controls which status plugins are active. Each
status plugin gets a reference to the top-level Status object. From
there they can request information on each Builder, Build, Step, and
LogFile. This query-on-demand interface is used by the html.Waterfall
plugin to create the main status page each time a web browser hits the
main URL.
The status plugins can also subscribe to hear about new Builds as they
occur: this is used by the MailNotifier to create new email messages
for each recently-completed Build.
The Status object records the status of old builds on disk in the
buildmaster's base directory. This allows it to return information
about historical builds.
There are also status objects that correspond to Schedulers and
BuildSlaves. These allow status plugins to report information about
upcoming builds, and the online/offline status of each buildslave.
@node Control Flow, , System Architecture, Introduction
@comment node-name, next, previous, up
@section Control Flow
A day in the life of the buildbot:
@itemize @bullet
@item
A developer commits some source code changes to the repository. A hook
script or commit trigger of some sort sends information about this
change to the buildmaster through one of its configured Change
Sources. This notification might arrive via email, or over a network
connection (either initiated by the buildmaster as it ``subscribes''
to changes, or by the commit trigger as it pushes Changes towards the
buildmaster). The Change contains information about who made the
change, what files were modified, which revision contains the change,
and any checkin comments.
@item
The buildmaster distributes this change to all of its configured
Schedulers. Any ``important'' changes cause the ``tree-stable-timer''
to be started, and the Change is added to a list of those that will go
into a new Build. When the timer expires, a Build is started on each
of a set of configured Builders, all compiling/testing the same source
code. Unless configured otherwise, all Builds run in parallel on the
various buildslaves.
@item
The Build consists of a series of Steps. Each Step causes some number
of commands to be invoked on the remote buildslave associated with
that Builder. The first step is almost always to perform a checkout of
the appropriate revision from the same VC system that produced the
Change. The rest generally perform a compile and run unit tests. As
each Step runs, the buildslave reports back command output and return
status to the buildmaster.
@item
As the Build runs, status messages like ``Build Started'', ``Step
Started'', ``Build Finished'', etc, are published to a collection of
Status Targets. One of these targets is usually the HTML ``Waterfall''
display, which shows a chronological list of events, and summarizes
the results of the most recent build at the top of each column.
Developers can periodically check this page to see how their changes
have fared. If they see red, they know that they've made a mistake and
need to fix it. If they see green, they know that they've done their
duty and don't need to worry about their change breaking anything.
@item
If a MailNotifier status target is active, the completion of a build
will cause email to be sent to any developers whose Changes were
incorporated into this Build. The MailNotifier can be configured to
only send mail upon failing builds, or for builds which have just
transitioned from passing to failing. Other status targets can provide
similar real-time notification via different communication channels,
like IRC.
@end itemize
@node Installation, Concepts, Introduction, Top
@chapter Installation
@menu
* Requirements::
* Installing the code::
* Creating a buildmaster::
* Upgrading an Existing Buildmaster::
* Creating a buildslave::
* Launching the daemons::
* Logfiles::
* Shutdown::
* Maintenance::
* Troubleshooting::
@end menu
@node Requirements, Installing the code, Installation, Installation
@section Requirements
At a bare minimum, you'll need the following (for both the buildmaster
and a buildslave):
@itemize @bullet
@item
Python: http://www.python.org
Buildbot requires python-2.3 or later, and is primarily developed
against python-2.4. It is also tested against python-2.5 .
@item
Twisted: http://twistedmatrix.com
Both the buildmaster and the buildslaves require Twisted-2.0.x or
later. It has been tested against all releases of Twisted up to
Twisted-2.5.0 (the most recent as of this writing). As always, the
most recent version is recommended.
Twisted is delivered as a collection of subpackages. You'll need at
least "Twisted" (the core package), and you'll also want TwistedMail,
TwistedWeb, and TwistedWords (for sending email, serving a web status
page, and delivering build status via IRC, respectively). You might
also want TwistedConch (for the encrypted Manhole debug port). Note
that Twisted requires ZopeInterface to be installed as well.
@end itemize
Certain other packages may be useful on the system running the
buildmaster:
@itemize @bullet
@item
CVSToys: http://purl.net/net/CVSToys
If your buildmaster uses FreshCVSSource to receive change notification
from a cvstoys daemon, it will require CVSToys be installed (tested
with CVSToys-1.0.10). If the it doesn't use that source (i.e. if you
only use a mail-parsing change source, or the SVN notification
script), you will not need CVSToys.
@end itemize
And of course, your project's build process will impose additional
requirements on the buildslaves. These hosts must have all the tools
necessary to compile and test your project's source code.
@node Installing the code, Creating a buildmaster, Requirements, Installation
@section Installing the code
@cindex installation
The Buildbot is installed using the standard python @code{distutils}
module. After unpacking the tarball, the process is:
@example
python setup.py build
python setup.py install
@end example
where the install step may need to be done as root. This will put the
bulk of the code in somewhere like
/usr/lib/python2.3/site-packages/buildbot . It will also install the
@code{buildbot} command-line tool in /usr/bin/buildbot.
To test this, shift to a different directory (like /tmp), and run:
@example
buildbot --version
@end example
If it shows you the versions of Buildbot and Twisted, the install went
ok. If it says @code{no such command} or it gets an @code{ImportError}
when it tries to load the libaries, then something went wrong.
@code{pydoc buildbot} is another useful diagnostic tool.
Windows users will find these files in other places. You will need to
make sure that python can find the libraries, and will probably find
it convenient to have @code{buildbot} on your PATH.
If you wish, you can run the buildbot unit test suite like this:
@example
PYTHONPATH=. trial buildbot.test
@end example
This should run up to 192 tests, depending upon what VC tools you have
installed. On my desktop machine it takes about five minutes to
complete. Nothing should fail, a few might be skipped. If any of the
tests fail, you should stop and investigate the cause before
continuing the installation process, as it will probably be easier to
track down the bug early.
If you cannot or do not wish to install the buildbot into a site-wide
location like @file{/usr} or @file{/usr/local}, you can also install
it into the account's home directory. Do the install command like
this:
@example
python setup.py install --home=~
@end example
That will populate @file{~/lib/python} and create
@file{~/bin/buildbot}. Make sure this lib directory is on your
@code{PYTHONPATH}.
@node Creating a buildmaster, Upgrading an Existing Buildmaster, Installing the code, Installation
@section Creating a buildmaster
As you learned earlier (@pxref{System Architecture}), the buildmaster
runs on a central host (usually one that is publically visible, so
everybody can check on the status of the project), and controls all
aspects of the buildbot system. Let us call this host
@code{buildbot.example.org}.
You may wish to create a separate user account for the buildmaster,
perhaps named @code{buildmaster}. This can help keep your personal
configuration distinct from that of the buildmaster and is useful if
you have to use a mail-based notification system (@pxref{Change
Sources}). However, the Buildbot will work just fine with your regular
user account.
You need to choose a directory for the buildmaster, called the
@code{basedir}. This directory will be owned by the buildmaster, which
will use configuration files therein, and create status files as it
runs. @file{~/Buildbot} is a likely value. If you run multiple
buildmasters in the same account, or if you run both masters and
slaves, you may want a more distinctive name like
@file{~/Buildbot/master/gnomovision} or
@file{~/Buildmasters/fooproject}. If you are using a separate user
account, this might just be @file{~buildmaster/masters/fooproject}.
Once you've picked a directory, use the @command{buildbot
create-master} command to create the directory and populate it with
startup files:
@example
buildbot create-master @var{basedir}
@end example
You will need to create a configuration file (@pxref{Configuration})
before starting the buildmaster. Most of the rest of this manual is
dedicated to explaining how to do this. A sample configuration file is
placed in the working directory, named @file{master.cfg.sample}, which
can be copied to @file{master.cfg} and edited to suit your purposes.
(Internal details: This command creates a file named
@file{buildbot.tac} that contains all the state necessary to create
the buildmaster. Twisted has a tool called @code{twistd} which can use
this .tac file to create and launch a buildmaster instance. twistd
takes care of logging and daemonization (running the program in the
background). @file{/usr/bin/buildbot} is a front end which runs twistd
for you.)
In addition to @file{buildbot.tac}, a small @file{Makefile.sample} is
installed. This can be used as the basis for customized daemon startup,
@xref{Launching the daemons}.
@node Upgrading an Existing Buildmaster, Creating a buildslave, Creating a buildmaster, Installation
@section Upgrading an Existing Buildmaster
If you have just installed a new version of the Buildbot code, and you
have buildmasters that were created using an older version, you'll
need to upgrade these buildmasters before you can use them. The
upgrade process adds and modifies files in the buildmaster's base
directory to make it compatible with the new code.
@example
buildbot upgrade-master @var{basedir}
@end example
This command will also scan your @file{master.cfg} file for
incompatibilities (by loading it and printing any errors or deprecation
warnings that occur). Each buildbot release tries to be compatible
with configurations that worked cleanly (i.e. without deprecation
warnings) on the previous release: any functions or classes that are
to be removed will first be deprecated in a release, to give users a
chance to start using their replacement.
The 0.7.6 release introduced the @file{public_html/} directory, which
contains @file{index.html} and other files served by the
@code{WebStatus} and @code{Waterfall} status displays. The
@code{upgrade-master} command will create these files if they do not
already exist. It will not modify existing copies, but it will write a
new copy in e.g. @file{index.html.new} if the new version differs from
the version that already exists.
The @code{upgrade-master} command is idempotent. It is safe to run it
multiple times. After each upgrade of the buildbot code, you should
use @code{upgrade-master} on all your buildmasters.
@node Creating a buildslave, Launching the daemons, Upgrading an Existing Buildmaster, Installation
@section Creating a buildslave
Typically, you will be adding a buildslave to an existing buildmaster,
to provide additional architecture coverage. The buildbot
administrator will give you several pieces of information necessary to
connect to the buildmaster. You should also be somewhat familiar with
the project being tested, so you can troubleshoot build problems
locally.
The buildbot exists to make sure that the project's stated ``how to
build it'' process actually works. To this end, the buildslave should
run in an environment just like that of your regular developers.
Typically the project build process is documented somewhere
(@file{README}, @file{INSTALL}, etc), in a document that should
mention all library dependencies and contain a basic set of build
instructions. This document will be useful as you configure the host
and account in which the buildslave runs.
Here's a good checklist for setting up a buildslave:
@enumerate
@item
Set up the account
It is recommended (although not mandatory) to set up a separate user
account for the buildslave. This account is frequently named
@code{buildbot} or @code{buildslave}. This serves to isolate your
personal working environment from that of the slave's, and helps to
minimize the security threat posed by letting possibly-unknown
contributors run arbitrary code on your system. The account should
have a minimum of fancy init scripts.
@item
Install the buildbot code
Follow the instructions given earlier (@pxref{Installing the code}).
If you use a separate buildslave account, and you didn't install the
buildbot code to a shared location, then you will need to install it
with @code{--home=~} for each account that needs it.
@item
Set up the host
Make sure the host can actually reach the buildmaster. Usually the
buildmaster is running a status webserver on the same machine, so
simply point your web browser at it and see if you can get there.
Install whatever additional packages or libraries the project's
INSTALL document advises. (or not: if your buildslave is supposed to
make sure that building without optional libraries still works, then
don't install those libraries).
Again, these libraries don't necessarily have to be installed to a
site-wide shared location, but they must be available to your build
process. Accomplishing this is usually very specific to the build
process, so installing them to @file{/usr} or @file{/usr/local} is
usually the best approach.
@item
Test the build process
Follow the instructions in the INSTALL document, in the buildslave's
account. Perform a full CVS (or whatever) checkout, configure, make,
run tests, etc. Confirm that the build works without manual fussing.
If it doesn't work when you do it by hand, it will be unlikely to work
when the buildbot attempts to do it in an automated fashion.
@item
Choose a base directory
This should be somewhere in the buildslave's account, typically named
after the project which is being tested. The buildslave will not touch
any file outside of this directory. Something like @file{~/Buildbot}
or @file{~/Buildslaves/fooproject} is appropriate.
@item
Get the buildmaster host/port, botname, and password
When the buildbot admin configures the buildmaster to accept and use
your buildslave, they will provide you with the following pieces of
information:
@itemize @bullet
@item
your buildslave's name
@item
the password assigned to your buildslave
@item
the hostname and port number of the buildmaster, i.e. buildbot.example.org:8007
@end itemize
@item
Create the buildslave
Now run the 'buildbot' command as follows:
@example
buildbot create-slave @var{BASEDIR} @var{MASTERHOST}:@var{PORT} @var{SLAVENAME} @var{PASSWORD}
@end example
This will create the base directory and a collection of files inside,
including the @file{buildbot.tac} file that contains all the
information you passed to the @code{buildbot} command.
@item
Fill in the hostinfo files
When it first connects, the buildslave will send a few files up to the
buildmaster which describe the host that it is running on. These files
are presented on the web status display so that developers have more
information to reproduce any test failures that are witnessed by the
buildbot. There are sample files in the @file{info} subdirectory of
the buildbot's base directory. You should edit these to correctly
describe you and your host.
@file{BASEDIR/info/admin} should contain your name and email address.
This is the ``buildslave admin address'', and will be visible from the
build status page (so you may wish to munge it a bit if
address-harvesting spambots are a concern).
@file{BASEDIR/info/host} should be filled with a brief description of
the host: OS, version, memory size, CPU speed, versions of relevant
libraries installed, and finally the version of the buildbot code
which is running the buildslave.
The optional @file{BASEDIR/info/access_uri} can specify a URI which will
connect a user to the machine. Many systems accept @code{ssh://hostname} URIs
for this purpose.
If you run many buildslaves, you may want to create a single
@file{~buildslave/info} file and share it among all the buildslaves
with symlinks.
@end enumerate
@menu
* Buildslave Options::
@end menu
@node Buildslave Options, , Creating a buildslave, Creating a buildslave
@subsection Buildslave Options
There are a handful of options you might want to use when creating the
buildslave with the @command{buildbot create-slave <options> DIR <params>}
command. You can type @command{buildbot create-slave --help} for a summary.
To use these, just include them on the @command{buildbot create-slave}
command line, like this:
@example
buildbot create-slave --umask=022 ~/buildslave buildmaster.example.org:42012 myslavename mypasswd
@end example
@table @code
@item --usepty
This is a boolean flag that tells the buildslave whether to launch child
processes in a PTY or with regular pipes (the default) when the master does not
specify. This option is deprecated, as this particular parameter is better
specified on the master.
@item --umask
This is a string (generally an octal representation of an integer)
which will cause the buildslave process' ``umask'' value to be set
shortly after initialization. The ``twistd'' daemonization utility
forces the umask to 077 at startup (which means that all files created
by the buildslave or its child processes will be unreadable by any
user other than the buildslave account). If you want build products to
be readable by other accounts, you can add @code{--umask=022} to tell
the buildslave to fix the umask after twistd clobbers it. If you want