-
Notifications
You must be signed in to change notification settings - Fork 331
/
JUnit.shtml
1303 lines (1115 loc) · 55.6 KB
/
JUnit.shtml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"
"http://www.w3.org/TR/html4/loose.dtd">
<html lang="en">
<head>
<meta name="generator" content=
"HTML Tidy for Mac OS X (vers 31 October 2006 - Apple Inc. build 15.17), see www.w3.org">
<title>JMRI: Unit testing with JUnit</title>
<meta name="author" content="Bob Jacobsen">
<meta name="keywords" content="JMRI technical code JUnit testing">
<!-- The combination of "Define" and {Header,Style, Logo and Footer} comments -->
<!-- are an arbitrary design pattern used by the update.pl script to -->
<!-- easily replace the common header/footer code for all the web pages -->
<!-- delete the following 2 Defines if you want to use the default JMRI logo -->
<!-- or change them to reflect your alternative logo -->
<!-- Style -->
<meta http-equiv="Content-Type" content=
"text/html; charset=us-ascii">
<link rel="stylesheet" type="text/css" href="/css/default.css"
media="screen">
<link rel="stylesheet" type="text/css" href="/css/print.css"
media="print">
<link rel="icon" href="/images/jmri.ico" type="image/png">
<link rel="home" title="Home" href="/"><!-- /Style -->
</head><!--#include virtual="/Header.shtml" -->
<body>
<div id="mBody">
<!--#include virtual="Sidebar.shtml" -->
<div id="mainContent">
<h1>JMRI Code: Unit testing with JUnit</h1>
<ul>
<li><a href="#Introduction">Introduction</a></li>
<li><a href="#run">Running the Tests</a></li>
<li><a href="#Continuous">Continuous Integration Test
Execution</a></li>
<li><a href="#Reporting">Error Reporting</a></li>
<li><a href="#Coverage">Code Coverage Reports</a></li>
<li><a href="#write">Writing Tests</a></li>
<li style="list-style: none">
<ul>
<li><a href="#writeAddl4ExistClass">Writing Additional
Tests for an Existing Class</a></li>
<li><a href="#write4NewClass">Writing Tests for a New
Class</a></li>
<li><a href="#Write4NewPackage">Writing Tests for a New
Package</a></li>
</ul>
</li>
<li><a href="#keyMetaphors">Key Metaphors</a></li>
<li style="list-style: none">
<ul>
<li><a href="#HandlingLogOutput">Handling Log4J Output
From Tests</a></li>
<li><a href="#ResetInstMgr">Resetting the
InstanceManager</a> and other Managers</li>
<li><a href="#RunningListeners">Working with
Listeners</a></li>
<li><a href="#threads">Working with Threads</a></li>
<li><a href="#io">Testing I/O</a></li>
<li><a href="#tempFileCreation">Temporary File Creation
in Tests</a></li>
<li><a href="#J4rules">JUnit4 Rules</a></li>
<li style="list-style: none">
<ul>
<li><a href="#TimeoutRule">Timeout rule</a>
<li><a href="#RetryRule">RetryRule</a>
</ul>
</li>
<li><a href="#control">Controlling JUnit4 Tests</a></li>
</ul>
</li>
<li><a href="#testSwingCode">Testing Swing Code</a></li>
<li style="list-style: none">
<ul>
<li><a href="#jemmy">Using Jemmy</a></li>
</ul>
</li>
<li><a href="#testScriptCode">Testing Script Code</a></li>
<li style="list-style: none">
<ul>
<li><a href="#sampleScriptTesting">Testing Jython
sample scripts</a></li>
</ul>
</li>
<li><a href="#issues">Issues</a></li>
<li><a href="#junit4">Migrating to JUnit4</a></li>
</ul>
<a name="Introduction" id="Introduction"></a> JUnit is a
system for building "unit tests" of software. Unit tests are
small tests that make sure that individual parts of the
software do what they're supposed to do. In a distributed
project like JMRI, where there are lots of developers in only
loose communication with each other, unit tests are a good
way to make sure that the code hasn't been broken by a
change.
<p>For more information on JUnit, see <a href=
"http://www.junit.org">the JUnit home page</a>.
We now use JUnit version 4 (JUnit4), although
a lot of JMRI code originally had tests
written in the previous version, JUnit3.
For instructions on how to convert existing JUnit3
tests to JUnit4, see the
"<a href="#junit4">Migrating to JUnit4</a>" section below.
<p>
A very
interesting example of test-based development is available
from <a href=
"http://www.objectmentor.com/publications/xpepisode.htm">Robert
Martin</a>'s book.</p>
<p>All of the JMRI classes have JUnit tests available; we
have decided that our
<a href="ContinuousIntegration.shtml">Continuous Integration system</a>
will insist on that. It's good
to add JUnit tests as you make changes to classes (they test your new
functionality to make sure that it is working, and keeps
working as other people change it later), when you have to figure out what somebody's code
does (the test documents exactly what should happen!), and
when you track down a bug (make sure it doesn't come
back).</p>
<h2><a name="run" id="run">Running the Tests</a></h2>To run
the existing tests, say
<pre style="font-family: monospace;">
ant alltest
</pre>
This will compile the test code, which lives in the "test"
subdirectory of the "java" directory in our usual code
distributions, and then run the tests under a GUI. (To make sure
you've recompiled everything, you may want to do <code>ant
clean</code> first)<br>
If you know the name of your test class, or the
test class for your package, you can run that directly with the
"runtest" script:
<pre style="font-family: monospace;">
ant tests
./runtest.csh jmri.jmrit.powerpanel.PowerPanelTest
</pre>
The first line compiles all the test code, and the second
runs a specific test or test suite.<br>
(Hint: How to set this up <a href="IntelliJ.shtml#test">using IntelliJ</a>)
<h3><a name="testingvars" id="testingvars">Optional Checks</a></h3>
There are a number of
run-time optional checks
that can be turned on by setting environmental variables.
We periodically run them to check on how the overall test
system is working, but they're too time intensive
to leave on all the time.
<dl>
<dt>jmri.skipschematests</dt>
<dd>If true, JUnit tests will skip checking the schema of all the test XML files.</dd>
<dt>jmri.skipjythontests</dt>
<dd>If true, JUnit tests will skip running the jython/tests scripts.</dd>
<dt>jmri.log4jconfigfilename</dt>
<dd>Override the default "tests.lcf" logging control file name.</dd>
<dt>jmri.demo</dt>
<dd>When set true, leave certain windows open to demonstrate their capability</dd>
<dt>jmri.migrationtests</dt>
<dd>When set true, run some extra tests; usually used during code migration,
where not everything is right yet but you want to be able to include
tests for individual running. </dd>
<dt>jmri.util.JUnitUtil.printSetUpTearDownNames</dt>
<dd>If true, JUnit tests will print out each JUnitUtil.setUp() and JUnitUtil.teardown() call.
This can be useful if i.e. the CI tests are hanging, and you can't figure out which
test class is the problem.</dd>
<dt>jmri.util.JUnitUtil.checkSetUpTearDownSequence</dt>
<dd>If true, check for whether JUnitUtil.setUp() and JUnitUtil.teardown() follow each other
int the proper sequence. Print a message if not. (This slows execution a
bit due to the time needed to keep history for the message)</dd>
<dt>jmri.util.JUnitUtil.checkSequenceDumpsStack</dt>
<dd>If true, makes jmri.util.JUnitUtil.checkSetUpTearDownSequence more verbose by also
including the current stack trace along with the traces of the most recent setUp and
tearDown calls.</dd>
<dt>jmri.util.JUnitUtil.checkRemnantThreads</dt>
<dd>If true, checks for any threads that have not yet been terminated during
the test tearDown processing. If found, the context is logged as a warning.</dd>
<dt>jmri.util.JUnitUtil.checkTestDuration</dt>
<dd>If true, issues a warning if a test takes too long. The default
limit is 5000 msec, but you can change it defining the
jmri.util.JUnitUtil.checkTestDurationMax environment variable.</dd>
</dl>
For more on setting these, see the
<a href="StartUpScripts.shtml#prop">start-up scripts page</a>.
<h3><a name="I18N" id="I18N">A Note on Internationalization (I18N)</a></h3>
Tests check the correctness of text in GUI elements, warning messages, and
other places. Many of these are
<a href="I8N.shtml">internationalized</a>,
varying depending on the computer's Locale.
<p>To avoid false failures, the
<a href="Ant.shtml">Ant</a> and
<a href="Ant.shtml#maven">Maven</a>
build control files set the locale to en_US
before running tests. This covers
<a href="ContinuousIntegration.shtml">continuous integration</a> running,
and running locally using e.g. "ant headlesstest" or "ant alltest".
<p>
The ./runtest.csh mechanism does <u>not</u>
automatically set the locale. To do that, the easiest approach
is to set the JMRI_OPTIONS environment variable via one of:
<pre style="font-family: monospace;">
setenv JMRI_OPTIONS "-Duser.language=en -Duser.region=US"
export JMRI_OPTIONS="-Duser.language=en -Duser.region=US"
</pre>
depending on what kind of OS and shell you're using.
For more on how this works, see the page on
<a href="StartUpScripts.shtml">startup scripts</a>.
<h2><a name="Continuous" id="Continuous">Continuous
Integration Test Execution</a></h2>The <a href=
"ContinuousIntegration.shtml">continuous integration
environment</a> senses changes in the code repository,
rebuilds the code, performs a variety of checks. If no fatal
issues are found, the continuous integration process executes
the "alltest" ant target against the build to run the tests
against the successful build of the code base.
<h3><a name="Reporting" id="Reporting">Error
Reporting</a></h3>If a test fails during the continuous
integration execution of "alltest", an e-mail is sent to the
jmri-build e-mail list as well as to the developers who have
checked in code which was included in the build.
<p>You may visit the web site <a href=
"https://lists.sourceforge.net/lists/listinfo/jmri-builds">to
subscribe to the jmri-builds e-mail list</a> to get the bad
news as quickly as possible, or monitor <a href=
"http://sourceforge.net/mailarchive/forum.php?forum_name=jmri-builds">
to view the archives of the e-mail list</a> and see past
logs. Or you can monitor the "dashboard" at the <a href=
"ContinuousIntegration.shtml">continuous integration</a> web
site.</p>
<p>(When the build succeeds, nothing is mailed, to cut down
on traffic)</p>
<h3><a name="Coverage" id="Coverage">Code Coverage
Reports</a></h3>As part of running the tests, Jenkins
accumulates information on how much of the code was executed,
called the "code coverage". We use the <a href=
"http://eclemma.org/jacoco/">JaCoCo tool</a> to do the
accounting. It provides detailed reports at multiple levels:
<ul>
<li><a href=
"http://builds.jmri.org/jenkins/job/Development/job/JaCoCo/jacoco/">
A plot of coverage as a whole</a>. Click on the graph to
see a</li>
<li><a href=
"http://builds.jmri.org/jenkins/job/Development/job/JaCoCo/lastBuild/jacoco/">
summary by Java package</a>. Click on a package to see
a</li>
<li><a href=
"http://builds.jmri.org/jenkins/job/Development/job/JaCoCo/lastBuild/jacoco/jmri.jmrit.blockboss/">
summary by file</a> (e.g. class). Click on a class to see
a</li>
<li><a href=
"http://builds.jmri.org/jenkins/job/Development/job/JaCoCo/lastBuild/jacoco/jmri.jmrit.blockboss/BlockBossLogic/">
summary by method</a>. Click on a method to see</li>
<li><a href=
"http://builds.jmri.org/jenkins/job/Development/job/JaCoCo/lastBuild/jacoco/jmri.jmrit.blockboss/BlockBossLogic/defineIO()/">
how each part of the code was covered</a> (may require
scrolling down).</li>
</ul>
<h2><a name="write" id="write">Writing Tests</a></h2>By
convention, we have a "test" class shadowing (almost) every
real class. The "test" directory contains a tree of package
directories parallel to the "src" tree. Each test class has
the same name as the class to be tested, except with "Test"
appended, and will appear in the "test" source tree. For
example, the "jmri.Version" class's source code is in
"src/jmri/Version.java", and it's test class is
"jmri.VersionTest" found in "test/jmri/VersionTest.java".
<p>There are additional classes which are used to group the
test classes for a particular package into JUnit test
suites.</p>
<h3><a name="writeAddl4ExistClass" id=
"writeAddl4ExistClass">Writing Additional Tests for an
Existing Class</a></h3>To write additional tests for a class
with existing tests, first locate the test class. (If one
doesn't exist, see the section below about writing tests for
a new class)
<p>If the test suite has not been converted to JUnit4 yet,
one or more test methods can be added to the class using the
JUnit conventions. Basically, each method needs a name that
starts with "test", e.g. "testFirst", and has to have a
"public void" signature. JUnit will handle everything after
that.</p>
<p>If the test suite has been converted to JUnit4, the JUnit4
conventions require that the test be preceeded by the
"<code>@Test</code>" annotation:
<pre style="font-family: monospace;">
@Test
public void testSomething() {
...
}
</pre>
<p>See the section on <a href="#junit4">JUnit4 Migration</a> for
more information on JUnit4.</p>
<p>In general, test methods should be small, testing just one
piece of the classes operation. That's why they're called
"unit" tests.</p>
<h3><a name="write4NewClass" id="write4NewClass">Writing
Tests for a New Class</a></h3>
<p>To write a test for a new class, you need to create a file that
shadows your new class. For our example, consider creating a
test for a new class that appears in
"<code>src/jmri/jmrix/foo/Foo.java</code>". The new test would
be created in a file named "<code>test/jmri/jmrix/foo/FooTest.java</code>".</p>
<p>Assuming that the Foo class has a default constructor named <code>foo()</code>,
Then the following would be minimal contents for the <code>test/jmri/jmrix/foo/FooTest.java</code> file:
<pre style="font-family: monospace;">
package jmri.jmrix.foo;
import jmri.util.JUnitUtil;
import org.junit.After;
import org.junit.Assert;
import org.junit.Before;
import org.junit.Ignore;
import org.junit.Test;
/*
* Tests for the Foo Class
* @author Your Name Copyright (C) 2016
*/
public class FooTest {
@Test
public void testCtor() {
Assert.AssertNotNull("Foo Constructor Return",new foo());
}
@Before
public void setUp() {
JUnitUtil.setUp();
}
@After
public void tearDown() {
JUnitUtil.tearDown();
}
}
</pre>
<p>
Note that you should be invoking
<code>jmri.util.JUnitUtil.setUp()</code>
and
<code>jmri.util.JUnitUtil.tearDown()</code>
as above.
<p>In addition, the tearDown() method should set all member variable
references to null. This is because JUnit4 keeps the test class
objects around until <em>all</em> the tests are complete, so any
memory you allocate in one class can't be garbage collected until all
the tests are done. Setting the references to null allows the objects
to be collected. (If the allocated objects have a dispose() method
or similar, you should call that too). You should <u>not</u>
<a href="#ResetInstMgr">reset the InstanceManager</a>
or other managers in the tearDown
method; any necessary manager resets will be done automatically,
and duplicating those wastes test time.
<p>
You may also choose to copy an existing test file and make
modifications to suite the needs of your new class. Please
make sure you're copying a file in the new JUnit4 format,
with the @Test statements, to keep us from having to update
your new file later.
</p>
<p>
After the test class is created it needs to be added to the
package test for the package. In the case of our example,
that should be the file <code>test/jmri/jmrix/foo/PackageTest.java</code>
</p>
<p>
"<code>FooTest.class</code>" needs to be added to the list test classes in the <code>@Suite.SuiteClasses</code> annotation that appears before the beginning of the PackageTest class.</p>
<h3><a name="Write4NewPackage" id="Write4NewPackage">Writing
Tests for a New Package</a></h3>
<p>To write a tests for a new package, in addition to <a href="#write4NewClass">writing tests for each class</a>, you need to create a "<code>PackageTest.java</code> file that calls your new tests. For our example, we will create the file "<code>test/jmri/jmrix/foo/PackageTest.java</code>" and call the tests in "<code>test/jmri/jmrix/foo/FooTest.java</code>".</p>
The following would be minimal contents for the <code>test/jmri/jmrix/foo/PackageTest.java</code> file:
<pre style="font-family: monospace;">
package jmri.jmrix.foo;
import org.junit.runner.RunWith;
import org.junit.runners.Suite;
/**
* tests for the jmri.jmrix.foo package
*
* @author Your Name Copyright (C) 2016
*/
@RunWith(Suite.class)
@Suite.SuiteClasses({
FooTest.class
})
public class PackageTest{
}
</pre>
<p>
You may also choose to copy an existing test file and make
modifications to suite the needs of your new class.
<p>
After the PackageTest class is created it needs to be added to the
PackageTest for the enclosing package. In the case of our example,
the enclosing package test would be the file <code>java/test/jmri/jmrix/PackageTest.java</code>
</p>
<p>
"<code>jmri.jmrix.foo.PackageTest.class</code>" needs to be added to the list test classes in the <code>@RunWithSuite</code> annotation that appears before the beginning of the enclosing PackageTest class.</p>
<h2><a name="keyMetaphors" id="keyMetaphors">Key Test
Metaphors</a></h2>
<h3><a name="HandlingLogOutput" id=
"HandlingLogOutput">Handling Log4J Output From
Tests</a></h3>JMRI uses <a href=
"http://logging.apache.org/log4j/docs/index.html">Log4j</a>
to <a href="Logging.shtml">handle logging of various
conditions</a>, including error messages and debugging
information. Tests are intended to run without error or
warning output, so that it's immediately apparent from an
empty standard log that they ran cleanly.
<p>Log4j usage in the test classes themselves has two
aspects:</p>
<ol>
<li>It's perfectly OK to use log.debug(...) statements to
make it easy to debug problems in test statements.
log.info(...) can be used sparingly to indicate normal
progress, because it's normally turned off when running the
tests.</li>
<li>In general, log.warn or log.error should only be used
when the test then goes on to trigger a JUnit assertion or
exception, because the fact that an error is being logged
does not show up directly in the JUnit summary of
results.</li>
</ol>
<p>On the other hand, you might want to deliberately provoke
errors in the code being tested to make sure that the
conditions are being handled properly. This will often
produce log.error(...) or log.warn(...) messages, which must
be intercepted and checked.</p>
<p>To allow this, JMRI runs it's using tests with a special
log4j appender, which stores messages so that the JUnit tests
can look at them before they are forwarded to the log. There
are two aspects to making this work:</p>
<ol>
<li>All the test classes should include common code in
their setUp() and tearDown() code to ensure that log4j is
properly initiated, and that the custom appender is told
when a test is beginning and ending.
<pre style="font-family: monospace;">
@Before
public void setUp() throws Exception {
JUnitUtil.setUp();
}
@After
public void tearDown() throws Exception {
JUnitUtil.tearDown();
}
</pre>
</li>
<li>When a test is deliberately invoking a message, it
should then use
<a href="https://github.com/JMRI/JMRI/blob/master/java/test/jmri/util/JUnitAppender.java">JUnitAppender class</a>
methods to check that the message was
created. For example, if the class under test is expected
to do<br>
<pre style="font-family: monospace;">
log.warn("Provoked message");
</pre>
the invoking test case should follow the under-test calls that provoke that
with the line:<br>
<pre style="font-family: monospace;">
jmri.util.JUnitAppender.assertWarnMessage("Provoked message");
</pre>
<p>It will be a JUnit error if a log.warn(...) or
log.error(...) message is produced that isn't matched to
a JUnitAppender.assertWarnMessage(...) call.</p>
</li>
<li><a id="warnOnce" name="warnOnce"></a>
The <code>Log4JUtil.warnOnce(..)</code> requires some special
handling in tests. We want each test to be independent, so
we reset the "want <u>only</u> once" logic early in the
<code>JUnitUtil.setUp()</code> that's routinely
invoked <code>@Before</code> the tests. This means that the
first invocation, and only the first invocation, for each
message will be logged.
<li><a id="deprecationWarning" name="deprecationWarning"></a>
We want to make it easy to add a
<code>Log4JUtil.deprecationWarning</code> call when
<a href="RP.shtml#deprecating">a method is deprecated</a>.
This will log a message the first time it's invoked.
We want to warn that deprecated code
is being invoked during normal operation, so this is normally becomes a
<code>Log4JUtil.warnOnce(..)</code> call. When you
see those warnings messages, you should remove them
by completing the migration away from the deprecated method.
<p>The one exception is during unit and CI testing of the actual deprecated
method. We want to keep those tests around until the
deprecated method is finally removed. That ensures it keeps working
until it's deliberately removed, and not inadvertently broken in the
meantime. In this case, you should turn off the
<code>Log4JUtil.deprecationWarning</code> in just that
test method using
<code>Log4JUtil.setDeprecatedLogging(false)</code>
before invoking the deprecated method. (You can also
do an <code>JUnitAppender.assertWarn</code> for all the messages emitted,
but it's easier to just turn them off.)
</ol>
<p>Note: Our <a href="ContinuousIntegration.shtml">CI
test</a> executables are configured to fail if any FATAL or
ERROR messages are emitted instead of being handled. This
means that although you can run your tests successfully on
your own computer if they're emitting ERROR messages, but you
won't be able to merge your code into the common repository
until those are handled. It's currently OK to emit
WARN-level messages during CI testing, but that will also
be restricted (cause the test to fail) during tne 4.15.*
development series, so please suppress or handle those messages too.</p>
<h3><a name="ResetInstMgr" id="ResetInstMgr">Resetting the InstanceManager</a></h3>
If you are testing code that is going
to reference the InstanceManager, you should clear and reset
it to ensure you get reproducible results.
<p>Depending on what managers your code needs, your
<code>setUp()</code> implementation could start with:</p>
<pre style="font-family: monospace;">
JUnitUtil.setUp();
JUnitUtil.resetInstanceManager();
JUnitUtil.resetProfileManager();
JUnitUtil.initConfigureManager();
JUnitUtil.initShutDownManager();
JUnitUtil.initDebugCommandStation();
JUnitUtil.initInternalTurnoutManager();
JUnitUtil.initInternalLightManager();
JUnitUtil.initInternalSensorManager();
JUnitUtil.initReporterManager();
</pre>
(You can omit the initialization managers not needed for your tests)
See the
<a href="https://github.com/JMRI/JMRI/blob/master/java/test/jmri/util/JUnitUtil.java">jmri.util.JUnitUtil</a>
class for the full list of available ones,
and please add more if you need ones that are not in JUnitUtil yet.
<p>Your <code>tearDown()</code> should end with:</p>
<pre style="font-family: monospace;">
JUnitUtil.tearDown();
</pre>
<h3><a name="RunningListeners" id="RunningListeners">Working
with Listeners</a></h3>
JMRI is a multi-threaded application.
Listeners for JMRI objects are notified on various threads.
Sometimes you have to wait for that to take place.
<p>If you want to wait for some specific condition to be
true, e.g. receiving a reply object, you can use a waitFor
method call which looks like:</p>
<pre style="font-family: monospace;">
JUnitUtil.waitFor(()->{reply!=null}, "reply didn't arrive");
</pre>
The first argument is a lambda closure, a small piece of code
that'll be evaluated repeatedly until true. The String second
argument is the text of the assertion (error message) you'll get if
the condition doesn't come true in a reasonable length of time.
<p>Waiting for a specific result is fastest and most
reliable. If you can't do that for some reason, you can do a
short time-based wait:</p>
<pre style="font-family: monospace;">
JUnitUtil.releaseThread(this);
</pre>
This uses a nominal delay. But you might want to consider the
structure of either your code (that you're testing) or the
test itself: If you can't tell whether it succeeded, what's the
purpose of the operation?
<p>Note that this should <b>not</b> be used to synchronize
with Swing threads. See the <a href="#testSwingCode">Testing
Swing Code</a> section for that.</p>
<p>In general, you should not have calls to sleep(), wait()
or yield() in your code. Use the JUnitUtil and JFCUtil
support for those instead.</p>
<h3><a name="threads" id="threads">Working with
Threads</a></h3>(See a <a href="#testSwingCode">following
section</a> for how to work with <a href=
"#testSwingCode">Swing (GUI) objects</a> and the <a href=
"#testSwingCode">Swing/AWT thread</a>)
<p>Some tests will need to start threads, for example to test
signal controls or aspects of layout I/O.</p>
<p>General principles your tests must obey for reliable
operation:</p>
<ul>
<li>At the end of each test, you need to stop() any threads
you started. Doing this in tearDown() can be most reliable,
because tearDown runs even if your test method exists due
to an error.
<p>If you're doing multiple tests with threads, you
should wait for thread to actually stop before moving on
to the next operation. You can do that with a
<code>JUnitUtil.waitFor(..)</code> call that waits on
some flag in the thread.</p>
</li>
<li>If your thread does any operations at
<code>code()</code> that need to happen before you test its
operation, you also have to wait for those to
complete.</li>
</ul>
<p>For example, if creating a thread based on <a href=
"http://jmri.org/JavaDoc/doc/jmri/jmrit/automat/AbstractAutomaton.html">
AbstractAutomat</a>, you can check the start with:</p>
<pre style="font-family: monospace;">
AbsractAutomat p = new MyThreadClass();
p.start();
JUnitUtil.waitFor(()->{return p.isRunning();}, "logic running");
</pre>
and ensure termination with
<pre style="font-family: monospace;">
p.stop();
JUnitUtil.waitFor(()->{return !p.isRunning();}, "logic stopped");
</pre>
<p>
Please make sure your unit tests clean up after themselves!
They should not leave any threads running. Any threads they
start should have either terminated normally by the end of the test
(don't let them just time out and crash later during some other test!)
or you should add code to terminate them.
<p>
You can check whether you've left any threads running by
setting the
<code>jmri.util.JUnitUtil.checkRemnantThreads</code>
environment variable to true, with i.e.
<pre style="font-family: monospace;">
setenv JMRI_OPTIONS -Djmri.util.JUnitUtil.checkRemnantThreads=true
</pre>
or the equivalent for your computer type.
This tells the
<code>JUnitUtil.tearDown()</code>
method to check for any (new) threads that are
still running at the end of each test. This check
is a bit time-intensive, so we don't leave it on
all the time.
<h3><a id="io" name="io"></a>Testing I/O</h3>
Some test
environments don't automatically flush I/O operations such as
streams during testing. If you're testing something that does
I/O, for example a TrafficController, you'll need to add
"flush()" statements on all your output streams. (Having to
wait a long time to make a test reliable is a clue that this
is happening somewhere in your code)
<h3><a name="tempFileCreation" id=
"tempFileCreation">Temporary File Creation in
Tests</a></h3>
Test cases which create temporary files must be
carefully created so that there will not be any problems with
file path, filesystem security, pre-existence of the file,
etc. These tests must also be written in a way that will
operate successfully in the <a href=
"ContinuousIntegration.shtml">continuous integration
build</a> environment. And the temporary files should not
become part of the JMRI code repository.
This section discusses ways to avoid these types of
problems.
<p>
If you need a temporary file or directory, and your test
uses JUnit4, you can use a Rule to create a file or directory
before each test runs.
<pre style="font-family: monospace;">
import org.junit.Rule;
import org.junit.rules.TemporaryFolder;
...
@Rule
public TemporaryFolder folder = new TemporaryFolder();
</pre>
<p>
You then reference "folder" in your test code:
<pre style="font-family: monospace;">
// create a temporary file
File randomNameFile = folder.newFile();
// create a temporary directory
File randomNameDir = folder.newFolder();
</pre>
JUnit4 will make sure the file or folder is removed afterwards regardless of whether the
test succeeds or fails. For more information on this, see the
<a href="http://junit.org/junit4/javadoc/latest/org/junit/rules/TemporaryFolder.html">Javadoc for TemporaryFolder</a>.
<h3>
<a name="J4rules" id="J4rules">JUnit Rules</a></h3>
JUNit4 added the concept of
"<a href="https://github.com/junit-team/junit4/wiki/rules">Rules</a>"
that can modify the execution of tests. They work at the class or test
level to modify the context JUnit4 uses for running the classes tests.
Generally, you start by creating one:
<pre style="font-family: monospace;">
@Rule
public final TemporaryFolder tempFolder = new TemporaryFolder();
</pre>
<p>
Some standard JUnit4 rules you might find useful:
<ul>
<li><a href="https://github.com/junit-team/junit4/wiki/rules#temporaryfolder-rule">TemporaryFolder</a>
- work with temporary files and folders, ensuring
they're cleaned up at the end. See the
<a href="#tempFileCreation">above section</a> for more on using that.
<li><a href="https://github.com/junit-team/junit4/wiki/rules#expectedexception-rules">ExpectedException</a>
- handles test methods that are expected to throw exceptions
</ul>
<h4>
<a name="TimeoutRule" id="TimeoutRule">Timeout - limit duration</a></h4>
The <a href="https://github.com/junit-team/junit4/wiki/rules#timeout-rule">Timeout rule</a>
imposes a timeout on all test methods in a class.
<pre style="font-family: monospace;">
@Rule
public org.junit.rules.Timeout globalTimeout = org.junit.rules.Timeout.seconds(10);
</pre>
<p>
Note that you can also add a timeout option to an individual test via an
argument to the @Test annotation. For example,
<pre style="font-family: monospace;">
@Test(timeout=2000)
</pre>
will put a 2 second (2,000 milliseconds) timeout on that test. If you use
both the rule and the option, the option will control the behavior.
For a bit more info, see the
<a href=
"https://github.com/junit-team/junit4/wiki/Timeout-for-tests">
JUnit4 Timeouts for Tests page</a>.
<h4>
<a name="RetryRule" id="RetryRule">RetryRule - run test several times</a></h4>
JMRI has
<code><a href="https://github.com/JMRI/JMRI/tree/master/java/test/jmri/util/junit/rules/RetryRule.java">jmri.util.junit.rules.RetryRule</a></code>
which can rerun a test multiple times until
it reaches a limit or finally passes. Although it's
better to write reliable tests, this can be a way
to make the CI system more reliable while you
try to find out why a test isn't reliable.
<p>
For a working example, see
<a href="https://github.com/JMRI/JMRI/tree/master/java/test/jmri/jmrit/logix/LearnWarrantTest.java">java/test/jmri/jmrit/logix/LearnWarrantTest.java</a>
<p>
Briefly, you just add the lines
<pre style="font-family: monospace;">
import jmri.util.junit.rules.RetryRule;
@Rule
public RetryRule retryRule = new RetryRule(3); // allow 3 retries
</pre>
to your test class. This will modify how JUnit4
handles errors during all of the tests in that class.
<h3><a id="control" name="control"></a>Tools for Controlling JUnit4 tests</H3>
<ul>
<li><a href=
"https://github.com/junit-team/junit4/wiki/Categories">Categories</a>
- useful ones in our case could be headless/not headless,
hardware specific (loco buffer attached, NCE PowerPro
attached, etc)</li>
<li><a href=
"http://junit.sourceforge.net/javadoc/org/junit/Assume.html">
Assumptions</a> - to conditionally ignore a test. For
example, a test that would fail in a headless environment
can be ignored in headless mode if the first line of the
test method is:<br>
<code>Assume.assumeFalse(GraphicsEnvironment.isHeadless());</code></li>
<li>
<a href="http://builds.jmri.org/jenkins/job/Development/job/Ignored%20Test%20Scan/lastBuild/testReport/">
<img src="http://builds.jmri.org/jenkins/job/Development/job/Ignored%20Test%20Scan/warnings7/trendGraph/png?url=PRIORITY" align="right">
</a>
<a href="http://junit.sourceforge.net/javadoc/org/junit/Ignore.html">Ignore</a> -
mark a test to be unconditionally ignored. For
example, a test that fails because it isn't fully implemented yet can be marked to be ignored:<br>
<pre style="font-family: monospace;">
@org.junit.Ignore("not done yet")
@jmri.util.junit.annotations.ToDo("Need to create some mock Framistat Objects")
@Test
public void notDoneYet() {
// some code that compiles but doesn't run
}
</pre><br>
You should provide the reason for ignoring this test in the <code>@Ignore</code> argument.
<code>@Ignore</code> without an argument
will compile, but Jenkins will mark it as an error.
<p>Also note the
<a href="http://jmri.org/JavaDoc/doc/jmri/util/junit/annotations/ToDo.html">@jmri.util.junit.annotations.ToDo annotation</a>
which indicates that this needs work and provides some more information
about what needs to be done.
<p>
In general, we'd rather have working tests rather than ignored ones, so
we track the number that have been ignored in a Jenkins job, see the image
to the right.
<li>On the other hand, sometimes a test super class (i.e. some abstract base)
requires implementation of a test method that's not applicable to this
particular concrete test
class. It might, for example, test a feature or message that's not
applicable for a specific system's hardware. In that case, you
provide a null body to do nothing, and mark the test as not applicable
with the
<a href="http://jmri.org/JavaDoc/doc/jmri/util/junit/annotations/NotApplicable.html">@jmri.util.junit.annotations.NotApplicable</a>
annotation like this:
<pre style="font-family: monospace;">
@Override
@jmri.util.junit.annotations.NotApplicable("System X doesn't use Framistat Objects")
@Test
public void testFramistatUsage() {}
</pre><br>
</ul>
<h2><a id="testSwingCode" name="testSwingCode"></a>Testing
Swing Code</h2>AWT and Swing code runs on a separate thread
from JUnit tests. Once a Swing or AWT object has been
displayed (via <code>show()</code> or
<code>setVisible(true)</code>), it cannot be reliably
accessed from the JUnit thread. Even using the listener delay
technique described above isn't reliable.
<p>For the simplest possible test, displaying a window for
manual interaction, it's OK to create and invoke a Swing
object from a JUnit test. Just don't try to interact with it
once it's been displayed!</p>
<p>Because we run tests in "headless" mode during the
<a href="ContinuousIntegration.shtml">continuous integration
builds</a>, it's important that tests needing access to the screen start
with:</p>
<pre><code>Assume.assumeFalse(GraphicsEnvironment.isHeadless());</code></pre>
<p>This will run the test only when a
display is available.</p>
<p>GUI tests should close windows when they're done, and in
general clean up after themselves. If you want to keep
windows around so you can manipulate them, e.g. for manual
testing or debugging, you can use the jmri.demo system
parameter to control that:</p>
<pre style="font-family: monospace;">
if (!System.getProperty("jmri.demo", "false").equals("false")) {
myFrame.setVisible(false);
myFrame.dispose();
}
</pre>
<p>For many tests, you'll both make testing reliable and
improve the structure of your code by separating the GUI
(Swing) code from the JMRI logic and communications. This
lets you check the logic code separately, but invoking those
methods and checking the state them update.</p>
<p>For more complicated GUI testing, the Jemmy tool is generally used.
<h3><a id="jemmy" name="jemmy">Using Jemmy</a></h3>
Recently, e.g. since 2016, some tests have been developed
using the
<a href="http://wiki.netbeans.org/Jemmy">jemmy</a> tool. See e.g. the
<a href="http://wiki.netbeans.org/Jemmy_Samples">samples on the NetBeans pages</a>.
<p>
For more information on Jemmy, please see the
<a href="http://wiki.netbeans.org/Jemmy_Tutorial">tutorial Wiki page</a> and the
<a href="http://atetric.com/atetric/javadoc/org.netbeans/jemmy/2.2.7.5/overview-summary.html">Jemmy Javadoc</a>.
<a id="jemmylocate" name="jemmylocate"></a>
<h4>Locating GUI Items using Jemmy</h4>
<p>Jemmy must be able to find the objects on the screen.
Jemmy Operators are generally used to both locate and manipulate items on the screen.</p>
<p>Here are a few tips for locating items with Jemmy:</p>
<ul>
<li>Some of the Jemmy Operator Constructors allow leaving off an identifier.
If there is only one object of a given type on the screen at any time,
it is acceptable to use this version of the constructor,
but the test may be fragile.</li>
<li>It is easiest to find objects if they have a unique identifier.
In the case where no unique identifier exists,
Jemmy provides a version of most searches that allows you to specify an ordinal index.
Using these may result in tests that break when GUI elements are
added or removed from the frame.</li>
<li>If an item contains its own text (Buttons, for example),
it is recommended you use the text to search for a component.</li>
<li>If an item does not contain its own description,
but the GUI contains a JLabel describing that component,
be certain the JLabel's LabelFor property is set.
A Jemmy JLabelOperator can then be used to find the label,
and retrieve the object.</li>
<li>When looking for a button, window or other item by it's label text,
the default Jemmy string comparison is a case insensitive <code>caption.contains(match)</code>.
If <code>ce</code> is true, then the <code>equals(..)</code> method is used.
The <code>ccs</code> option controls case sensitivity.
</ul>