-
Notifications
You must be signed in to change notification settings - Fork 1.9k
/
HOWTO.use_it.txt
92 lines (48 loc) · 3.19 KB
/
HOWTO.use_it.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
Help file for using this testing framework
------------------------------------------
If you ever forget what you're currently reading, you can always see it again by typing
make help
To run all the tests until one fails, type
make
To run all the tests regardless of failures, type
make -i
Either way, you would be making the default target, named 'valid'. (Naming it 'test' would get confusing when you start referring to data train/test splits.)
To clean up so that you can start over, type
make clean
To download and prepare all the relevant data at once, type
make prepData
To erase all the data, type
make eraseData
If you change some code so that the output of some tests changes, or you add some new tests, and you want to save the new output so that future outputs will be compared with it, type
make expected
You can work with individual tests by prepending their name to the make target, as in
make 2.valid
make t17.clean
make myNewTest.expected
Of course, you can make multiple targets together, as in
make 2.clean 2.valid
There are also pre-defined groups of tests that you can run, such as
make regression_group
If you want to create the correct outputs for a test without actually running it, type
make $testName.pretend
This can be useful if you want to run test X, which depends on the outputs of test Y, but you can't run test Y for some reason.
The "valid" target for each test is actually a "run" followed by a "compare". Thus
make 3.run 3.compare
is equivalent to
make 3.valid
and it's possible to invoke either part without the other.
If you want to use an executable other than ../vowpalwabbit/vw, you can supply it on the command line like this:
make EXEC=myProgram
or make ../vowpalwabbit/vw a sym-link to it.
Other variables, such as ARCH, DIFF, TIME, WGET, and GREP, can also be set on the command line similarly.
If you want to pass extra parameters to all tests, set the EP variable, like this:
make EP="--random_seed 13"
If you want to check speed in addition to accuracy, invoke with TIMING=y . Due to the many factors that can change execution speed, we do not expect speed results to be portable across machines. Therefore, "expected" speed files are not checked into the repository. If you'd like to do regression testing for speed, you should `make TIMING=y; make TIMING=y expected` on your particular machine before making any code changes. In fact, it's a good idea to then `make TIMING=y` a couple more times, and `make TIMING=y expected` again if the speed drops significantly. This can happen even on the same machine, if it's dividing attention between multiple processes.
If you want to add or modify tests, read HOWTO.write_new_tests.txt .
Working with data archives
--------------------------
To save time and bandwidth, you can create and retrieve all the data used in all the tests at once. To create an archive of all the data, type
make ARF=<archive file> archive
You can then stash the archive wherever you like, probably at a URL-accessible location in the cloud. Later, to retrieve the data again, you can type
make URL=<url> prepData
This will cause the data to be downloaded and installed in prepared form, so that you don't need to run the preparation steps.