-
Notifications
You must be signed in to change notification settings - Fork 0
/
TODO
117 lines (81 loc) · 4.36 KB
/
TODO
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
Add support for TERMINATED test status, meaning the suite was terminated with
SIGINT or SIGTERM. Ignore SIGINT/SIGTERM in tests which are waiting for
suites/tests to finish. Terminate the whole suite if a suite/test finishes
with TERMINATED, similarly to panic.
Add verification of the suite and test construction validity. E.g. check if
shell initialization corresponds to the type of shell being entered, that it
is done at exactly the right bash level, that any test command is being
executed within appropriately-initialized shell, that appropriate ep_*end is
called, etc.
Consider implementing ep_glob_valid or ep_glob_is_valid for checking pattern
validity. This might require pattern "compilation" and common library.
Consider adding support for disabling ep_abort_assert execution to improve
performance. Measure the difference.
Add support for remote test execution. Make the shell stdout be the log pipe
specified through the appropriate environment variables.
Add support for entering interactive mode before/after a specific or a failed
test.
Consider adding abbreviated interface support:
ep_s
ep_si
ep_sb
ep_se
ep_t
ep_ti
ep_tb
ep_te
Console logging should be done to stdout. Stderr should receive only
pre-logging error messages and out-of-band framework errors.
Consider adding status explanations to the human-summary sink.
Use the following log structure:
STRUCT BEGIN 'path' description
END 'path' status reason
OUTPUT line
TRACE line
There should only be such top level tags as make sense to be separate
filtering levels.
Attach descriptions to external environment variables and add support for
"--help-env" option, which prints them.
Don't ignore skipped command exit status, verify that it skipped itself by
checking for specific exit status.
Output assertion path on beginning/entrance, overwriting previous, append
status on exit/end in ep_log_cook.
Add support for breakpoints invoked with ep_break [name].
Think about making a vim syntax highlighting for main framework functions.
Think about introducing assertion dependencies.
Think about introducing WAIVED and SKIPPED assertions with STRUCT BEGIN.
Consider replacing _ with - in executable names.
Add tracing support.
Add "ep_log_cite" tool to the log pipe, right after "ep_log_mix". The tool
should cite last output line as the possible failure reason in STRUCT END
messages if there was none. Remove citing from "ep_log_cook".
Consider reporting WAIVED tests with rlReport "..." WARN in ep_log_beakerlib.
Think how to automate test independency test - running every test separately.
Investigate possibility of running each suite in its own process group.
Consider adding a pair of functions to compliment ep_suite/ep_test, which
would run the command in an initialized suite/test subshell.
Consider making current STRUCT BEGIN/END - STRUCT ENTER/EXIT events and
restoring STRUCT ENTER/EXIT events, so uncontrolled exit from the shell is
handled properly, such as accessing undefined variables.
Consider quoting function arguments in stack backtrace.
Add "-r" option to "read" invocations where necessary.
Fix filter output of STRUCT BEGIN for SKIPPED suites, refer to recursive suite
passing suite argument in place of include pattern.
Consider removing "nounset" from the default attribute set as not required.
Use ep_env in tests.
Consider adding support for longer assertion descriptions, provided via stdin
to ep_(suite|test)(_begin)? Consider closing stdin by default so absent input
could be detected. As a backup plan, consider adding an option which would
signify there is a description on the input. As another backup plan, consider
specifying longer description via additional option value.
Consider tracing only the shell/file which invoked ep_(suite|test)_init.
Consider outputting assertion's brief description alongside PASSED status in
ep_log_cook.
Consider hiding empty suites with default filter.
Consider adding support for an environment variable containing executable's
command line options, properly escaped to allow use in "eval". Consider
replacing EP_PROTOCOL and EP(_DONT)?_(INCLUDE|ENABLE|CLAIM) with that.
Consider adding support for setup/teardown assertions. A setup assertion
causes the containing suite to ERROR if it fails, a teardown assertion causes
the containing suite to PANIC if it fails. A teardown assertion can be pushed
to teardown stack and executed later.