test_development
- Security/Performance/Usability are not currently considered.
- ls command has no state(regarding input data), so no need to consider it.
- Test design refers to the gnu specification.
Although there are aspects of the ls command specification that should be tested, such as "What information is listed" and "Sorting the output," for the proof of test design concepts, we will only cover the basic behavior and the typical options of the ls command, such as "Which files are listed".
Verify the basic behavior as a command, as described in the following document.
https://www.gnu.org/software/coreutils/manual/html_node/ls-invocation.html#ls-invocation
The test design by CFD according to the above document is as follows(Using GIHOZ):
https://github.com/snsk/ls_testing/blob/main/features/Functional_SpecConfirmance_BasicBehavior.feature https://github.com/snsk/ls_testing/blob/main/features/steps/Functional_SpecConfirmance_BasicBehavior.py
Verify the single arguments as a command, described in the following document. In this section, we will test the "which files to list" argument alone.
https://www.gnu.org/software/coreutils/manual/html_node/Which-files-are-listed.html
The test design by CFD according to the above document is as follows(Using GIHOZ): *vi(value independent): Specify appropriate values according to test conditions at the time of test implementation, independent of test design.
https://github.com/snsk/ls_testing/blob/main/features/Functional_SpecConfirmance_Arguments_Single_WhichFilesAreListed.feature https://github.com/snsk/ls_testing/blob/main/features/steps/Functional_SpecConfirmance_Arguments_Single_WhichFilesAreListed.py
Two combinations of options were generated with the following code.
arg_combi_list = list(itertools.combinations(args ,2))
There is no expected behavior in the Gnu manual for the combination of options. Therefore, Tested the behavior on Ubuntu 20.04 LTS with pytest by setting the expected behavior with an immediate value as a test oracle.
https://github.com/snsk/ls_testing/blob/main/combi_testware.py
Various directory and file structures will be generated and given to the ls command to verify that no serious malfunctions such as crashes or freezes occur. The following elements of directory and file structures are assumed.
- Number of directories
- Length of directory name
- Directory hierarchy
- Number of files
- Length of file names
General restrictions for Linux and file systems are as follows
- Maximum length of file names (directory names) is 255 bytes
- Maximum length of path names is 1023 bytes
Although not a specific system constraint, it has been reported that the shell crashes if the number of files is too large, and this is not a crash of the ls command, so the maximum number of files to be generated should be limited to about 1000.
Execute a random combination of options for the ls command. The directory structure to be executed is a 1/10th of the structure of Hierarchy Fuzzing. The options to be combined are those supported by Ubuntsu at the time of the test run and selected within the scope of this test as follows.
- __arg_array = ["-a", "-A", "-B", "-d", "-H", "--dereference-command-line-symlink-to-dir", "--sort=size", "--sort=extension","--sort=none", "--hide=.a","--ignore=.a", "-L","-R", "."]
This test deals with noise that is not covered by the conformance test based on the specification. A ralph chart is used to organize the structure of the test.
- The input to the ls command is the directory/file structure, command line arguments and the output is the data that will be formatted and printed to standard output. These are covered in the conformance testing, hierarchy fuzzing and arguments fuzzing.
- In pure behavior, the ls command does not maintain state affected by input data, execution history, etc.
- Therefore, noise is the main consideration here. The expected noise for the ls command is as follows.
- ID1:Has no permission.
- ID2:Not exist file/directory name.
- Long long file/directory name(fuzzing covered).
- Long long file path(fuzzing covered).
https://github.com/snsk/ls_testing/blob/main/features/Robustness_Consider_Noise.feature https://github.com/snsk/ls_testing/blob/main/features/steps/Robustness_Consider_Noise.feature.py