Skip to content
Permalink
Branch: master
Find file Copy path
Find file Copy path
Fetching contributors…
Cannot retrieve contributors at this time
132 lines (116 sloc) 18.9 KB

Command-Line Options

EvoMaster has several options that can be configured. Those can be set on the command-line when EvoMaster is started.

There are 3 types of options:

  • Important: those are the main options that most users will need to set and know about.

  • Internal: these are low-level tuning options, which most users do not need to modify. These were mainly introduced when experimenting with different configuration to maximize the performance of EvoMaster.

  • Experimental: these are work-in-progress options, for features still under development and testing.

The list of available options can also be displayed by using --help, e.g.:

java -jar evomaster.jar --help

Options might also have constraints, e.g. a numeric value within a defined range, or a string being an URL. In some cases, strings might only be chosen within a specific set of possible values (i.e., an Enum). If any constraint is not satisfied, EvoMaster will fail with an error message.

When used, all options need to be prefixed with a --, e.g., --maxTime.

Important Command-Line Options

Options Description
maxTime String. Maximum amount of time allowed for the search. The time is expressed with a string where hours (h), minutes (m) and seconds (s) can be specified, e.g., 1h10m120s and 72m are both valid and equivalent. Each component (i.e., h, m and s) is optional, but at least one must be specified. In other words, if you need to run the search for just 30 seconds, you can write 30s instead of 0h0m30s. The more time is allowed, the better results one can expect. But then of course the test generation will take longer. For how long should EvoMaster be left run? The default 1 minute is just for demonstration. We recommend to run it between 1 and 24 hours, depending on the size and complexity of the tested application. Constraints: regex (\s*)((?=(\S+))(\d+h)?(\d+m)?(\d+s)?)(\s*). Default value: 60s.
outputFolder String. The path directory of where the generated test classes should be saved to. Default value: src/em.
outputFormat Enum. Specify in which format the tests should be outputted. If left on DEFAULT, then the value specified in the EvoMaster Driver will be used. But a different value must be chosen if doing Black-Box testing. Valid values: DEFAULT, JAVA_JUNIT_5, JAVA_JUNIT_4, KOTLIN_JUNIT_4, KOTLIN_JUNIT_5. Default value: DEFAULT.
testSuiteFileName String. The name of generated file with the test cases, without file type extension. In JVM languages, if the name contains '.', folders will be created to represent the given package structure. Also, in JVM languages, should not use '-' in the file name, as not valid symbol for class identifiers. Constraints: regex [-a-zA-Z$_][-0-9a-zA-Z$_]*(.[-a-zA-Z$_][-0-9a-zA-Z$_]*)*. Default value: EvoMasterTest.
blackBox Boolean. Use EvoMaster in black-box mode. This does not require an EvoMaster Driver up and running. However, you will need to provide further option to specify how to connect to the SUT. Default value: false.
bbSwaggerUrl String. When in black-box mode for REST APIs, specify where the Swagger schema can be downloaded from. Constraints: URL. Default value: "".
bbTargetUrl String. When in black-box mode, specify the URL of where the SUT can be reached. If this is missing, the URL will be inferred from Swagger. Constraints: URL. Default value: "".

Internal Command-Line Options

Options Description
algorithm Enum. The algorithm used to generate test cases. Valid values: MIO, RANDOM, WTS, MOSA. Default value: MIO.
appendToStatisticsFile Boolean. Whether should add to an existing statistics file, instead of replacing it. Default value: false.
archiveTargetLimit Int. Limit of number of individuals per target to keep in the archive. Constraints: min=1.0. Default value: 10.
baseTaintAnalysisProbability Double. Probability to use input tracking (i.e., a simple base form of taint-analysis) to determine how inputs are used in the SUT. Default value: 0.9.
bbExperiments Boolean. Only used when running experiments for black-box mode, where an EvoMaster Driver would be present, and can reset state after each experiment. Default value: false.
bloatControlForSecondaryObjective Boolean. Whether secondary objectives are less important than test bloat control. Default value: false.
createTests Boolean. Specify if test classes should be created as output of the tool. Usually, you would put it to 'false' only when debugging EvoMaster itself. Default value: true.
customNaming Boolean. Enable custom naming and sorting criteria. Default value: true.
e_u1f984 Boolean. QWN0aXZhdGUgdGhlIFVuaWNvcm4gTW9kZQ==. Default value: false.
enableBasicAssertions Boolean. Generate basic assertions. Basic assertions (comparing the returned object to itself) are added to the code. NOTE: this should not cause any tests to fail. Default value: true.
endNumberOfMutations Int. Number of applied mutations on sampled individuals, by the end of the search. Constraints: min=0.0. Default value: 10.
expandRestIndividuals Boolean. Enable to expand the genotype of REST individuals based on runtime information missing from Swagger. Default value: true.
extraHeuristicsFile String. Where the extra heuristics file (if any) is going to be written (in CSV format). Default value: extra_heuristics.csv.
extractSqlExecutionInfo Boolean. Enable extracting SQL execution info. Default value: true.
feedbackDirectedSampling Enum. Specify whether when we sample from archive we do look at the most promising targets for which we have had a recent improvement. Valid values: NONE, LAST, FOCUSED_QUICKEST. Default value: LAST.
focusedSearchActivationTime Double. The percentage of passed search before starting a more focused, less exploratory one. Constraints: probability 0.0-1.0. Default value: 0.5.
geneMutationStrategy Enum. Strategy used to define the mutation probability. Valid values: ONE_OVER_N, ONE_OVER_N_BIASED_SQL. Default value: ONE_OVER_N_BIASED_SQL.
generateSqlDataWithSearch Boolean. Enable EvoMaster to generate SQL data with direct accesses to the database. Use a search algorithm. Default value: true.
heuristicsForSQL Boolean. Tracking of SQL commands to improve test generation. Default value: true.
maxActionEvaluations Int. Maximum number of action evaluations for the search. A fitness evaluation can be composed of 1 or more actions, like for example REST calls or SQL setups. The more actions are allowed, the better results one can expect. But then of course the test generation will take longer. Only applicable depending on the stopping criterion. Constraints: min=1.0. Default value: 1000.
maxResponseByteSize Int. Maximum size (in bytes) that EM handles response payloads in the HTTP responses. If larger than that, a response will not be stored internally in EM during the test generation. This is needed to avoid running out of memory. Default value: 1000000.
maxSearchSuiteSize Int. Define the maximum number of tests in a suite in the search algorithms that evolve whole suites, e.g. WTS. Constraints: min=1.0. Default value: 50.
maxSqlInitActionsPerMissingData Int. When generating SQL data, how many new rows (max) to generate for each specific SQL Select. Constraints: min=1.0. Default value: 5.
maxTestSize Int. Max number of 'actions' (e.g., RESTful calls or SQL commands) that can be done in a single test. Constraints: min=1.0. Default value: 10.
maxTimeInSeconds Int. Maximum number of seconds allowed for the search. The more time is allowed, the better results one can expect. But then of course the test generation will take longer. Only applicable depending on the stopping criterion. If this value is 0, the setting 'maxTime' will be used instead. Constraints: min=0.0. Default value: 0.
populationSize Int. Define the population size in the search algorithms that use populations (e.g., Genetic Algorithms, but not MIO). Constraints: min=1.0. Default value: 30.
probOfRandomSampling Double. Probability of sampling a new individual at random. Constraints: probability 0.0-1.0. Default value: 0.5.
probOfSmartSampling Double. When sampling new test cases to evaluate, probability of using some smart strategy instead of plain random. Constraints: probability 0.0-1.0. Default value: 0.5.
problemType Enum. The type of SUT we want to generate tests for, e.g., a RESTful API. Valid values: REST, WEB. Default value: REST.
secondaryObjectiveStrategy Enum. Strategy used to handle the extra heuristics in the secondary objectives. Valid values: AVG_DISTANCE, AVG_DISTANCE_SAME_N_ACTIONS, BEST_MIN. Default value: AVG_DISTANCE_SAME_N_ACTIONS.
seed Long. The seed for the random generator used during the search. A negative value means the CPU clock time will be rather used as seed. Default value: -1.
showProgress Boolean. Whether to print how much search done so far. Default value: true.
snapshotInterval Double. If positive, check how often, in percentage % of the budget, to collect statistics snapshots. For example, every 5% of the time. Constraints: max=50.0. Default value: -1.0.
snapshotStatisticsFile String. Where the snapshot file (if any) is going to be written (in CSV format). Default value: snapshot.csv.
startNumberOfMutations Int. Number of applied mutations on sampled individuals, at the start of the search. Constraints: min=0.0. Default value: 1.
statisticsColumnId String. An id that will be part as a column of the statistics file (if any is generated). Default value: -.
statisticsFile String. Where the statistics file (if any) is going to be written (in CSV format). Default value: statistics.csv.
stoppingCriterion Enum. Stopping criterion for the search. Valid values: TIME, FITNESS_EVALUATIONS. Default value: TIME.
structureMutationProbability Double. Probability of applying a mutation that can change the structure of a test. Constraints: probability 0.0-1.0. Default value: 0.5.
sutControllerHost String. Host name or IP address of where the SUT REST controller is listening on. Default value: localhost.
sutControllerPort Int. TCP port of where the SUT REST controller is listening on. Constraints: min=0.0, max=65535.0. Default value: 40100.
tournamentSize Int. Number of elements to consider in a Tournament Selection (if any is used in the search algorithm). Constraints: min=1.0. Default value: 10.
useMethodReplacement Boolean. Apply method replacement heuristics to smooth the search landscape. Default value: true.
writeExtraHeuristicsFile Boolean. Whether we should collect data on the extra heuristics. Only needed for experiments. Default value: false.
writeStatistics Boolean. Whether or not writing statistics of the search process. This is only needed when running experiments with different parameter settings. Default value: false.
xoverProbability Double. Probability of applying crossover operation (if any is used in the search algorithm). Constraints: probability 0.0-1.0. Default value: 0.7.

Experimental Command-Line Options

Options Description
S1dR Double. Specify a probability to apply S1dR when resource sampling strategy is 'Customized'. Constraints: probability 0.0-1.0. Default value: 0.25.
S1iR Double. Specify a probability to apply S1iR when resource sampling strategy is 'Customized'. Constraints: probability 0.0-1.0. Default value: 0.25.
S2dR Double. Specify a probability to apply S2dR when resource sampling strategy is 'Customized'. Constraints: probability 0.0-1.0. Default value: 0.25.
SMdR Double. Specify a probability to apply SMdR when resource sampling strategy is 'Customized'. Constraints: probability 0.0-1.0. Default value: 0.25.
adaptivePerOfCandidateGenesToMutate Boolean. Specify whether to decide a top percent of genes to mutate adaptively. Default value: false.
archiveGeneMutation Enum. Whether to enable archive-based gene mutation. Valid values: NONE, SPECIFIED, ADAPTIVE. Default value: NONE.
coveredTargetFile String. Specify a file which saves covered targets info regarding generated test suite. Default value: coveredTargets.txt.
coveredTargetSortedBy Enum. Specify a format to organize the covered targets by the search. Valid values: NAME, TEST. Default value: NAME.
dependencyFile String. Specify a file that saves derived dependencies. Default value: dependencies.csv.
disableStructureMutationDuringFocusSearch Boolean. Specify whether to disable structure mutation during focus search. Default value: false.
doesApplyNameMatching Boolean. Whether to apply text/name analysis with natural language parser to derive relationships between name entities, e.g., a resource identifier with a name of table. Default value: false.
enableCompleteObjects Boolean. Enable EvoMaster to generate, use, and attach complete objects to REST calls, rather than just the needed fields/values. Default value: false.
enableProcessMonitor Boolean. Whether or not enable a search process monitor for archiving evaluated individuals and Archive regarding an evaluation of search. This is only needed when running experiments with different parameter settings. Default value: false.
enableTrackEvaluatedIndividual Boolean. Whether to enable tracking the history of modifications of the individuals with its fitness values (i.e., evaluated individual) during the search. Note that we enforced that set enableTrackIndividual false when enableTrackEvaluatedIndividual is true since information of individual is part of evaluated individual. Default value: false.
enableTrackIndividual Boolean. Whether to enable tracking the history of modifications of the individuals during the search. Default value: false.
endPerOfCandidateGenesToMutate Double. Specify a percentage (after starting a focus search) which is used by archived-based gene selection method (e.g., APPROACH_IMPACT) for selecting top percent of genes as potential candidates to mutate. Constraints: probability 0.0-1.0. Default value: 0.1.
executiveSummary Boolean. Generate an executive summary, containing an example of each category of potential fault found. Default value: false.
expectationsActive Boolean. Enable Expectation Generation. If enabled, expectations will be generated. A variable called expectationsMasterSwitch is added to the test suite, with a default value of false. If set to true, an expectation that fails will cause the test case containing it to fail. Default value: false.
exportCoveredTarget Boolean. Specify whether to export covered targets info. Default value: false.
exportDependencies Boolean. Specify whether to export derived dependencies among resources. Default value: false.
exportImpacts Boolean. Specify whether to export derived impacts among genes. Default value: false.
geneSelectionMethod Enum. Specify whether to enable archive-based selection for selecting genes to mutate. Valid values: NONE, AWAY_NOIMPACT, APPROACH_IMPACT, APPROACH_LATEST_IMPACT, APPROACH_LATEST_IMPROVEMENT, BALANCE_IMPACT_NOIMPACT, ALL_FIXED_RAND. Default value: NONE.
generateSqlDataWithDSE Boolean. Enable EvoMaster to generate SQL data with direct accesses to the database. Use Dynamic Symbolic Execution. Default value: false.
impactFile String. Specify a file that saves derived genes. Default value: impact.csv.
maxLengthOfTraces Int. Specify a maxLength of tracking when enableTrackIndividual or enableTrackEvaluatedIndividual is true. Note that the value should be specified with a non-negative number or -1 (for tracking all history). Constraints: min=-1.0. Default value: 10.
minRowOfTable Int. Specify a minimal number of rows in a table that enables selection (i.e., SELECT sql) to prepare resources for REST Action. In other word, if the number is less than the specified, insertion is always applied. Constraints: min=0.0. Default value: 10.
probOfApplySQLActionToCreateResources Double. Specify a probability to apply SQL actions for preparing resources for REST Action. Constraints: probability 0.0-1.0. Default value: 0.0.
probOfArchiveMutation Double. Specify a probability to enable archive-based mutation. Constraints: probability 0.0-1.0. Default value: 0.0.
probOfEnablingResourceDependencyHeuristics Double. Specify whether to enable resource dependency heuristics, i.e, probOfEnablingResourceDependencyHeuristics > 0.0. Note that the option is available to be enabled only if resource-based smart sampling is enable. This option has an effect on sampling multiple resources and mutating a structure of an individual. Constraints: probability 0.0-1.0. Default value: 0.0.
probOfSelectFromDatabase Double. Specify a probability that enables selection (i.e., SELECT sql) of data from database instead of insertion (i.e., INSERT sql) for preparing resources for REST actions. Constraints: probability 0.0-1.0. Default value: 0.1.
processFiles String. Specify a folder to save results when a search monitor is enabled. Default value: process_data.
processInterval Int. Specify how often to save results when a search monitor is enabled. Default value: 100.
resourceSampleStrategy Enum. Specify whether to enable resource-based strategy to sample an individual during search. Note that resource-based sampling is only applicable for REST problem with MIO algorithm. Valid values: NONE, Customized, EqualProbability, Actions, TimeBudgets, Archive, ConArchive. Default value: NONE.
sortFaultsByCluster Boolean. Sort potential faults based on their similarity (i.e. the cluster they belong to). Default value: false.
startPerOfCandidateGenesToMutate Double. Specify a percentage (before starting a focus search) which is used by archived-based gene selection method (e.g., APPROACH_IMPACT) for selecting top percent of genes as potential candidates to mutate. Constraints: probability 0.0-1.0. Default value: 0.9.
testSuiteSplitType Enum. Instead of generating a single test file, it could be split in several files, according to different strategies. Valid values: NONE, CLUSTER, SUMMARY, CODE. Default value: NONE.
You can’t perform that action at this time.