Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Expectations #127

Merged
merged 40 commits into from Nov 18, 2019
Merged

Expectations #127

merged 40 commits into from Nov 18, 2019

Conversation

@bogdanmarculescu
Copy link
Contributor

bogdanmarculescu commented Nov 5, 2019

Part of the wrapping up.

Two types of expectations are generated:

  1. expectations about the content (activated by expectationsMasterSwitch) which should duplicate basic assertions.
  2. expectations about the structure (activated by responseStructureOracle) which make statements about the expected structure of returned objects (where the swagger has such objects to begin with).

Not sure why the CI assessments are not run at the moment.

1. an "expectationMasterSwitch"
2. individual activations for each individual expectation.
Initial work on partial oracles.
Expectations are now generated (based on result, rather than
full objects for the moment).
Basic works (based on behaviour, like the assertions)
Next step: basic oracle based on some expected REST behaviour.
Basic works (based on behaviour, like the assertions)
Next step: basic oracle based on some expected REST behaviour.
Getting model information on response objects to the writer.
Getting Model information to the writer.
Next step: expect that returned objects conform to the stated object
structure. (E.g. if the object definition has required fields, those
fields are present in the returned object)
This will be added as a first in a series of partial oracles that will
drive the expectation generation.
# Conflicts:
#	core/src/main/kotlin/org/evomaster/core/output/TestCaseWriter.kt
# Conflicts:
#	core/src/main/kotlin/org/evomaster/core/output/TestCaseWriter.kt
WiP.
Generating structure expectations based on the swagger structure.
Still quite a few issues.
WiP.
Filtering out "logged" (to avoid ID troubles).

expectedRefs is now a map that also stores the code where
each particular ref is expected.

Checking that structure is only assessed for codes that should
expect some object (i.e. not 500s, or variations of 400).

Known issue: when a MapGene is processed, it's name gets a "_map" added
to its tail. This causes problems when comparing expected keys vs
present keys.
WiP.
Fix to allow EM to run even if swagger is not initialized.
In that case, no structure expectations should be generated.
WiP.
Ensuring proper init and handling of swagger. Plus additional minor things.
# Conflicts:
#	core/src/main/kotlin/org/evomaster/core/output/TestCaseWriter.kt
#	core/src/main/kotlin/org/evomaster/core/output/service/TestSuiteWriter.kt
and for expectations).
Still a few known problems exist, e.g.:
OCVN - contains messages with the server and port number. Since the port
 changes, this causes failure in the assertions.
Preparing to temporarily wrap up current work on expectations.

Still a few known problems exist:
OCVN - contains messages with the server and port number. Since the port
 changes, this causes failure in the assertions.
@arcuri82

This comment has been minimized.

Copy link
Collaborator

arcuri82 commented Nov 6, 2019

hi,
all CI checks are failing. please look into them

Preparing to temporarily wrap up current work on expectations.

Still a few known problems exist:
OCVN - contains messages with the server and port number. Since the port
 changes, this causes failure in the assertions.
import org.evomaster.core.search.gene.OptionalGene


class PartialOracles {

This comment has been minimized.

Copy link
@arcuri82

arcuri82 Nov 8, 2019

Collaborator

provide some comments explaining what are the the partial oracles

This comment has been minimized.

Copy link
@bogdanmarculescu

bogdanmarculescu Nov 8, 2019

Author Contributor

Will do. I'm expecting this to change quite a bit, but I will do my best to have the explanation there and to keep it as up to date as possible.

import org.hamcrest.Matcher;
import org.hamcrest.TypeSafeMatcher;

public class StringMatcher extends TypeSafeMatcher<String> {

This comment has been minimized.

Copy link
@arcuri82

arcuri82 Nov 8, 2019

Collaborator

can u add some test cases for this new class?

import org.hamcrest.Matcher;
import org.hamcrest.TypeSafeMatcher;

public class SubStringMatcher extends TypeSafeMatcher<String> {

This comment has been minimized.

Copy link
@arcuri82

arcuri82 Nov 8, 2019

Collaborator

can u add some test cases for this new class?

@@ -703,7 +703,7 @@ class EMConfig {

@Experimental
@Cfg("Enable Expectation Generation. If enabled, expectations will be generated. " +
"A variable called activeExpectations is added to each test case, with a default value of false. If set to true, an expectation that fails will cause the test case containing it to fail.")
"A variable called expectationsMasterSwitch is added to the test suite, with a default value of false. If set to true, an expectation that fails will cause the test case containing it to fail.")
var expectationsActive = false

This comment has been minimized.

Copy link
@arcuri82

arcuri82 Nov 8, 2019

Collaborator

are the e2e tests passing when this is set to true?

This comment has been minimized.

Copy link
@bogdanmarculescu

bogdanmarculescu Nov 8, 2019

Author Contributor

Generated tests pass, yes.
EM's e2e tests fail (at least the TestCaseWriterTest does). I'll adjust that and see if anything else causes a problem.

Late edit: There are some problems with generating code for Kotlin test cases. Will take a bit longer, but doesn't seem too difficult.

Trying to make expectationsActive = true pass CI tests.
# Conflicts:
#	core/src/main/kotlin/org/evomaster/core/Main.kt
#	core/src/main/kotlin/org/evomaster/core/problem/rest/RestCallAction.kt
@bogdanmarculescu

This comment has been minimized.

Copy link
Contributor Author

bogdanmarculescu commented Nov 14, 2019

Hej. I think I've covered the required changes. The CI tests seem to pass. (Note that for the tests regarding TestSuiteWriter and TestCaseWriter, I forced the expectationsEnabled option to false).

Do you intend to have the expectations enabled by default?

@arcuri82 arcuri82 merged commit 4c9226e into master Nov 18, 2019
3 checks passed
3 checks passed
ci/circleci: build Your tests passed on CircleCI!
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
continuous-integration/travis-ci/push The Travis CI build passed
Details
@arcuri82

This comment has been minimized.

Copy link
Collaborator

arcuri82 commented Nov 18, 2019

hi. thx. we can wait a bit before before putting them on by default (want to do some manual testing first, later this week)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants
You can’t perform that action at this time.