The EiffelTestCaseFinishedEvent declares that a previously started test case (declared by EiffelTestCaseStartedEvent) has finished and reports the outcome.
Note that while similar, the data.outcome object is different from that of EiffelActivityFinishedEvent. The outcome of the test case reports not only the conclusion of the test case execution - whether the test case was successfully executed - but also passes a verdict on the implementation under test. To highlight this conceptual difference, both data.outcome.verdict and data.outcome.conclusion are included.
Also note that unlike EiffelTestSuiteFinishedEvent, EiffelTestCaseFinishedEvent must report both data.outcome.verdict and data.outcome.conclusion.
Type: Object
Required: Yes
Description: The outcome of the test case.
Type: String
Required: Yes
Legal values: PASSED, FAILED, INCONCLUSIVE
Description: A terse standardized verdict on the item or items under test.
PASSED signifies that the item or items under test successfully passed the test case.
FAILED signifies that the item or items under test failed to pass the test case.
INCONCLUSIVE signifies that the verdict of the test case was inconclusive. This SHOULD be the case if data.outcome.conclusion is not SUCCESSFUL, but may in combination with a SUCCESSFUL conclusion be used to represent unreliability or flakiness.
Type: String
Required: Yes
Legal values: SUCCESSFUL, FAILED, ABORTED, TIMED_OUT, INCONCLUSIVE
Description: A terse standardized conclusion of the test case, designed to be machine readable.
SUCCESSFUL signifies that the test case was successfully concluded. Note that this does not imply that the implementation under test passed the tests.
FAILED signifies that the test case could not be successfully executed. To exemplify, one or more tests failed to run due to required environments being unavailable.
ABORTED signifies that the test case was aborted before it could be concluded.
TIMED_OUT signifies that the test case did not conclude within the allowed time frame.
INCONCLUSIVE signifies that the outcome of the test case could not be determined.
Type: String
Required: No
Description: A verbose description of the test case outcome, designed to provide human readers with further information.
Type: Object[]
Required: No
Description: A list of metrics collected during the test case execution. Note that while complete freedom is allowed in metrics names and value types, it is highly recommended to keep reported metrics concise and consistent. In other words, do not include excessive amounts of data (use data.persistentLogs for that), and avoid unnecessary variations in value names or types over time.
Type: String
Required: Yes
Description: The metrics name.
Type: Any
Required: Yes
Description: The metrics value.
Type: Object[]
Required: No
Description: An array of persistent log files generated during execution.
Type: String
Required: No
Description: The media type of the URI's payload. Can be used to differentiate between various representations of the same log, e.g. text/html for human consumption and text/plain or application/json for the machine-readable form.
Type: String
Required: Yes
Description: The name of the log file.
Type: String[]
Required: No
Description: Arbitrary tags and keywords that describe this log.
Type: String
Required: Yes
Description: The URI at which the log can be retrieved.
Type: Object
Required: No
Description: An optional object containing a digest of the file's contents, i.e. a checksum, computed using the specified algorithm.
Type: String
Format: One of the hash algorithms listed in section 1 of NIST FIPS 180-4, excluding "SHA-1".
Required: Yes
Legal values: SHA-224, SHA-256, SHA-384, SHA-512, SHA-512/224, SHA-512/256
Description: The cryptographic algorithm used to compute the digest of the file's contents.
Type: String
Format: A lowercase string of hexadecimal digits.
Required: Yes
Description: The digest of the file contents.
This section describes which link types are valid for this event type. For details on how to express the link objects themselves see The Links Object.
Required: No
Legal targets: Any
Multiple allowed: Yes
Description: Identifies a cause of the event occurring. SHOULD not be used in conjunction with CONTEXT: individual events providing CAUSE within a larger context gives rise to ambiguity. It is instead recommended to let the root event of the context declare CAUSE.
Required: No
Legal targets: EiffelActivityTriggeredEvent, EiffelTestSuiteStartedEvent
Multiple allowed: No
Description: Identifies the activity or test suite of which this event constitutes a part.
Required: No
Legal targets: EiffelFlowContextDefinedEvent
Multiple allowed: Yes
Description: Identifies the flow context of the event: which is the continuous integration and delivery flow in which this occurred – e.g. which product, project, track or version this is applicable to.
Required: Yes
Legal targets: EiffelTestCaseTriggeredEvent
Multiple allowed: No
Description: Identifies the relevant test case execution. In other words, EiffelTestCaseTriggeredEvent acts as a handle for a particular test case execution. This differs from CONTEXT. In TEST_CASE_EXECUTION the source carries information pertaining to the target (i.e. the test case execution started, finished or was canceled). In CONTEXT, on the other hand, the source constitutes a subset of the target (e.g. this test case was executed as part of that activity or test suite).
Type: String
Format: UUID
Required: Yes
Description: The unique identity of the event, generated at event creation.
Type: String
Format: An event type name
Required: Yes
Description: The type of event. This field is required by the recipient of the event, as each event type has a specific meaning and a specific set of members in the data and links objects.
Type: String
Format: Semantic Versioning 2.0.0
Required: Yes
Description: The version of the event type. This field is required by the recipient of the event to interpret the contents. Please see Versioning for more information.
Type: Integer
Format: UNIX Epoch time, in milliseconds.
Required: Yes
Description: The event creation timestamp.
Type: String[]
Format: Free text
Required: No
Description: Any tags or keywords associated with the events, for searchability purposes.
Type: Object
Required: No
Description: A description of the source of the event. This object is primarily for traceability purposes, and while optional, some form of identification of the source is HIGHLY RECOMMENDED. It offers multiple methods of identifying the source of the event, techniques which may be select from based on the technology domain and needs in any particular use case.
Type: String
Format: Free text
Required: No
Description: Identifies the domain that produced an event.
Type: String
Format: Hostname
Required: No
Description: The hostname of the event sender.
Type: String
Format: Free text
Required: No
Description: The name of the event sender.
Type: String
Format: purl specification
Required: No
Description: The identity of the serializer software used to construct the event, in purl format.
Type: String
Format: URI
Required: No
Description: The URI of, related to or describing the event sender.
Type: Object
Required: No
Description: An optional object for enclosing security related information, particularly supporting data integrity. See Security for further information.
Type: String
Format: Distinguished Name
Required: Yes
Description: The identity of the author of the event. This property is intended to enable the recipient to identify the author of the event contents and/or look up the appropriate public key for decrypting the meta.security.integrityProtection.signature value and thereby verifying author identity and data integrity.
Type: Object
Required: No
Description: An optional object for enabling information integrity protection via cryptographic signing. To generate a correct meta.security.integrityProtection object:
- Generate the entire event, but with the meta.security.integrityProtection.signature value set to an empty string.
- Serialize the event on Canonical JSON Form.
- Generate the signature using the meta.security.integrityProtection.alg algorithm.
- Set the meta.security.integrityProtection.signature value to the resulting signature while maintaining Canonical JSON Form. To verify the integrity of the event, the consumer then resets meta.security.integrityProtection.signature to an empty string and ensures Canonical JSON Form before verifying the signature.
Type: String
Required: Yes
Description: The signature produced by the signing algorithm.
Type: String
Format: A valid JWA RFC 7518 alg parameter value, excluding "none"
Required: Yes
Description: The cryptographic algorithm used to digitally sign the event. If no signing is performed, the meta.security.integrityProtection SHALL be omitted rather than setting meta.security.integrityProtection.alg to "none".
Type: String
Required: No
Description: The producer of the event may include the relevant public key for convenience, rather than relying a separate key distribution mechanism. Note that this property, along with the rest of the event, is encompassed by the integrity protection offered via meta.security.integrityProtection.
Type: Object[]
Required: No
Description: An optional object for enabling verification of intact event sequences in a distributed environment, thereby protecting against data loss, race conditions and replay attacks. It allows event publishers to state the order in which they produce a certain set of events. In other words, it cannot provide any global guarantees as to event sequencing, but rather per-publisher guarantees. Every object in the array represents a named sequence of which this event forms a part. For every event including a given named sequence, the publisher SHALL increment meta.security.sequenceProtection.position by 1. The first event produced in a given named sequence SHALL numbered 1.
Type: String
Required: Yes
Description: The name of the sequence. There MUST not be two identical meta.security.sequenceProtection.sequenceName values in the same event.
Type: Integer
Required: Yes
Description: The number of the event within the named sequence.
Type: String
Format: URI
Required: No
Description: A URI pointing at a location from where the schema used when creating this event can be retrieved. It can be used to parse event data for validation and extraction purposes, for example. Note, that the schema on that URI should be considered immutable.
| Version | Introduced in | Changes |
|---|---|---|
| 4.1.0 | Not yet released in an edition | Add data.persistentLogs.integrityProtection.{alg, digest} (see Issue 358). |
| 4.0.0 | Not yet released in an edition | Update meta schema to Draft 2020-12 and add link validation. |
| 3.3.1 | Not yet released in an edition | Change IUT to Implementation Under Test (see Issue 235). |
| 3.3.0 | edition-arica | Add schema URL to the meta object (see Issue 280). |
| 3.2.0 | edition-lyon | Add links.domainId member (see Issue 233). |
| 3.1.0 | edition-lyon | Add data.persistentLogs.{mediaType,tags}. |
| 3.0.0 | edition-agen | Improved information integrity protection (see Issue 185). |
| 2.0.0 | edition-agen | Introduced purl identifiers instead of GAVs (see Issue 182) |
| 1.1.0 | edition-toulouse | Multiple links of type FLOW_CONTEXT allowed. |
| 1.0.1 | edition-toulouse | data.outcome.metrics.value and data.outcome.metrics.name made mandatory. |
| 1.0.0 | edition-bordeaux | Initial version. |