[Regression] CTS failure (3.9)

[planetf1]

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

When testing 3.9 ( #6539 ), in-memory CTS fails with:

First cts test run (inmemory) has failed

"testLabResults": {
    "class": "OpenMetadataConformanceTestLabResults",
    "testCaseCount": 5826,
    "testFailedCount": 1,
    "testPassCount": 5825,
    "testResultsFromWorkbenches": [
        {
            "class": "OpenMetadataConformanceWorkbenchResults",
            "failedTestCases": [
                {
                    "class": "OpenMetadataTestCaseResult",
                    "conformanceException": {
                        "errorMessage": "CTS test repository-entity-reference-copy-lifecycle caught exception EntityNotKnownException from method getEntityDetail whilst trying to retrieve an entity of type APISchemaType.  Exception message was : OMRS-REPOSITORY-404-002 The entity identified with guid 3709ed6f-01c7-4901-bf6f-cdc8cba05334 passed on the getEntityDetail call is not known to the open metadata repository tut.  Method was invoked with parameters: entityGUID : 3709ed6f-01c7-4901-bf6f-cdc8cba05334",
                        "exceptionClassName": "java.lang.Exception"
                    },
                    "notSupportAssertions": [],
                    "successfulAssertions": [
                        "repository-test-case-base-01: Repository connector supplied to conformance suite.",
                        "repository-test-case-base-02: Metadata collection for repository connector supplied to conformance suite.",
                        "repository-entity-reference-copy-lifecycle-00: APISchemaType entity type definition matches known type  ",
                        "repository-entity-reference-copy-lifecycle-01: APISchemaType reference entity created; repository supports storage of reference copies.",
                        "repository-entity-reference-copy-lifecycle-02: APISchemaType reference entity can be retrieved as EntitySummary.",
                        "repository-entity-reference-copy-lifecycle-03: APISchemaType reference entity can be retrieved as EntityDetail.",
                        "repository-entity-reference-copy-lifecycle-04: APISchemaType reference entity matches the entity that was saved.",
                        "repository-entity-reference-copy-lifecycle-05: APISchemaType reference entity has no relationships.",
                        "repository-entity-reference-copy-lifecycle-06: APISchemaType reference entity status cannot be updated.",
                        "repository-entity-reference-copy-lifecycle-06: APISchemaType reference entity status cannot be updated.",
                        "repository-entity-reference-copy-lifecycle-07: APISchemaType reference entity properties cannot be updated.",
                        "repository-entity-reference-copy-lifecycle-08: APISchemaType reference entity type cannot be changed.",
                        "repository-entity-reference-copy-lifecycle-09: APISchemaType reference entity identity cannot be changed.",
                        "repository-entity-reference-copy-lifecycle-10: APISchemaType reference entity copy purged at TUT.",
                        "repository-entity-reference-copy-lifecycle-11: APISchemaType reference entity refresh requested by TUT.",
                        "repository-test-case-base-01: Repository connector supplied to conformance suite.",
                        "repository-test-case-base-02: Metadata collection for repository connector supplied to conformance suite."
                    ],
                    "testCaseDescriptionURL": "https://egeria.odpi.org/open-metadata-conformance-suite/docs/repository-workbench/test-cases/repository-entity-reference-copy-lifecycle-test-case.md",
                    "testCaseId": "repository-entity-reference-copy-lifecycle-APISchemaType",
                    "testCaseName": "Repository entity reference copy lifecycle test case",
                    "unsuccessfulAssertions": [
                        "Unexpected Exception Exception : CTS test repository-entity-reference-copy-lifecycle caught exception EntityNotKnownException from method getEntityDetail whilst trying to retrieve an entity of type APISchemaType.  Exception message was : OMRS-REPOSITORY-404-002 The entity identified with guid 3709ed6f-01c7-4901-bf6f-cdc8cba05334 passed on the getEntityDetail call is not known to the open metadata repository tut.  Method was invoked with parameters: entityGUID : 3709ed6f-01c7-4901-bf6f-cdc8cba05334"
                    ]
                }
            ],
            "passedTestCases": [

The entity is not found. 

poll counts were doubled in 7e4a90952fdd9373ef808bbc59f5869be033ec1a - but unlike a prior test failure in 3.9, this does not appear to be a timeout - No POLLING_OVERFLOW audit event - CONFORMANCE-SUITE-0021 - is reported in the stdout/stderr.


### Expected Behavior

CTS passes

### Steps To Reproduce

Run CTS, inmemory, via cts helm chart

### Environment

```markdown
- Egeria:
- OS:
- Java:
- Browser (for UI issues):
- Additional connectors and integration:

Any Further Information?

No response

[planetf1]

A second test ran clean, potentially suggesting this as another timing issue - no timeout message, though another scenario where an entity does not exist. Likely a test problem. Any additional comments @mandy-chessell @cmgrote ?

[planetf1]

Any ideas @mandy-chessell @cmgrote ? otherwise I'll revisit during 3.10 testing & capture more logs

[mandy-chessell]

@planetf1 Please capture the audit logs for both the CTS Server and SUT - I am concerned this has turned up recently in code that is not changing. Most recent changes have been in the enterprise connector and handlers above it which should not affect the CTS.

[planetf1]

@mandy-chessell I found that I had indeed kept the logs - apologies for not posting here earlier. Just sorting how to share the files...

[planetf1]

Files are at https://1drv.ms/u/s!ApVqcIDT57-fmskcY1rBdA_2dfhpAg?e=KeDmbm & hopefully accessible
I have added both a failing case (try1) and a good case (try2) for inmemory, as well as the (good) graph cts.
This was done with the 3.9 release code.

[planetf1]

Same seen in 3.10 . Copied logs from this release into the '3.10' folder in the onedrive directory above

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 20 days if no further activity occurs. Thank you for your contributions.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 20 days if no further activity occurs. Thank you for your contributions.

[planetf1]

Closing as non-reproducible for now