CTS failures - inmemory

[planetf1]

The following exceptions are seen when running CTS against the inmemory repository

profile-details cat Reference_copies.json| jq '.profileResult.requirementResults[].negativeTestEvidence'
[
  {
    "profileId": 1,
    "requirementId": 14,
    "testCaseId": "repository-entity-reference-copy-lifecycle-APISchemaType",
    "testCaseName": "Repository entity reference copy lifecycle test case",
    "testCaseDescriptionURL": "https://egeria-project.org/guides/cts/repository-workbench/test-cases/repository-entity-reference-copy-lifecycle-test-case",
    "testEvidenceType": "UNEXPECTED_EXCEPTION",
    "assertionId": "Unexpected Exception Exception : CTS test repository-entity-reference-copy-lifecycle caught exception EntityNotKnownException from method getEntityDetail whilst trying to retrieve an entity of type APISchemaType.  Exception message was : OMRS-REPOSITORY-404-002 The entity identified with guid 6cb32eb5-08d2-44a6-b9d1-b3eef74164f8 passed on the getEntityDetail call is not known to the open metadata repository tut.  Method was invoked with parameters: entityGUID : 6cb32eb5-08d2-44a6-b9d1-b3eef74164f8",
    "conformanceException": {
      "exceptionClassName": "java.lang.Exception",
      "errorMessage": "CTS test repository-entity-reference-copy-lifecycle caught exception EntityNotKnownException from method getEntityDetail whilst trying to retrieve an entity of type APISchemaType.  Exception message was : OMRS-REPOSITORY-404-002 The entity identified with guid 6cb32eb5-08d2-44a6-b9d1-b3eef74164f8 passed on the getEntityDetail call is not known to the open metadata repository tut.  Method was invoked with parameters: entityGUID : 6cb32eb5-08d2-44a6-b9d1-b3eef74164f8"
    }
  },
  {
    "profileId": 1,
    "requirementId": 14,
    "testCaseId": "repository-entity-reference-copy-lifecycle-ControlPointDefinition",
    "testCaseName": "Repository entity reference copy lifecycle test case",
    "testCaseDescriptionURL": "https://egeria-project.org/guides/cts/repository-workbench/test-cases/repository-entity-reference-copy-lifecycle-test-case",
    "testEvidenceType": "UNEXPECTED_EXCEPTION",
    "assertionId": "Unexpected Exception Exception : CTS test repository-entity-reference-copy-lifecycle caught exception EntityNotKnownException from method getEntityDetail whilst trying to retrieve an entity of type ControlPointDefinition.  Exception message was : OMRS-REPOSITORY-404-002 The entity identified with guid df571951-f65e-4d5a-b8a2-b51818f3e702 passed on the getEntityDetail call is not known to the open metadata repository tut.  Method was invoked with parameters: entityGUID : df571951-f65e-4d5a-b8a2-b51818f3e702",
    "conformanceException": {
      "exceptionClassName": "java.lang.Exception",
      "errorMessage": "CTS test repository-entity-reference-copy-lifecycle caught exception EntityNotKnownException from method getEntityDetail whilst trying to retrieve an entity of type ControlPointDefinition.  Exception message was : OMRS-REPOSITORY-404-002 The entity identified with guid df571951-f65e-4d5a-b8a2-b51818f3e702 passed on the getEntityDetail call is not known to the open metadata repository tut.  Method was invoked with parameters: entityGUID : df571951-f65e-4d5a-b8a2-b51818f3e702"
    }
  }
]
null
null

See also

• [BUG] CTS failures  #7148
• Full CTS output/job run https://github.com/planetf1/cts/actions/runs/3726901430

It does not appear this always occurs, though it does on github actions. Suspect this is timing related.

[planetf1]

This is not a regression - and can be reproduced on 3.13 as well as 3.14

[planetf1]

Configuring a local rancher environment with the same 7GBm 2 CPU configuration as github actions led to 1 exception of this type.

Inclined to consider this as timing related. A prior fix boosted the retry count in CTS to avoid these.

[planetf1]

Same issue as seen before. See above PR for timeout increase which hopefully will be sufficient
cc: @lpalashevski @mandy-chessell @cmgrote
Suggest main only,

[planetf1]

The change did not prevent errors occuring on last night's run.
graph, xtdb again passing clean, but inmemory failing with 2 reference copy tests

[planetf1]

@cmgrote @mandy-chessell Any thoughts? Beyond that question and reviewing any relevant code, a next step might be to try and run just those tests in a more development oriented environment - possibly needing to introduce delays/latency if we don't figure out why the assets can sometimes not be found - even with a long delay (and no exceptions reporting we've hit the timeout?)

[planetf1]

These (1 typically) continue to occur on the cicd pipeline.
A standalone test in a 3x64GB/8cpu k8s cluster passes clean

This points to a cts framework/parameter issue which needs investigating, presumably timing related.

[planetf1]

Quick update - Automated CTS ran clean for 2 days, including in memory. On day 3 the inmemory failed, with no relevant code changes, suggesting still some intermittency in this constrained environment

[planetf1]

CTS tests now running cleanly for inmemory.