[RELEASE] Release 3.7

[planetf1]

Work Plan

Create release x.y :

Prior to the release work

• slack post advising of upcoming release & linking to issue
• advance warning in developer/community call
• final agreement to start branch in team call & identification of outstanding issues
• Agree required updates/versions for additional repos including egeria-ui, egeria-reactui, connectors etc

Branching & Correcting versions

• Create branch
• Reassign any issues not being worked on to the next release
• Update version for master (ie x.y-SNAPSHOT > x.y+1-SNAPSHOT)
• Update version for branch (ie x.y-SNAPSHOT -> x.y)

Final updates to the release

• Ensure any remaining fixes are merged into branch (and vice-versa to master)

Generate a release image for testing

• Start release pipeline manually to generate container image

Updating the React UI (as this is part of the notebook test). Note egeria-ui is out of scope other than inclusion of current release

• Create branch
• Update version in master
• Update version in new branch
• Ensure new container image is available for testing (there is no distinct release pipeline)

Updating the Helm Charts (egeria-charts repo)

• checker correct container images are on docker.io & quay.io (these are built by the 'merge' build of a release)
• update image versions for helm charts (egeria-charts repo) (using -prerelease for chart version)

Final tests

• Check swagger doc renders (no regressions)
• Verify odpi-egeria-lab chart (pods active/ready)
• Verify egeria-base chart (pods active/ready)
• Check notebooks (config, start, data catalog at a minimum)
• Check polymer UI (only possible to check it runs - no demo scenario for more)
• Check React UI (rex, tex, glossary author)
• CTS - graph
• CTS - inmemory

Final Docs

• Update release notes in egeria-docs

Final build and publish

• Run 'release' pipeline on branch to push candidates to oss.sonatype.org
• 'close' staging repo & Validate artifacts ok (number, structure, validations) on oss.sonatype.org
• Create final github releases for egeria (add link to egeria docs)
• Close repo on oss.sonatype.org (once updated) for egeria, release
• Check 'release' repo on oss.sonatype.org has artifacts
• Update final versions of egeria-charts to release ie x.y
• Publish that release is now shipped via slack #egeria-announce
• Additional posts to social media
• Communicate to other repo owners ie for connectors so that they can be rebuilt/shipped as needed

Additional components

• ReactUI
  • Create final github release
• PostGres Connector
  • [n/a] Create branch
  • [n/a] Update version in master
  • [n/a] Update version in new branch of connector, and of prereq egeria artifacts
  • [n/a] Run release action, check sonatype & release
  • [n/a] Create github release

[planetf1]

Agreed on developer call 20220324 that we would aim to branch on Monday

• I will manage the egeria branch/release, alongside the charts & egeria-docs
• @davidradl will do a branch of the ecosystem UI on Monday
• expect @cmgrote to update xtdb connector as usual
• @lpalashevski @sarbull unless I hear otherwise i'll use the same version of the business UI as 3.6
• other connectors will not be updated

for future releases we should clearly list our other repos (some combination of an entry in the issue, with other content in egeria-docs?) so that reader can understand more accurately the scope of the release, including for these additional components & our growing set of connectors and additional projects

[planetf1]

Additional task

• Check other shipped repositories for old certificates & update
  • egeria-react-ui @davidradl

[planetf1]

The branch and version updates have now been done (subject to merge). If any fixes are identified that are needed in 3.7
please respond here asap. Will start testing of 3.7 shortly.

[planetf1]

First iteration of helm chart updates done to support testing (CTS/notebooks)
@davidradl if/when you have a egeria-react-ui release ready & released (docker image available on quay.io) then the charts can be further updated, but I wanted to get the cts (takes a couple of days) and notebook testing underway first

See note also for the react-ui certificates.

[planetf1]

Noted automated curation lab fails - see odpi/egeria-jupyter-notebooks#32 (lab still being developed, same failure in 3.6)

[planetf1]

Noted swagger UI still has some bad links - #6158

[planetf1]

CTS graph completed with no test failures -> https://gist.github.com/d2366654723bedb24cef00588ade98b6

@cmgrote I've previously used the cts notebook for validation. - The overall results look similar.

I'm now using the cts chart -- Can you clarify this is normal -- and is any action needed

            "id": 15,
             "name": "Relationship advanced search",
             "description": "The technology under test supports the use of regular expressions to search for relationship instances.",
             "documentationURL": "https://odpi.github.io/egeria-docs/guides/cts/repository-profiles/relationship-advanced-search",
             "profilePriority": "OPTIONAL_PROFILE",
             "conformanceStatus": "CONFORMANT_FULL_SUPPORT",
             "requirementSummary": [
               {
                 "class": "OpenMetadataConformanceRequirementSummary",
                 "id": 0,
                 "name": "Cohort registration",
                 "description": "The technology under test is able to register with a cohort.",
                 "documentationURL": "https://odpi.github.io/egeria-docs/guides/cts/repository-profiles/metadata-sharing/#cohort-          registration",
                 "conformanceStatus": "UNKNOWN_STATUS"
               },

In this example the overall profile result is CONFORMANCE, but there are a lot of UNKNOWN_STATUS in the requirement summary

Secondly, do you have any methodology you've previously used to parse the overall results into a formatted presentation suitable, for example, for release notes - or issue annotation?
cc: @mandy-chessell

[planetf1]

CTS complete for mem (see odpi/egeria-charts#138 -- changed needed to chart + possible cohort issue)

[planetf1]

#6353 was opened from the cts investigation - this looks like a very specific timing condition, and not a regression. I am inclined therefore not to hold release until resolved. Please comment if you object. cc: @mandy-chessell

[planetf1]

CTS results are consistent with previous releases. For clarity the graph and inmem profile results are as follows

local graph (Janus)

jonesn:export/ $ jq '.testLabSummary.testSummariesFromWorkbenches[].profileSummaries[] | { profile: .name, status: .conformanceStatus }'  openmetadata_cts_summary.json
{
  "profile": "Metadata sharing",
  "status": "CONFORMANT_FULL_SUPPORT"
}
{
  "profile": "Reference copies",
  "status": "CONFORMANT_FULL_SUPPORT"
}
{
  "profile": "Metadata maintenance",
  "status": "CONFORMANT_FULL_SUPPORT"
}
{
  "profile": "Dynamic types",
  "status": "UNKNOWN_STATUS"
}
{
  "profile": "Graph queries",
  "status": "CONFORMANT_FULL_SUPPORT"
}
{
  "profile": "Historical search",
  "status": "CONFORMANT_NO_SUPPORT"
}
{
  "profile": "Entity proxies",
  "status": "CONFORMANT_FULL_SUPPORT"
}
{
  "profile": "Soft-delete and restore",
  "status": "CONFORMANT_FULL_SUPPORT"
}
{
  "profile": "Undo an update",
  "status": "CONFORMANT_NO_SUPPORT"
}
{
  "profile": "Reidentify instance",
  "status": "CONFORMANT_FULL_SUPPORT"
}
{
  "profile": "Retype instance",
  "status": "CONFORMANT_FULL_SUPPORT"
}
{
  "profile": "Rehome instance",
  "status": "CONFORMANT_FULL_SUPPORT"
}
{
  "profile": "Entity search",
  "status": "CONFORMANT_FULL_SUPPORT"
}
{
  "profile": "Relationship search",
  "status": "CONFORMANT_FULL_SUPPORT"
}
{
  "profile": "Entity advanced search",
  "status": "CONFORMANT_FULL_SUPPORT"
}
{
  "profile": "Relationship advanced search",
  "status": "CONFORMANT_FULL_SUPPORT"
}

And in-memory:

jonesn:export/ $ jq '.testLabSummary.testSummariesFromWorkbenches[].profileSummaries[] | { profile: .name, status: .conformanceStatus }'  openmetadata_cts_summary.json
{
  "profile": "Metadata sharing",
  "status": "CONFORMANT_FULL_SUPPORT"
}
{
  "profile": "Reference copies",
  "status": "CONFORMANT_FULL_SUPPORT"
}
{
  "profile": "Metadata maintenance",
  "status": "CONFORMANT_FULL_SUPPORT"
}
{
  "profile": "Dynamic types",
  "status": "UNKNOWN_STATUS"
}
{
  "profile": "Graph queries",
  "status": "CONFORMANT_PARTIAL_SUPPORT"
}
{
  "profile": "Historical search",
  "status": "CONFORMANT_FULL_SUPPORT"
}
{
  "profile": "Entity proxies",
  "status": "CONFORMANT_FULL_SUPPORT"
}
{
  "profile": "Soft-delete and restore",
  "status": "CONFORMANT_FULL_SUPPORT"
}
{
  "profile": "Undo an update",
  "status": "CONFORMANT_FULL_SUPPORT"
}
{
  "profile": "Reidentify instance",
  "status": "CONFORMANT_FULL_SUPPORT"
}
{
  "profile": "Retype instance",
  "status": "CONFORMANT_FULL_SUPPORT"
}
{
  "profile": "Rehome instance",
  "status": "CONFORMANT_FULL_SUPPORT"
}
{
  "profile": "Entity search",
  "status": "CONFORMANT_FULL_SUPPORT"
}
{
  "profile": "Relationship search",
  "status": "CONFORMANT_FULL_SUPPORT"
}
{
  "profile": "Entity advanced search",
  "status": "CONFORMANT_FULL_SUPPORT"
}
{
  "profile": "Relationship advanced search",
  "status": "CONFORMANT_FULL_SUPPORT"
}

[MaximVetlugin]

Good day.
We are using Egeria with egeria-connector-xtdb (with jdbc, kafka, rockdb, lucene configuration).

Is it possible to make tests for releases using egeria-connector-xtdb for local repository?
Thank you.

XTDB configuration as example.

{{baseURL}}/open-metadata/admin-services/users/{{user}}/servers/{{server}}/local-repository/mode/plugin-repository/connection

{
"class": "Connection",
"connectorType": {
"class": "ConnectorType",
"connectorProviderClassName": "org.odpi.egeria.connectors.juxt.xtdb.repositoryconnector.XtdbOMRSRepositoryConnectorProvider"
},
"configurationProperties": {
"xtdbConfigEDN": "{:xtdb/index-store {:kv-store {:xtdb/module xtdb.rocksdb/->kv-store :db-dir "data/servers/platform-test/rdb-index"}} :kafka-config {:xtdb/module xtdb.kafka/->kafka-config :bootstrap-servers "kafka-node-1:9092,kafka-node-2:9092,kafka-node-3:9092"} :xtdb/tx-log {:xtdb/module xtdb.kafka/->tx-log :kafka-config :kafka-config :tx-topic-opts {:topic-name "platform-tx-test" :replication-factor 1} :poll-wait-duration "PT0.050S"} :xtdb.lucene/lucene-store {:db-dir "data/servers/platform-test/lucene" :indexer {:xtdb/module xtdb.lucene.egeria/->egeria-indexer} :analyzer {:xtdb/module xtdb.lucene.egeria/->ci-analyzer}} :xtdb/document-store {:xtdb/module xtdb.jdbc/->document-store :connection-pool {:dialect {:xtdb/module xtdb.jdbc.psql/->dialect} :pool-opts {:maximumPoolSize 1 :session_pool_size 3} :db-spec {:jdbcUrl "jdbc:postgresql://postgres/platform-test?user=egeria&password=egeria&ApplicationName=TEST"}}}}"
}
}

[planetf1]

@MaximVetlugin That's a good question - what happens currently is that after base egeria ships, the XTDB connector is typically tested (CTS) by @cmgrote, and then we release it. this is usually a few days after the base. The xtdb config may not match your exact configuration

I think you're right to stress the importance of this, as XTDB in my view is our best connector - most capable, temporal queries, flexible, high availability.

In 3.6 one improvement is that the CTS chart now

• works with inmemory in addition to graph, xtdb (previously I would test release using the CTS notebook)
• works out of the box on more platforms - no openshift security issues, will work on amd64 or arm64
• uses strimzi, same as our other charts

I think we could improve here (some are already planned)

• including info about the xtdb release in our 'release process' docs
• making clear in our user docs what to expect in terms of releases across repos
• automating the testing of xtdb fully - to the point where we test continually (perhaps daily) - this would apply to all connectors - this shouldn't be a big leap (ci/cd - KinD, cts chart....) - at least not if we can fit within our project's github actions quote/usage. Currently the release testing is done by my on an external k8s cluster primarily.
• using xtdb as default in our other charts (so we get more use/coverage)

Contributions very welcome, I'd suggest we open a fresh issue (probably we start with one in the crux repo as that's the focus) to discuss further and figure out next steps with explicit links to other activities, and can also talk about any contributions you may wish to make ;-)

[planetf1]

Release for base ready to be pushed to sonatype/maven central. However javadoc is failing validation on all artifacts

• Fix javadoc publish to maven central

[planetf1]

Charts 3.7.0 are released to support the egeria 3.7 release
Note that egeria-react-ui remains at 3.5.0 - we may release updated charts at some point ie 3.7.1 - we hit issues in testing, due to certificate validation failing (the ui code is correct - the certs in use are setup for localhost, which is not the case in k8s & would result in the UI not working)

[planetf1]

Closing the release issue at this point
Watch https://github.com/odpi/egeria-connector-xtdb for updates on a new release of the xtdb connector