Unknown custom field / xxx instance not found for new custom properties

[ecerulm]

Affected module
Backend

Describe the bug
On OM 1.10.14 on Kubernetes (with multiple pods)

After creating a custom property via the UI > Settings > Custom Properties > Databases dialect/ String

I tried to add this custom property via REST API

 curl -v -H "Proxy-Authorization: Bearer $IAP_BEARER" -H "Authorization: Bearer $JWT" -H 'Content-Type: application/json-patch+json' -X PATCH "$OM_BASE_URL/api/v1/databases/57dd8e35-4c7a-4409-a609-b4a9203a6723" -d '[{"op": "replace", "path": "/extension/dialect", "value": "xxxxx"}]'| jq .

this succeeds only 25% of the times I try (on the same entity).

Usually I get

 curl -v -H "Proxy-Authorization: Bearer $IAP_BEARER" -H "Authorization: Bearer $JWT" -H 'Content-Type: application/json-patch+json' -X PATCH "$OM_BASE_URL/api/v1/databases/57dd8e35-4c7a-4409-a609-b4a9203a6723" -d '[{"op": "replace", "path": "/extension/dialect", "value": "xxxxx"}]'| jq .
{
  "code": 404,
  "message": "dialect instance for database not found"
}

This happens for both PATCH and GET

$ curl  -H "Proxy-Authorization: Bearer $IAP_BEARER" -H "Authorization: Bearer $JWT" --data-urlencode "fields=extension" -G "$OM_BASE_URL/api/v1/databases/57dd8e35-4c7a-4409-a609-b4a9203a6723"  | jq .
{
  "code": 404,
  "message": "dialect instance for database not found"
}

I have tried the same setup on a local OM 1.10.14 on my laptop and there I don't see this problem, I only see it on a multipod kubernetes setup. I suspect that the failures depend on what pod the request lands on.

To Reproduce

Expected behavior

I expect it perform the PATCH or GET correctly.
I expect the custom properties to sync between all pods (since I suspect only one of the pods know that this new custom property exist)

Version:

• OS: whatever is defined in the official helm chart.
• Python version: whatever is in the official helm chart
• OpenMetadata version: 1.10.14
• OpenMetadata Ingestion package version: [e.g. openmetadata-ingestion[docker]==XYZ]

Additional context

I only experienced this in a multipod kubernetes deployment, I suspect that only one of the pods is aware of the changes in the custom properties.

[ecerulm]

Other users reported in #support that they saw the same behavior since 1.6.0

[ecerulm]

I restarted all pods in my deployment with kubectl -n xxxx rollout restart deployment/openmetadata and after all pods were recreated I don't see the problem, so I have a strong feeling that everytime I create a custom property the other pods will not be "synced" and remain unaware of this new custom property.

The workaround of restarting all pods is obviously not ideal in any production environment.