[Bug]: Appsmith-EE version v1.63 crash with former APPSMITH_KEYCLOAK_DB_URL format

[ssichynskyi]

Is there an existing issue for this?

• I have searched the existing issues

Description

Setup

1. Appsmith-ee, self-hosted, deployed in k8s cluster using helm chart. Chart version 3.6.2. Appsmith image version v1.62
2. SaaS Postgress DB is used
3. APPSMITH_KEYCLOAK_DB_URL format used: <user>:<password>@<host>:<port> (which worked fine for ~ 9 months with constant updates)

Misbehavior

1. After update Appsmith container is in a crashloopbackoff state. Completely inoperable. Following logs observed: Invalid DB connection string, error: Connection String: <contents of APPSMITH_KEYCLOAK_DB_URL envvar>}
2. Connection string (contents of APPSMITH_KEYCLOAK_DB_URL) is shown unmasked in logs, revealing user credentials historically stored there.

Expected behavior

1. No crash, old format of APPSMITH_KEYCLOAK_DB_URL envvar is handled without errors
2. The content of APPSMITH_KEYCLOAK_DB_URL is not displayed in logs in a plain text. Because of historical reasons it contains DB username and password: [Bug]: appsmith-ee helm v3.1.0 - Temporal relies on keycloak postgres database url having all information and does not allow @ symbol in password #34153
3. Error message shall explain which format of connection string is expected and how it is related to existing envvars.
4. Documentation mentioning APPSMITH_KEYCLOAK_DB_URL format shall be correct. It was wrong for versions <= v1.62 and it seems it is still wrong now. jdbc:postgresql://<hostname>:<port>/<database_name> doesn't seem to be the right format.
5. Helm chart shall refer to a specific version of the appsmith-ee image, not latest. Otherwise such disruptive update can happen unexpectedly.

Recovery method

Forcibly downgrade the appsmith-ee image version back to v1.62

Steps To Reproduce

1. Update appsmith image version from v1.62 to v1.63

Public Sample App

No response

Environment

Production

Severity

Critical (Broken Production apps)

Issue video log

No response

Version

Self Hosted v1.63 (Enterprise Edition)

[wyattwalter]

Hi @ssichynskyi , thanks for the report here.

As far as the connection string format, it looks like it was a bug that including the username and password in the URL worked previously. Keycloak is expecting only hostname, port, and database in the URL. Then username and password are provided separately.

The Helm chart is defaulting to providing it as:

  APPSMITH_KEYCLOAK_DB_URL: appsmith-postgresql.appsmith.svc.cluster.local:5432/keycloak

which indeed is different than the docs indicate. The startup script has some logic to manipulate this URL because it's actually used by both Keycloak and Temporal (the workflows engine). I can see that there's some logic changes recently to strip credentials out (as it expects) when passing to Keycloak and that may not be working as expected in some scenarios.

I can't reproduce the crashloopbackoff issue when specifying the format as <user>:<pass>@<host>:<port>/<database> on 1.64. However, I do see that Keycloak isn't starting correctly because it can't connect.

It does say URL, but it looks like the way to make this work is to set:

APPSMITH_KEYCLOAK_DB_URL as <host>:<port>/<database> (no protocol and no credentials)

Then pass your username/password in APPSMITH_KEYCLOAK_DB_USERNAME and APPSMITH_KEYCLOAK_DB_PASSWORD.

For now, the above hopefully should unblock you, but we definitely need to clean this up as well because all the points you raise above are valid as well.

[ssichynskyi]

@wyattwalter thank you for taking care. I've tried to run v1.64 using the format ofAPPSMITH_KEYCLOAK_DB_USERNAME that you've specified, but I still experience a similar crash. (I actually tried this format already back then).
The results are the same. I see PG/Keycloak related log:
Invalid DB connection string, error: Connection String: xx-N-xxxxx-xxxxxxxxxxx.xxxxxxxxxx.com:NNNNN/keycloak ...
where x is a letter and N is a number.

I also see mongok migration failure in logs, maybe that's causing the the crashLoopBackOff? Worth to mention, MongoDB is also a SaaS service. But since there are plenty of successful migration logs, it's not a connectivity/auth issue.

backend stdout | [2025-03-18 19:09:57,627] [main] requestId= userEmail= traceId= spanId= - Error in mongock process. ABORTED MIGRATION
backend stdout | io.mongock.api.exception.MongockException: io.mongock.api.exception.MongockException: Error in method[DatabaseChangelogEE.moveLicenseKeyToDB] : Failed to convert from type [java.lang.String] to type [com.appsmith.external.enums.FeatureFlagEnum] for value [release_jit_package_pull_enabled]

APPSMITH_LICENSE_KEY envvar is passed through the secret and looks correct in the container shell.

I'll pass startup log through the support channel to avoid exposing configuration details.

[wyattwalter]

Ok, that MongockException is most certainly what's causing the crashloopbackoff on the pod and is the most important issue here.

The APPSMITH_LICENSE_KEY var is ignored after the first time it's set, so that shouldn't be the issue.

I also can see that Keycloak is starting up, but Temporal can't:

temporal stderr | sql schema version compatibility check failed: pq: no pg_hba.conf entry for host "<ip>", user "keycloak", database "keycloak", no encryption

Temporal has some additional configuration that's needed for connecting to a Postges server that requires TLS, and I'd be surprised if this behavior is new. It causes a lot of noise, but shouldn't cause the pod to fail health checks. Clearly a lot to clean up, but let's focus first on the Mongock migratio issue. I've escalated that part internally.

[ssichynskyi]

For anyone else who faces this issue, the recovery method is to roll back to v1.62 -> open Appsmith UI -> Admin settings -> License & plans -> hit Refresh button. After this you can upgrade to v1.63+ without issues.

[github-actions]

This critical issue has not seen activity for a while. It will be closed in 7 days unless further activity is detected or the Critical tag is removed.

[ssichynskyi]

@wyattwalter @Nikhil-Nandagopal please consider reducing the priority on this issue since the state is recoverable due to the suggested workaround.