Site URL setting reverting to default randomly

[pitstop-sirish-bajpai]

We are running metabase docker ver 0.41.5 in a kubernetes cluster. The application is exposed externally via an AWS ALB terminating a public https URL. Thus, we need to configure the setting Admin->Settings->General->SITE URL to the public URL, so that metabase links embedded in emails, slack etc. are reachable. This setting seems to be randomly getting reset back to its default, which in our case is the kubernetes nodes private IP and port. The links are then off course unreachable.
There does not seem to be a pattern. I suspected upgrades first, but it even happens days after a version upgrade.

No logs available, since the exact time of config revert is not caught.

Steps to reproduce the behavior:

1. Go to Admin->Settings->General->SITE URL
2. Change it to your public URL and save
3. Monitor any embedded link e.g. in a new user invite email or report posted in slack.
4. LInks start failing after few days, with embedded url showing the private IP address:3000 of the hosting machine.

Expect the setting to persist till changed administratively

Very, very annoying. Get's discovered and reported very quickly, but a simple thing as config persistence is expected to work.

{
  "browser-info": {
    "language": "en-US",
    "platform": "Linux x86_64",
    "userAgent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36",
    "vendor": "Google Inc."
  },
  "system-info": {
    "file.encoding": "UTF-8",
    "java.runtime.name": "OpenJDK Runtime Environment",
    "java.runtime.version": "11.0.13+8",
    "java.vendor": "Eclipse Adoptium",
    "java.vendor.url": "https://adoptium.net/",
    "java.version": "11.0.13",
    "java.vm.name": "OpenJDK 64-Bit Server VM",
    "java.vm.version": "11.0.13+8",
    "os.name": "Linux",
    "os.version": "4.14.252-195.483.amzn2.x86_64",
    "user.language": "en",
    "user.timezone": "UTC"
  },
  "metabase-info": {
    "databases": [
      "mysql",
      "googleanalytics"
    ],
    "hosting-env": "unknown",
    "application-database": "mysql",
    "application-database-details": {
      "database": {
        "name": "MySQL",
        "version": "5.7.12"
      },
      "jdbc-driver": {
        "name": "MariaDB Connector/J",
        "version": "2.6.2"
      }
    },
    "run-mode": "prod",
    "version": {
      "date": "2021-12-16",
      "tag": "v0.41.5",
      "branch": "release-x.41.x",
      "hash": "fbfffc6"
    },
    "settings": {
      "report-timezone": "Asia/Calcutta"
    }
  }
}

[flamber]

Hi @pitstop-sirish-bajpai
What have you defined as the health check endpoint? Unless it's /api/health, then that's likely the cause.
You can define MB_SITE_URL, which make sure that the envvar overrules anything else.

History #16754 @noahmoss

Expected behavior

• Never ever auto-guess the URL unless Metabase is in setup-mode.

[pitstop-sirish-bajpai]

The AWS target group health check endpoint is /api/health.
Kubernetes livenessProbe and readinessProbe endpoints are defaulted to helm chart values from pmin93 repo. These endpoints fields are not even listed in values.yaml file from the repo. Do I need to add the standard enpoint fields in there?
Still smells like a bug though. Configuration should not be changing due to health check misconfigurations.

[flamber]

@pitstop-sirish-bajpai This issue is a bug, and should not happen, which is why it was been labelled as such.
But define MB_SITE_URL, then you won't have this problem happening again.
There's no official helm chart, so you'll need to add that yourself.

[pitstop-sirish-bajpai]

Oh yes..right ! Marked a bug.
Will setup the env and report back.
Thanks.

[noahmoss]

Very weird. We definitely have logic to automatically set site-url only if it's not already set. It seems that somehow the value in the database is being cleared, but I don't currently have any theories for how that would bee happening.

[Caerbannog]

Something similar happened to us on the day we upgraded to 0.43.0.
Our application is also exposed externally via an AWS ALB.
We noticed the problem when links in Slack alerts broke.

The value of site-url that was set is:
3NUN/o/vFBwggqM...snip...OEyv+AcyzO68ltqC6hGTqga8= (length 108)
It looks like some base-64 encoded value?

It didn't happen to us for the past 5 years, so there is definitely a correlation with the upgrade event.

[paoliniluis]

@Caerbannog were you using the env var for setting the site url?

[Caerbannog]

@paoliniluis No, we were not. The URL was configured in the UI.