[Bug]: Can't upgrade coolify instance, stuck on beta.418

[JakeKydd]

Error Message and Logs

I manage a self hosted Coolify instance and noticed I can't update since beta.418. Clicking the Upgrade button in the UI (semi-auto) will result in the process starting, but it never finishes. When trying the manual way, the install script quits right after telling the first joke.

I looked around the repository and on the web but could not find any actions I can take to solve this.

Steps to Reproduce

1. Try to upgrade from 418 to 432

Example Repository URL

No response

Coolify Version

v4.0.0-beta.418

Are you using Coolify Cloud?

No (self-hosted)

Operating System and Version (self-hosted)

Ubuntu 24.04.2 LTS

Additional Information

This is the output from running the install script manually:

sudo curl -fsSL https://cdn.coollabs.io/coolify/install.sh | bash
Welcome to Coolify Installer!
This script will install everything for you. Sit back and relax.
Source code: https://github.com/coollabsio/coolify/blob/v4.x/scripts/install.sh
Using registry URL from .env: ghcr.io
Found existing Docker network pool: 10.0.0.0/8/24
---------------------------------------------
| Operating System  | ubuntu 24.04
| Docker            | 27.0
| Coolify           | 4.0.0-beta.432
| Helper            | 1.0.11
| Realtime          | 1.0.10
| Docker Pool       | 10.0.0.0/8 (size 24)
| Registry URL      | ghcr.io
---------------------------------------------

1. Installing required packages (curl, wget, git, jq, openssl). 
2. Check OpenSSH server configuration. 
 - OpenSSH server is installed.
 - SSH PermitRootLogin is enabled.
3. Check Docker Installation. 
 - Docker is installed.
4. Check Docker Configuration. 
 - Network pool configuration: 10.0.0.0/8/24
 - To override existing configuration: DOCKER_POOL_FORCE_OVERRIDE=true
 - Log configuration is up to date
 - Configuration is up to date
5. Download required files from CDN. 
6. Setting up environment variable file
 - Creating backup of existing .env file to .env-20250929-145739
 - Merging .env.production values into .env
 - .env file merged successfully
7. Checking and updating environment variables if necessary...
8. Checking for SSH key for localhost access.
9. Installing Coolify (4.0.0-beta.432)
 - It could take a while based on your server's performance, network speed, stars, etc.
 - Please wait.
 - Until then, here's a joke for you:

Debugging: Removing the needles from the haystack.

[ShadowArcanist]

Could you please try running the install script 2 or 3 times, I noticed this issue with other versions (and cannot reproduce it) and re-running the install script fixes the issue

[typed-sigterm]

+1 from #6742, and nether re-running for 3 times nor | bash -s 4.0.0-beta.418 works

[JakeKydd]

Thanks for looking into this @ShadowArcanist. It is worrysome that you cannot reproduce it yourself, this will make it more difficult to identify the root cause I fear.

I tried rerunning the install script 6 times, nothing changed.

I believe the script runs till line 821 and then somehow terminates.

https://github.com/coollabsio/coolify/blob/v4.x/scripts/install.sh

if [[ $- == *x* ]]; then
    bash -x /data/coolify/source/upgrade.sh "${LATEST_VERSION:-latest}" "${LATEST_HELPER_VERSION:-latest}" "${REGISTRY_URL:-ghcr.io}" "true"
else
    bash /data/coolify/source/upgrade.sh "${LATEST_VERSION:-latest}" "${LATEST_HELPER_VERSION:-latest}" "${REGISTRY_URL:-ghcr.io}" "true"
fi
echo " - Coolify installed successfully."

So it should run the upgrade.sh file and then report back that coolify is installed, but it never does.
Is there a log for the upgrade.sh we could look at to see where that one terminates unexpectedly?

[JakeKydd]

Aha! I found the upgrade.sh logs in /data/coolify/source

And here's what the latest log says:

# cat /data/coolify/source/upgrade-2025-10-01-07-31-37.log
Merging .env.production values into .env
.env file merged successfully
Checking and updating environment variables if necessary...
Unable to find image 'ghcr.io/coollabsio/coolify-helper:1.0.11' locally
docker: Error response from daemon: Head "https://ghcr.io/v2/coollabsio/coolify-helper/manifests/1.0.11": denied: denied

So using cat ~/.docker/config.json I can see an auth entry for ghcr.io and proceeded to use docker logout, which did not remove the ghcr.io token. However using docker logout ghcr.io did remove the auth token successfully.

Running the install script after the docker logout command now results in success!

 - Coolify installed successfully.
 - Waiting 20 seconds for Coolify database migrations to complete.

So now one question remains so others can also get a better understanding for this issue. Why did the auth token created by the install script expire? Could the install script include a check to validate the auth token is valid, if not refetch a new one?

[Cinzya]

Why did the auth token created by the install script expire? Could the install script include a check to validate the auth token is valid, if not refetch a new one?

The installation script does not create an auth token, so your server was already authenticated prior.