Database is not healthy #1415
Comments
|
Could you try starting the database only? npx supabase stop --no-backup
npx supabase db startIf that still errors, please also share the postgres container logs here. You can read it from docker desktop or a second terminal: docker logs -f supabase_db_<project> |
|
I have the same issue on Windows 10. The container is automatically deleted before I can read any logs in Docker Desktop. |
I have updated the cli version to latest before doing it, v1.93.0 (used brew to install) I have some migrations already in folder, ig they were applied. But still got the error I also get error on db pull. |
Just run and quickly switch to another terminal, then run the logs command 👍🏻 |
|
@AyushSehrawat the db logs look normal. May be it just needs more time to boot up as there are 5 more migrations to run https://github.com/supabase/postgres/tree/15.1.0.103/migrations/db/migrations. I will increase the timeout to 30 seconds for you to give a try. npx supabase@beta db start |
|
Thanks @sweatybridge , it seems like the timeout was the issue, it's fixed now! Thanks |
|
Great to know! Our stable release happens every 2 weeks. The next one will be on 19 Sept. |
|
Thanks @AyushSehrawat. Yeah, I had to have a second terminal ready to go. @sweatybridge The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.
The database cluster will be initialized with this locale configuration:
provider: libc
LC_COLLATE: C.UTF-8
LC_CTYPE: C.UTF-8
LC_MESSAGES: en_US.UTF-8
LC_MONETARY: en_US.UTF-8
LC_NUMERIC: en_US.UTF-8
LC_TIME: en_US.UTF-8
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".
Data page checksums are disabled.
fixing permissions on existing directory /var/lib/postgresql/data ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
initdb: warning: enabling "trust" authentication for local connections
initdb: hint: You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb.
syncing data to disk ... ok
Success. You can now start the database server using:
pg_ctl -D /var/lib/postgresql/data -l logfile start
waiting for server to start.... 2023-09-05 16:36:21.574 UTC [58] LOG: pgaudit extension initialized
2023-09-05 16:36:21.615 UTC [58] LOG: pgsodium primary server secret key loaded
2023-09-05 16:36:21.803 UTC [58] LOG: starting PostgreSQL 15.1 (Ubuntu 15.1-1.pgdg20.04+1) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0, 64-bit
2023-09-05 16:36:21.931 UTC [58] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2023-09-05 16:36:22.344 UTC [67] LOG: database system was shut down at 2023-09-05 16:36:18 UTC
. 2023-09-05 16:36:22.615 UTC [58] LOG: database system is ready to accept connections
2023-09-05 16:36:22.616 UTC [70] LOG: TimescaleDB background worker launcher connected to shared catalogs
2023-09-05 16:36:22.622 UTC [72] LOG: pg_cron scheduler started
done
server started
/usr/local/bin/docker-entrypoint.sh: ignoring /docker-entrypoint-initdb.d/init-scripts
/usr/local/bin/docker-entrypoint.sh: running /docker-entrypoint-initdb.d/migrate.sh
/docker-entrypoint-initdb.d/migrate.sh: running /docker-entrypoint-initdb.d/init-scripts/00-schema.sql
CREATE ROLE
REVOKE
CREATE SCHEMA
CREATE FUNCTION
REVOKE
GRANT
/docker-entrypoint-initdb.d/migrate.sh: running /docker-entrypoint-initdb.d/init-scripts/00000000000000-initial-schema.sql
CREATE PUBLICATION
CREATE ROLE
ALTER ROLE
CREATE ROLE
CREATE ROLE
GRANT ROLE
CREATE SCHEMA
CREATE EXTENSION
CREATE EXTENSION
CREATE EXTENSION
CREATE ROLE
CREATE ROLE
CREATE ROLE
CREATE ROLE
GRANT ROLE
GRANT ROLE
GRANT ROLE
GRANT ROLE
GRANT
ALTER DEFAULT PRIVILEGES
ALTER DEFAULT PRIVILEGES
ALTER DEFAULT PRIVILEGES
GRANT
ALTER ROLE
ALTER DEFAULT PRIVILEGES
ALTER DEFAULT PRIVILEGES
ALTER DEFAULT PRIVILEGES
ALTER ROLE
ALTER ROLE
/docker-entrypoint-initdb.d/migrate.sh: running /docker-entrypoint-initdb.d/init-scripts/00000000000001-auth-schema.sql
CREATE SCHEMA
CREATE TABLE
CREATE INDEX
CREATE INDEX
COMMENT
CREATE TABLE
CREATE INDEX
CREATE INDEX
CREATE INDEX
COMMENT
CREATE TABLE
COMMENT
CREATE TABLE
CREATE INDEX
COMMENT
CREATE TABLE
COMMENT
INSERT 0 7
CREATE FUNCTION
CREATE FUNCTION
CREATE FUNCTION
GRANT
CREATE ROLE
GRANT
GRANT
GRANT
ALTER ROLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
/docker-entrypoint-initdb.d/migrate.sh: running /docker-entrypoint-initdb.d/init-scripts/00000000000002-storage-schema.sql
CREATE SCHEMA
GRANT
ALTER DEFAULT PRIVILEGES
ALTER DEFAULT PRIVILEGES
ALTER DEFAULT PRIVILEGES
CREATE TABLE
CREATE INDEX
CREATE TABLE
CREATE INDEX
CREATE INDEX
ALTER TABLE
CREATE FUNCTION
CREATE FUNCTION
CREATE FUNCTION
CREATE FUNCTION
CREATE TABLE
CREATE ROLE
GRANT
GRANT
GRANT
ALTER ROLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER FUNCTION
ALTER FUNCTION
ALTER FUNCTION
ALTER FUNCTION
/docker-entrypoint-initdb.d/migrate.sh: running /docker-entrypoint-initdb.d/init-scripts/00000000000003-post-setup.sql
ALTER ROLE
ALTER ROLE
CREATE FUNCTION
CREATE EVENT TRIGGER
COMMENT
CREATE FUNCTION
COMMENT
DO
CREATE ROLE
GRANT
GRANT
GRANT
GRANT
GRANT
GRANT
GRANT
GRANT
GRANT
GRANT
GRANT
GRANT
ALTER ROLE
/docker-entrypoint-initdb.d/migrate.sh: running /docker-entrypoint-initdb.d/migrations/00-extension.sql
psql:/docker-entrypoint-initdb.d/migrations/00-extension.sql:1: NOTICE: schema "extensions" already exists, skipping
CREATE SCHEMA
CREATE EXTENSION
/docker-entrypoint-initdb.d/migrate.sh: running /docker-entrypoint-initdb.d/migrations/10000000000000_demote-postgres.sql
GRANT
GRANT
GRANT
GRANT
GRANT
GRANT
GRANT
GRANT
GRANT
GRANT
GRANT
GRANT
GRANT
ALTER ROLE
/docker-entrypoint-initdb.d/migrate.sh: running /docker-entrypoint-initdb.d/migrations/20211115181400_update-auth-permissions.sql
GRANT
GRANT
GRANT
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
ALTER TABLE
GRANT
GRANT
GRANT
GRANT
ALTER DEFAULT PRIVILEGES
ALTER DEFAULT PRIVILEGES
ALTER DEFAULT PRIVILEGES
/docker-entrypoint-initdb.d/migrate.sh: running /docker-entrypoint-initdb.d/migrations/20211118015519_create-realtime-schema.sql
CREATE SCHEMA
/docker-entrypoint-initdb.d/migrate.sh: running /docker-entrypoint-initdb.d/migrations/20211122051245_update-realtime-permissions.sql
GRANT
GRANT
GRANT
GRANT
/docker-entrypoint-initdb.d/migrate.sh: running /docker-entrypoint-initdb.d/migrations/20211124212715_update-auth-owner.sql
ALTER FUNCTION
ALTER FUNCTION
ALTER FUNCTION
/docker-entrypoint-initdb.d/migrate.sh: running /docker-entrypoint-initdb.d/migrations/20211130151719_update-realtime-permissions.sql
ALTER DEFAULT PRIVILEGES
ALTER DEFAULT PRIVILEGES
ALTER DEFAULT PRIVILEGES
/docker-entrypoint-initdb.d/migrate.sh: running /docker-entrypoint-initdb.d/migrations/20220118070449_enable-safeupdate-postgrest.sql
ALTER ROLE
/docker-entrypoint-initdb.d/migrate.sh: running /docker-entrypoint-initdb.d/migrations/20220126121436_finer-postgrest-triggers.sql
psql:/docker-entrypoint-initdb.d/migrations/20220126121436_finer-postgrest-triggers.sql:3: NOTICE: event trigger "api_restart" does not exist, skipping
DROP EVENT TRIGGER
DROP FUNCTION
psql:/docker-entrypoint-initdb.d/migrations/20220126121436_finer-postgrest-triggers.sql:4: NOTICE: function extensions.notify_api_restart() does not exist, skipping
CREATE FUNCTION
CREATE FUNCTION
DROP EVENT TRIGGER
psql:/docker-entrypoint-initdb.d/migrations/20220126121436_finer-postgrest-triggers.sql:59: NOTICE: event trigger "pgrst_ddl_watch" does not exist, skipping
CREATE EVENT TRIGGER
psql:/docker-entrypoint-initdb.d/migrations/20220126121436_finer-postgrest-triggers.sql:64: NOTICE: event trigger "pgrst_drop_watch" does not exist, skipping
DROP EVENT TRIGGER
CREATE EVENT TRIGGER
/docker-entrypoint-initdb.d/migrate.sh: running /docker-entrypoint-initdb.d/migrations/20220224211803_fix-postgrest-supautils.sql
DO
/docker-entrypoint-initdb.d/migrate.sh: running /docker-entrypoint-initdb.d/migrations/20220317095840_pg_graphql.sql
CREATE SCHEMA
CREATE FUNCTION
GRANT
ALTER DEFAULT PRIVILEGES
ALTER DEFAULT PRIVILEGES
ALTER DEFAULT PRIVILEGES
ALTER DEFAULT PRIVILEGES
ALTER DEFAULT PRIVILEGES
ALTER DEFAULT PRIVILEGES
CREATE FUNCTION
psql:/docker-entrypoint-initdb.d/migrations/20220317095840_pg_graphql.sql:88: NOTICE: event trigger "issue_pg_graphql_access" does not exist, skipping
DROP EVENT TRIGGER
CREATE EVENT TRIGGER
COMMENT
CREATE FUNCTION
DROP EVENT TRIGGER
psql:/docker-entrypoint-initdb.d/migrations/20220317095840_pg_graphql.sql:141: NOTICE: event trigger "issue_graphql_placeholder" does not exist, skipping
CREATE EVENT TRIGGER
COMMENT
/docker-entrypoint-initdb.d/migrate.sh: running /docker-entrypoint-initdb.d/migrations/20220321174452_fix-postgrest-alter-type-event-trigger.sql
DROP EVENT TRIGGER
psql:/docker-entrypoint-initdb.d/migrations/20220321174452_fix-postgrest-alter-type-event-trigger.sql:3: NOTICE: event trigger "api_restart" does not exist, skipping
psql:/docker-entrypoint-initdb.d/migrations/20220321174452_fix-postgrest-alter-type-event-trigger.sql:4: NOTICE: function extensions.notify_api_restart() does not exist, skipping
DROP FUNCTION
CREATE FUNCTION
CREATE FUNCTION
DROP EVENT TRIGGER
CREATE EVENT TRIGGER
DROP EVENT TRIGGER
CREATE EVENT TRIGGER
/docker-entrypoint-initdb.d/migrate.sh: running /docker-entrypoint-initdb.d/migrations/20220322085208_gotrue-session-limit.sql
ALTER ROLE
/docker-entrypoint-initdb.d/migrate.sh: running /docker-entrypoint-initdb.d/migrations/20220404205710_pg_graphql-on-by-default.sql
CREATE FUNCTION
CREATE FUNCTION
CREATE FUNCTION
DROP EXTENSION
psql:/docker-entrypoint-initdb.d/migrations/20220404205710_pg_graphql-on-by-default.sql:142: NOTICE: extension "pg_graphql" does not exist, skipping
DO
/docker-entrypoint-initdb.d/migrate.sh: running /docker-entrypoint-initdb.d/migrations/20220609081115_grant-supabase-auth-admin-and-supabase-storage-admin-to-postgres.sql
GRANT ROLE
/docker-entrypoint-initdb.d/migrate.sh: running /docker-entrypoint-initdb.d/migrations/20220613123923_pg_graphql-pg-dump-perms.sql
CREATE FUNCTION
DROP EXTENSION
DO
/docker-entrypoint-initdb.d/migrate.sh: running /docker-entrypoint-initdb.d/migrations/20220713082019_pg_cron-pg_net-temp-perms-fix.sql
DO
DO
/docker-entrypoint-initdb.d/migrate.sh: running /docker-entrypoint-initdb.d/migrations/20221028101028_set_authenticator_timeout.sql
ALTER ROLE
/docker-entrypoint-initdb.d/migrate.sh: running /docker-entrypoint-initdb.d/migrations/20221103090837_revoke_admin.sql
REVOKE ROLE
/docker-entrypoint-initdb.d/migrate.sh: running /docker-entrypoint-initdb.d/migrations/20221207154255_create_pgsodium_and_vault.sql
DO
/docker-entrypoint-initdb.d/migrate.sh: running /docker-entrypoint-initdb.d/migrations/20230201083204_grant_auth_roles_to_postgres.sql
GRANT ROLE
/docker-entrypoint-initdb.d/migrate.sh: running /docker-entrypoint-initdb.d/migrations/20230224042246_grant_extensions_perms_for_postgres.sql
GRANT
GRANT
GRANT
ALTER DEFAULT PRIVILEGES
ALTER DEFAULT PRIVILEGES
ALTER DEFAULT PRIVILEGES
/docker-entrypoint-initdb.d/migrate.sh: running /docker-entrypoint-initdb.d/migrations/20230306081037_grant_pg_monitor_to_postgres.sql
GRANT ROLE
/docker-entrypoint-initdb.d/migrate.sh: running /docker-entrypoint-initdb.d/migrations/20230327032006_grant_auth_roles_to_supabase_storage_admin.sql
GRANT ROLE
/docker-entrypoint-initdb.d/migrate.sh: running /docker-entrypoint-initdb.d/migrations/20230529180330_alter_api_roles_for_inherit.sql
ALTER ROLE
ALTER ROLE
ALTER ROLE
GRANT ROLE
/docker-entrypoint-initdb.d/migrate.sh: running /etc/postgresql.schema.sql
ALTER DATABASE
ALTER DATABASE
ALTER ROLE
ALTER ROLE
ALTER ROLE
ALTER ROLE
ALTER ROLE
ALTER ROLE
CREATE SCHEMA
ALTER SCHEMA
CREATE SCHEMA
ALTER SCHEMA
BEGIN
CREATE EXTENSION
CREATE SCHEMA
GRANT
ALTER DEFAULT PRIVILEGES
ALTER DEFAULT PRIVILEGES
ALTER DEFAULT PRIVILEGES
CREATE TABLE
INSERT 0 1
CREATE TABLE
CREATE INDEX
CREATE INDEX
COMMENT
CREATE FUNCTION
DO
GRANT
GRANT
GRANT
ALTER ROLE
psql:/etc/postgresql.schema.sql:150: NOTICE: drop cascades to trigger secrets_encrypt_secret_trigger_secret on table vault.secrets
psql:/etc/postgresql.schema.sql:150: NOTICE: trigger "secrets_encrypt_secret_trigger_secret" for relation "vault.secrets" does not exist, skipping
psql:/etc/postgresql.schema.sql:150: NOTICE: about to masking role vault.secrets vault.decrypted_secrets
ALTER TABLE
psql:/etc/postgresql.schema.sql:151: NOTICE: drop cascades to trigger secrets_encrypt_secret_trigger_secret on table vault.secrets
psql:/etc/postgresql.schema.sql:151: NOTICE: trigger "secrets_encrypt_secret_trigger_secret" for relation "vault.secrets" does not exist, skipping
psql:/etc/postgresql.schema.sql:151: NOTICE: about to masking role vault.secrets vault.decrypted_secrets
ALTER TABLE
ALTER FUNCTION
GRANT ROLE
DO
DO
CREATE FUNCTION
COMMENT
DO
INSERT 0 1
ALTER FUNCTION
ALTER FUNCTION
REVOKE
GRANT
COMMIT
pg_stat_statements_reset
--------------------------
(1 row)
pg_stat_reset
---------------
(1 row)
/usr/local/bin/docker-entrypoint.sh: ignoring /docker-entrypoint-initdb.d/migrations
2023-09-05 16:36:47.100 UTC [58] LOG: received fast shutdown request
waiting for server to shut down.... 2023-09-05 16:36:47.221 UTC [58] LOG: aborting any active transactions
2023-09-05 16:36:47.221 UTC [72] LOG: pg_cron scheduler shutting down
2023-09-05 16:36:47.223 UTC [70] FATAL: terminating background worker "TimescaleDB Background Worker Launcher" due to administrator command
2023-09-05 16:36:47.224 UTC [58] LOG: background worker "logical replication launcher" (PID 73) exited with exit code 1
2023-09-05 16:36:47.225 UTC [58] LOG: background worker "TimescaleDB Background Worker Launcher" (PID 70) exited with exit code 1
2023-09-05 16:36:47.230 UTC [65] LOG: shutting down
2023-09-05 16:36:47.343 UTC [65] LOG: checkpoint starting: shutdown immediate |
|
Hm @Sammy-T yeah issues on my side too. It worked fine in the test folder I made. But the error occured after i logged in and linked the project, then tried to run. First time start,, got database is not healthy. Stopped with no backup, ran db start and smh it worked. Then i tried few times with stop start, db start,,, but it didn't work. But once even the supabase start worked and everything was fine. I am not sure what's the reason. Some logs on my side - It worked And db start command worked most of the times, issue was sometimes i also got this error on supabase start, everything seemed to work fine but in the end error, edited: just noticed that after the start --debug suddenly terminated with this error, the postgres db docker was not stopped. It was running @sweatybridge Sorry to ping you again, but please look into it. |
|
I've further increased timeout to 40s in cli beta release. Let me know if it works. Also sorry for having go back and forth on this one because the timeout value is machine dependent. I also considered checking the available cpu to determine a timeout value dynamically, but I don't want to make things too complicated for now. |
|
I have 4 cores (virtual 8), so i guess there is no available cpu issue on my end. If possible, you can introduce a optional Anyways, regarding the issue @sweatybridge , the first time i ran the command I don't know if it's a temporary bug or what, but i will let you know here if anything comes up 👍🏻 Thanks |
|
@sweatybridge There seems to be "database not healthy" on
Gives, and the logs of docker ended here abruptly. Seems to be the timeout issue too ig, can you increase and send a patch to beta 🙏🏻 |
|
@sweatybridge please look into it. Issue seems to be similar to timeout one, need to increase for shadow db too i guess. |
|
It should have been increased already https://github.com/supabase/cli/blob/main/internal/db/diff/migra.go#L176. Are you on |
Yes i am on beta channel only, i have shared the logs and Postgres logs above. |
|
@sweatybridge same issue still there. I pruned all docker and it's network. Here are the logs This the the shadow postgres logs It seems to me that it suddently ended on |
|
@sweatybridge Can you look into the issue? Not sure what's the issue if not the timeout |
|
having the same problem running
on windows 11. Sometimes the diffing works but most of the time i am getting database ins not healthy |
|
@sweatybridge I think it would probably be a good idea to re-open this issue's status while this continues to be unresolved. |
|
It still looks like a timeout issue to me. How much resources did you allocate to docker desktop?
|
|
@sweatybridge I don't use Docker Desktop, only cli commands. And yes, this looks like timeout issue but i am not sure if it took 40s for this to give database is not healthy. |
|
I have Docker configured to use WSL 2 backend which should have the following resources available: |
db diff is sometime working but fails most of the time with datase is not healthy |
It's probably due to timeout issue too i guess. You can see the logs, run db diff with --debug and in another terminal run |
Could you try running |
I will try in a few minutes 👍 |
|
@sweatybridge Error response from daemon: manifest for public.ecr.aws/supabase/postgrest:v11.2.0 not found: manifest unknown: Requested image not found Issue on supabase@beta (1.99.2). I guess due to new file called rest-version. Which version should i downgrade it to ? |
|
Using v11.1.0 for now |
|
1st run 2nd run But there is issue with db reset now The docker logs of |
|
Seems like 2m is a sweet spot for your machine. There's probably some performance problem with docker on your machine for start up to take this long. But since faster hosts are unaffected by this timeout, I will just increase it to 2m. |
|
@sweatybridge Idk from which time to which timeout is calculated on, but those command ran perfectly fine, supabase db diff worked without errors. See the another issue with db reset now, issue occurred in it just after 40+ seconds. Ig you need to increase db reset time too 🤔. I installed Docker using official guide so not sure what's the issue for it to be slow |
|
@sweatybridge Did you find the issue ? |
|
Still having the issue here Below is the info. Supabase CLI version: 1.99.5 Host resources: Docker version: Docker version 24.0.6, build ed223bc Docker Desktop version: 4.23.0 (120376) Docker config: More info: Logs: |
|
@monsoir do you have the db container logs too? That would be helpful to know when the container stopped. |
|
Sorry for your inconvenience @sweatybridge . It turns out there's a little tricky here. I have executed the command several times, but always prompts me no such container. When I nearly give up, something shows up. 🥲 It seems a lot of duplicated logs. command |
|
That looks like a new error indeed. I've created a separate issue to track it #1526 Appreciate your help in getting the logs as it was useful. Let's continue our discussion in the new thread. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Describe the bug
When trying to start supabase, it ends up giving 'database is not healthy'. Similar issues are mentioned in #1384 and #1383
To Reproduce
Steps to reproduce the behavior:
Expected behavior
The local development environment works without any kind of error
Screenshots
None
Desktop (please complete the following information):
Additional context
Before logs, some more detail
Here are detailed logs
Try 1: Logs here -> https://pastebin.com/PhYHXCyY (due to github max body limit I am unable to send, but it's log are same as the below ones)
Try 2 with @silentworks -> https://pastebin.com/17uZw2Vb (github max body limit issue) . This log is the most detailed out of all 4 try
Try 3 with @silentworks , disabled all the services in beta cli version
Try 4 with @silentworks , here I changed the port of database from 54322 to 54522
Some more info I would like to add is that, i switched to github codespaces which come with docker preinstalled. I ran npm install, then npx supabase init, login , start stuff,, and it worked perfectly fine.
The text was updated successfully, but these errors were encountered: