Install failed: relation "gitops_config" does not exist, How to fix this problem manually?

[xuanyuanaosheng]

I have install devtron using:

helm repo add devtron https://helm.devtron.ai
helm install devtron devtron/devtron-operator --create-namespace --namespace devtroncd \
--set secrets.POSTGRESQL_PASSWORD=change-me

But the install failed:

The component devtron erros is:

{"level":"info","ts":1627623041.078266,"caller":"models/connection.go:61","msg":"connected with db","db":{"Addr":"postgresql-postgresql.devtroncd","Port":"5432","User":"postgres","Password":"change_me","Database":"orchestrator","ApplicationName":"orchestrator","LogQuery":true}}
time="2021-07-30T05:30:41Z" level=info msg="Starting configmap/secret informers"
time="2021-07-30T05:30:41Z" level=info msg="Configmap/secret informer synced"
2021/07/30 05:30:41 ERROR #42P01 relation "gitops_config" does not exist
panic: ERROR #42P01 relation "gitops_config" does not exist
goroutine 1 [running]:
log.Panic(0xc00081ff68, 0x1, 0x1)
        /usr/local/go/src/log/log.go:351 +0xac
main.main()
        /go/src/github.com/devtron-labs/devtron/main.go:33 +0x76

The postgresql error is:

2021-07-30 04:43:57.824 GMT [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
2021-07-30 04:43:57.824 GMT [1] LOG:  listening on IPv6 address "::", port 5432
2021-07-30 04:43:57.828 GMT [1] LOG:  listening on Unix socket "/tmp/.s.PGSQL.5432"
2021-07-30 04:43:57.891 GMT [182] LOG:  database system shutdown was interrupted; last known up at 2021-07-30 04:43:48 GMT
2021-07-30 04:43:58.113 GMT [183] FATAL:  the database system is starting up
2021-07-30 04:44:03.272 GMT [182] LOG:  database system was not properly shut down; automatic recovery in progress
2021-07-30 04:44:03.286 GMT [182] LOG:  redo starts at 0/1655490
2021-07-30 04:44:03.286 GMT [182] LOG:  invalid record length at 0/16554C8: wanted 24, got 0
2021-07-30 04:44:03.286 GMT [182] LOG:  redo done at 0/1655490
2021-07-30 04:44:03.320 GMT [1] LOG:  database system is ready to accept connections
2021-07-30 04:44:21.070 GMT [208] ERROR:  relation "chart_repo" does not exist at character 255
2021-07-30 04:44:21.070 GMT [208] STATEMENT:  SELECT "chart_repo"."id", "chart_repo"."name", "chart_repo"."url", "chart_repo"."active", "chart_repo"."is_default", "chart_repo"."external", "chart_repo"."created_on", "chart_repo"."created_by", "chart_repo"."updated_on", "chart_repo"."updated_by" FROM chart_repo AS "chart_repo" WHERE (external = TRUE) AND (active =TRUE)
2021-07-30 04:44:28.314 GMT [217] ERROR:  database "casbin" already exists
2021-07-30 04:44:28.314 GMT [217] STATEMENT:  CREATE DATABASE casbin
2021-07-30 04:44:28.906 GMT [216] ERROR:  relation "gitops_config" does not exist at character 399
2021-07-30 04:44:28.906 GMT [216] STATEMENT:  SELECT "git_ops_config"."id", "git_ops_config"."provider", "git_ops_config"."username", "git_ops_config"."token", "git_ops_config"."gitlab_group_id", "git_ops_config"."github_org_id", "git_ops_config"."azure_project", "git_ops_config"."host", "git_ops_config"."active", "git_ops_config"."created_on", "git_ops_config"."created_by", "git_ops_config"."updated_on", "git_ops_config"."updated_by" FROM gitops_config AS "git_ops_config" WHERE (active = TRUE) LIMIT 1
2021-07-30 04:44:46.527 GMT [244] FATAL:  password authentication failed for user "root"
2021-07-30 04:44:46.527 GMT [244] DETAIL:  Role "root" does not exist.
        Connection matched pg_hba.conf line 1: "host     all             all             0.0.0.0/0               md5"
2021-07-30 04:44:55.256 GMT [252] ERROR:  database "casbin" already exists
2021-07-30 04:44:55.256 GMT [252] STATEMENT:  CREATE DATABASE casbin
2021-07-30 04:44:55.417 GMT [251] ERROR:  relation "gitops_config" does not exist at character 399
2021-07-30 04:44:55.417 GMT [251] STATEMENT:  SELECT "git_ops_config"."id", "git_ops_config"."provider", "git_ops_config"."username", "git_ops_config"."token", "git_ops_config"."gitlab_group_id", "git_ops_config"."github_org_id", "git_ops_config"."azure_project", "git_ops_config"."host", "git_ops_config"."active", "git_ops_config"."created_on", "git_ops_config"."created_by", "git_ops_config"."updated_on", "git_ops_config"."updated_by" FROM gitops_config AS "git_ops_config" WHERE (active = TRUE) LIMIT 1
2021-07-30 04:45:48.186 GMT [329] ERROR:  database "casbin" already exists
2021-07-30 04:45:48.186 GMT [329] STATEMENT:  CREATE DATABASE casbin
2021-07-30 04:45:48.254 GMT [328] ERROR:  relation "gitops_config" does not exist at character 399
2021-07-30 04:45:48.254 GMT [328] STATEMENT:  SELECT "git_ops_config"."id", "git_ops_config"."provider", "git_ops_config"."username", "git_ops_config"."token", "git_ops_config"."gitlab_group_id", "git_ops_config"."github_org_id", "git_ops_config"."azure_project", "git_ops_config"."host", "git_ops_config"."active", "git_ops_config"."created_on", "git_ops_config"."created_by", "git_ops_config"."updated_on", "git_ops_config"."updated_by" FROM gitops_config AS "git_ops_config" WHERE (active = TRUE) LIMIT 1
2021-07-30 04:46:20.042 GMT [207] ERROR:  relation "git_material" does not exist at character 501
2021-07-30 04:46:20.042 GMT [207] STATEMENT:  SELECT "git_material".*, "git_provider"."id" AS "git_provider__id", "git_provider"."name" AS "git_provider__name", "git_provider"."url" AS "git_provider__url", "git_provider"."user_name" AS "git_provider__user_name", "git_provider"."password" AS "git_provider__password", "git_provider"."ssh_key" AS "git_provider__ssh_key", "git_provider"."access_token" AS "git_provider__access_token", "git_provider"."auth_mode" AS "git_provider__auth_mode", "git_provider"."active" AS "git_provider__active" FROM git_material AS "git_material" LEFT JOIN git_provider AS "git_provider" ON "git_provider"."id" = "git_material"."git_provider_id" WHERE (deleted =FALSE ) AND (checkout_status=TRUE ) ORDER BY "id" ASC
2021-07-30 04:47:15.199 GMT [446] ERROR:  database "casbin" already exists

How to repair this problem 'relation "gitops_config" does not exist'?

[pghildiyal]

Migration jobs which create tables in database must have failed because of which you are seeing this error.

Please run kubectl -n devtroncd get job see the status of the jobs. Also check the logs of pods created by these jobs.

Usually it happens because of either of 2 reasons

1. Github is blocked because of which scripts couldn't be downloaded
2. Postgres password is wrong

[xuanyuanaosheng]

@pghildiyal Ok thanks.

It is true, the jobs has wrong status.

NAME                            COMPLETIONS   DURATION   AGE
app-sync-cronjob-1626692400     1/1           26s        22h
app-sync-job                    0/1           59s        59s
devtron-minio-make-bucket-job   0/1           48s        48s
postgresql-migrate-casbin       0/1           53s        53s
postgresql-migrate-gitsensor    0/1           51s        51s
postgresql-migrate-lens         0/1           42s        42s

I think it is the job problem. Now after rerun the jobs, the job status is as below, it is not right.

 kubectl -n devtroncd get job
NAME                           COMPLETIONS   DURATION   AGE
app-sync-cronjob-1626951600    1/1           13s        16h
app-sync-job                   1/1           2s         17h
postgresql-migrate-gitsensor   1/1           6s         17h
postgresql-migrate-lens        1/1           106s       17h

But I think the job still runs failed and the job has disapper.

I have rerun the job as your advice, But it is failed.

@pghildiyal How to fix this problem manually?

[xuanyuanaosheng]

I have reinstall it , It works.