GitLib Not starting

[pnelsonsr]

It took a long time to finally get the audit message problem fixed with "--cap-add audit_write" which works. But now when I start the GitLab container I get new errors (that might be due to the fact that it has not been started since 2020-Dec. The errors seem to be associated with a database index missing. Any advice on how to fix this? I'd love to get GitLab working again....
---snip---
Loading /etc/docker-gitlab/runtime/env-defaults
Initializing logdir...
Initializing datadir...
Updating CA certificates...
Container TimeZone -> UTC
Installing configuration templates...
Configuring gitlab...
Configuring gitlab::database
Configuring gitlab::redis
Configuring gitlab::actioncable
Configuring gitlab::secrets...
Configuring gitlab::sidekiq...
Configuring gitlab::gitaly...
Configuring gitlab::monitoring...
Configuring gitlab::gitlab-workhorse...
Configuring gitlab::puma...
Configuring gitlab::timezone...
Configuring gitlab::rack_attack...
Configuring gitlab::ci...
Configuring gitlab::artifacts...
Configuring gitlab::packages...
Configuring gitlab::lfs...
Configuring gitlab::uploads...
Configuring gitlab::mattermost...
Configuring gitlab::project_features...
Configuring gitlab::smtp_settings...
Configuring gitlab::oauth...
Configuring gitlab::ldap...
Configuring gitlab::cron_jobs...
Configuring gitlab::backups...
Configuring gitlab::registry...
Configuring gitlab::pages...
Configuring gitlab::sentry...
Configuring gitlab::content_security_policy...
Configuring gitlab-shell...
Configuring nginx...
Configuring nginx::gitlab...
Configuring nginx::gitlab::ssl...
Configuring nginx::gitlab::hsts...
Migrating database...
2022-03-09 18:26:47,932 CRIT Supervisor is running as root. Privileges were not dropped because no user is specified in the config file. If you intend to run as root, you can set user=root in the config file to avoid this message.
2022-03-09 18:26:47,932 INFO Included extra file "/etc/supervisor/conf.d/cron.conf" during parsing
2022-03-09 18:26:47,932 INFO Included extra file "/etc/supervisor/conf.d/gitaly.conf" during parsing
2022-03-09 18:26:47,934 INFO Included extra file "/etc/supervisor/conf.d/gitlab-workhorse.conf" during parsing
2022-03-09 18:26:47,934 INFO Included extra file "/etc/supervisor/conf.d/groups.conf" during parsing
2022-03-09 18:26:47,935 INFO Included extra file "/etc/supervisor/conf.d/mail_room.conf" during parsing
2022-03-09 18:26:47,935 INFO Included extra file "/etc/supervisor/conf.d/nginx.conf" during parsing
2022-03-09 18:26:47,936 INFO Included extra file "/etc/supervisor/conf.d/puma.conf" during parsing
2022-03-09 18:26:47,936 INFO Included extra file "/etc/supervisor/conf.d/sidekiq.conf" during parsing
2022-03-09 18:26:47,936 INFO Included extra file "/etc/supervisor/conf.d/sshd.conf" during parsing
2022-03-09 18:26:47,943 INFO RPC interface 'supervisor' initialized
2022-03-09 18:26:47,943 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2022-03-09 18:26:47,944 INFO supervisord started with pid 2035
2022-03-09 18:26:48,947 INFO spawned: 'gitaly' with pid 2055
2022-03-09 18:26:48,951 INFO spawned: 'puma' with pid 2056
2022-03-09 18:26:48,954 INFO spawned: 'gitlab-workhorse' with pid 2057
2022-03-09 18:26:48,962 INFO spawned: 'sidekiq' with pid 2058
2022-03-09 18:26:48,965 INFO spawned: 'sshd' with pid 2059
2022-03-09 18:26:48,979 INFO spawned: 'nginx' with pid 2060
2022-03-09 18:26:48,996 INFO spawned: 'cron' with pid 2064
2022-03-09 18:26:50,049 INFO success: gitaly entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2022-03-09 18:26:50,050 INFO success: puma entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2022-03-09 18:26:50,050 INFO success: gitlab-workhorse entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2022-03-09 18:26:50,050 INFO success: sidekiq entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2022-03-09 18:26:50,050 INFO success: sshd entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2022-03-09 18:26:50,050 INFO success: nginx entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2022-03-09 18:26:50,050 INFO success: cron entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2022-03-09 18:27:32,443 INFO exited: puma (exit status 1; not expected)
rake aborted!
ActiveRecord::IrreversibleOrderError: Relation has no current order and table has no primary key to be used as default order
/home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.6/lib/active_record/relation/query_methods.rb:1334:in reverse_sql_order' /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.6/lib/active_record/relation/query_methods.rb:1015:in reverse_order!'
/home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.6/lib/active_record/relation/finder_methods.rb:151:in last' /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.6/lib/active_record/querying.rb:22:in last'
/home/git/gitlab/app/models/concerns/cacheable_attributes.rb:19:in current_without_cache' /home/git/gitlab/app/models/concerns/cacheable_attributes.rb:63:in rescue in current'
/home/git/gitlab/app/models/concerns/cacheable_attributes.rb:51:in current' /home/git/gitlab/lib/gitlab/current_settings.rb:60:in uncached_application_settings'
/home/git/gitlab/lib/gitlab/current_settings.rb:42:in ensure_application_settings!' /home/git/gitlab/lib/gitlab/current_settings.rb:15:in block in current_application_settings'
/home/git/gitlab/lib/gitlab/null_request_store.rb:34:in fetch' /home/git/gitlab/lib/gitlab/safe_request_store.rb:12:in fetch'
/home/git/gitlab/lib/gitlab/current_settings.rb:15:in current_application_settings' /home/git/gitlab/lib/gitlab/current_settings.rb:32:in method_missing'
/home/git/gitlab/lib/gitlab/metrics/prometheus.rb:98:in prometheus_metrics_enabled_unmemoized' /home/git/gitlab/lib/gitlab/metrics/prometheus.rb:24:in block in prometheus_metrics_enabled?'
/home/git/gitlab/lib/gitlab/utils/strong_memoize.rb:28:in strong_memoize' /home/git/gitlab/lib/gitlab/metrics/prometheus.rb:23:in prometheus_metrics_enabled?'
/home/git/gitlab/config/initializers/7_prometheus_metrics.rb:49:in <top (required)>' /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/railties-6.1.4.6/lib/rails/engine.rb:681:in load'
/home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/railties-6.1.4.6/lib/rails/engine.rb:681:in block in load_config_initializer' /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.4.6/lib/active_support/notifications.rb:205:in instrument'
/home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/railties-6.1.4.6/lib/rails/engine.rb:680:in load_config_initializer' /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/railties-6.1.4.6/lib/rails/engine.rb:634:in block (2 levels) in class:Engine'
/home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/railties-6.1.4.6/lib/rails/engine.rb:633:in each' /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/railties-6.1.4.6/lib/rails/engine.rb:633:in block in class:Engine'
/home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/railties-6.1.4.6/lib/rails/initializable.rb:32:in instance_exec' /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/railties-6.1.4.6/lib/rails/initializable.rb:32:in run'
/home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/railties-6.1.4.6/lib/rails/initializable.rb:61:in block in run_initializers' /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/railties-6.1.4.6/lib/rails/initializable.rb:50:in each'
/home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/railties-6.1.4.6/lib/rails/initializable.rb:50:in tsort_each_child' /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/railties-6.1.4.6/lib/rails/initializable.rb:60:in run_initializers'
/home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/railties-6.1.4.6/lib/rails/application.rb:391:in initialize!' /home/git/gitlab/config/environment.rb:7:in <top (required)>'
/home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/zeitwerk-2.5.4/lib/zeitwerk/kernel.rb:35:in require' /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/zeitwerk-2.5.4/lib/zeitwerk/kernel.rb:35:in require'
/home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.4.6/lib/active_support/dependencies.rb:332:in block in require' /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.4.6/lib/active_support/dependencies.rb:299:in load_dependency'
/home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.4.6/lib/active_support/dependencies.rb:332:in require' /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/railties-6.1.4.6/lib/rails/application.rb:367:in require_environment!'
/home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/railties-6.1.4.6/lib/rails/application.rb:533:in block in run_tasks_blocks' /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/rake-13.0.6/exe/rake:27:in <top (required)>'
/usr/local/bin/bundle:23:in load' /usr/local/bin/bundle:23:in

'

Caused by:
ActiveRecord::IrreversibleOrderError: Relation has no current order and table has no primary key to be used as default order
/home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.6/lib/active_record/relation/query_methods.rb:1334:in reverse_sql_order' /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.6/lib/active_record/relation/query_methods.rb:1015:in reverse_order!'
/home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.6/lib/active_record/relation/finder_methods.rb:151:in last' /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.6/lib/active_record/querying.rb:22:in last'
/home/git/gitlab/app/models/concerns/cacheable_attributes.rb:19:in current_without_cache' /home/git/gitlab/app/models/concerns/cacheable_attributes.rb:55:in current'
/home/git/gitlab/lib/gitlab/current_settings.rb:60:in uncached_application_settings' /home/git/gitlab/lib/gitlab/current_settings.rb:42:in ensure_application_settings!'
/home/git/gitlab/lib/gitlab/current_settings.rb:15:in block in current_application_settings' /home/git/gitlab/lib/gitlab/null_request_store.rb:34:in fetch'
/home/git/gitlab/lib/gitlab/safe_request_store.rb:12:in fetch' /home/git/gitlab/lib/gitlab/current_settings.rb:15:in current_application_settings'
/home/git/gitlab/lib/gitlab/current_settings.rb:32:in method_missing' /home/git/gitlab/lib/gitlab/metrics/prometheus.rb:98:in prometheus_metrics_enabled_unmemoized'
/home/git/gitlab/lib/gitlab/metrics/prometheus.rb:24:in block in prometheus_metrics_enabled?' /home/git/gitlab/lib/gitlab/utils/strong_memoize.rb:28:in strong_memoize'
/home/git/gitlab/lib/gitlab/metrics/prometheus.rb:23:in prometheus_metrics_enabled?' /home/git/gitlab/config/initializers/7_prometheus_metrics.rb:49:in <top (required)>'
/home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/railties-6.1.4.6/lib/rails/engine.rb:681:in load' /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/railties-6.1.4.6/lib/rails/engine.rb:681:in block in load_config_initializer'
/home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.4.6/lib/active_support/notifications.rb:205:in instrument' /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/railties-6.1.4.6/lib/rails/engine.rb:680:in load_config_initializer'
/home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/railties-6.1.4.6/lib/rails/engine.rb:634:in block (2 levels) in <class:Engine>' /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/railties-6.1.4.6/lib/rails/engine.rb:633:in each'
/home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/railties-6.1.4.6/lib/rails/engine.rb:633:in block in <class:Engine>' /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/railties-6.1.4.6/lib/rails/initializable.rb:32:in instance_exec'
/home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/railties-6.1.4.6/lib/rails/initializable.rb:32:in run' /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/railties-6.1.4.6/lib/rails/initializable.rb:61:in block in run_initializers'
/home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/railties-6.1.4.6/lib/rails/initializable.rb:50:in each' /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/railties-6.1.4.6/lib/rails/initializable.rb:50:in tsort_each_child'
/home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/railties-6.1.4.6/lib/rails/initializable.rb:60:in run_initializers' /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/railties-6.1.4.6/lib/rails/application.rb:391:in initialize!'
/home/git/gitlab/config/environment.rb:7:in <top (required)>' /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/zeitwerk-2.5.4/lib/zeitwerk/kernel.rb:35:in require'
/home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/zeitwerk-2.5.4/lib/zeitwerk/kernel.rb:35:in require' /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.4.6/lib/active_support/dependencies.rb:332:in block in require'
/home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.4.6/lib/active_support/dependencies.rb:299:in load_dependency' /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.4.6/lib/active_support/dependencies.rb:332:in require'
/home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/railties-6.1.4.6/lib/rails/application.rb:367:in require_environment!' /home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/railties-6.1.4.6/lib/rails/application.rb:533:in block in run_tasks_blocks'
/home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/rake-13.0.6/exe/rake:27:in <top (required)>' /usr/local/bin/bundle:23:in load'
/usr/local/bin/bundle:23:in `

'
Tasks: TOP => db:migrate => db:load_config => environment
(See full trace by running task with --trace)
---snip---

[sachilles]

Hi,

to be able to help you, we need more information. Could you provide these?

• Which version of GitLab CE (i.e. which docker image) are you using?
• What version of PostgreSQL are you using? Is it running externally or as a container?
• If relevant, what does your docker-compose file look like?

[pnelsonsr]

1. The GitLab Docker image is the latest. It will not start so it's hard to say:
   docker images -> docker.io/sameersbn/gitlab latest e9df05fee17f 4 hours ago 3.05 GB
2. It is an external PostgreSQL server (not a docker) running version -> PostgreSQL 13.4 on x86_64-redhat-linux-gnu
3. I do not use a docker-compose but here is the run: (note: removed some data inside <>)
   docker run --detach=true --name=gitlab --hostname=gitlab.<example.com> --publish=10022:22 --publish=10443:443 --env-file=/etc/gitlab.conf -v /var/lib/gitlab/data:/home/git/data --cap-add audit_write docker.io/sameersbn/gitlab

[sachilles]

Hi,

regarding your information you provide, there might be some problems. Let's go step by step throught your config.

1. In /home/git/data/tmp there should be a file called VERSION. What is the content of this file.
2. You should keep in mind, that our provided images have PostgreSQL 12 baked in. If you want to use PostgreSQL 13, then you'll need to update the Dockerfile (see

   docker-gitlab/Dockerfile

   Line 47 in 07d058f

   nginx openssh-server postgresql-client-12 postgresql-contrib-12 redis-tools \

   ) and build the image by yourself. Otherwise some functionalities (I guess at least the backup won't work).

In addition I have one question. Did you recently upgrade the image (if yes, from which version did you upgrade) or do you stayed so far with the same image?

[pnelsonsr]

1. It's currently version 3.6.1
2. I have used an external PostgreSQL Database for years. I backup from the external. So Pg 12 will work but Pg 13 won't?
3. I check for and upgrade the docker image every week. I do not remember when it stopped working but it was over a year ago. I moved to bitbucket online as this was our backup strategy and got promoted when GitLab stopped working.

I like GitLab and would like to get it working again.

[sachilles]

Hi,

1. Are your really sure? Based on your information I would expect to see something like 14.8.2 (or later) could you check this once more? (Or did you mean version 13.6.1?) I guess we need to find to last working version in your case and a working db dump in order to safely migrate to the latest version.

2. I strongly recommend to use the same verison of binaries by using PostgreSQL. In principle different major versions of PostgreSQL could be incompatible und thus, potentially errors could appear. Even by creating or restoring a database dump with tools of different versions I would expect problems. (Since in this image PostgreSQL in version 12 is used, you should be careful.)

3. Hmm, I see. Although we are not using the official omnibus image in this project, I still came across an older issue related to the omnibus image (https://gitlab.com/gitlab-org/gitlab/-/issues/36405). Do you know this post?

[pnelsonsr]

Yes 13.6.1!

[sachilles]

Can you please try to use the Docker image in version 13.6.1? After that, please let me know if your instance is running again.

[pnelsonsr]

I tried it with 13.6.1 and had pretty much the same result. It exits after about a minute. The output is almost exactly the same as the log listed above where it says Rake Aborted!

[pnelsonsr]

@sachilles I think it may have some correlation to the older issue. It was awhile ago that it stopped working. I will check the application_settings DB to see if it has a PK and try to apply it with ALTER TABLE application_settings ADD CONSTRAINT application_settings_pkey PRIMARY KEY (id);

[pnelsonsr]

That only brought on more errors. So it was time to restart. I created a blank database and external directory and started over. Thanks for the help!

[pnelsonsr]

Closing as I started over