A lot of page 500 when upgrading from 13.12.4 to 14.0.0

[sprucify]

I can upgrade up to gitlab version 13.12.4, but as soon as I go to 14.0.0 I'm facing a lot of page 500 errors.
When I turn 2FA off, it seems to work a bit better, but 2FA is required.

I tried to turn off 2FA in version 13.12.4 before starting the upgrade, but no luck.

The changes in my docker-compose file are:

• sameersbn/gitlab:13.12.4 to sameersbn/gitlab:14.0.0
• image: redis:5.0.9 to image: redis:6.2
• redis-data:/var/lib/redis:Z to redis-data:/data:Z

So a lot is changing on redis side.
Does the mapped redis volume maybe contain some previous 2FA cache?

I tried to remove and add 2FA, but this didn't trigger any cache clean up I think

[koubas]

Hi, I neded to set these envs:

• RACK_ATTACK_ENABLED=false (fixed 500s on login)
• prometheus_multiproc_dir=/dev/shm (fixed 500 in admin/metrics)

And I had to upgrade to the latest Redis because my ancient one didn't know the UNLINK command

[sprucify]

Thanks.
Did you turn off Rack Attack temporarily and turn it back on in later upgrades?

[kkimurak]

This should have fixed in #2426 . Could you try release sameersbn/gitlab:14.3.0 ? You don't need to set RACK_ATTACK_ENABLED=false.
BTW, 500 in admin/metrics (also reported in #2387) should be fixed. I'll check later.

[sprucify]

When jumping direct to release 14.3.0 I get this rake migrate error:
rake aborted! gitlab_1 | StandardError: An error has occurred, all later migrations canceled: gitlab_1 | gitlab_1 | Expected batched background migration for the given configuration to be marked as 'finished', but it is 'active': {:job_class_name=>"CopyColumnUsingBackgroundMigrationJob", :table_name=>"events", :column_name=>"id", :job_arguments=>[["id"], ["id_convert_to_bigint"]]}

Do I need to go to 14.0.0 first?

[kkimurak]

Yes of course. Please follow upgrade path. It says "latest 13.12.Z -> latest 14.0.Z -> latest 14.Y.Z" is supported.

[sprucify]

I tried to upgrade the following path and checked only if the login screen appeared:
13.12.4 -> 14.0.0 -> 14.0.6 -> 14.1.0 -> 14.1.3 -> 14.2.1

Between 14.1.3 and 14.2.1 I get the same rake aborted error and have no login screen

[kkimurak]

There are some background migration tasks after container up, so you should not stop container even you can see login screen. Make sure that all of tasks are finished (you can check the status in /admin/background_migrations ). Also check #2409

[sprucify]

I noticed 14.0.0 has a list of 5 active background migrations that seems to be stuck. All 0.00% after 1 hour up

[kkimurak]

Thanks for checking up, then you can run background migration manually. See https://docs.gitlab.com/ee/user/admin_area/monitoring/background_migrations.html#manually-finishing-a-batched-background-migration
Note that this repo provides source-instaled gitlab, so sudo gitlab-rake command should be replaced by docker exec -it your-gitlab-container-name bundle exec rake (or docker exec -it your-gitlab-container-name /bin/bash then bundle exec rake).

[sprucify]

After many OTP login attempts I got back in to view the list of waiting migrations:

• CopyColumnUsingBackgroundMigrationJob: ci_builds_metadata
• CopyColumnUsingBackgroundMigrationJob: events
• CopyColumnUsingBackgroundMigrationJob: push_event_payloads
• CopyColumnUsingBackgroundMigrationJob: ci_builds
• CopyColumnUsingBackgroundMigrationJob: ci_stages

After running bundle exec rake it outputs

Running the full GitLab test suite takes significant time to pass. We recommend using one of the following spec tasks:

Learn more at https://docs.gitlab.com/ee/development/rake_tasks.html#run-tests.

With two empty lines, so no information which task

[sprucify]

I think I got it working now.
It seems that 14.0.0 has difficulties with the background migrations in combination with 2FA

My steps:

• Disable 2FA in 13.12.4
• Upgrade to 14.0.0 and ignored the stuck background migrations
• Upgrade to 14.0.6 and background migrations started running and waited for it.
• Upgraded in multiple steps all the way to 14.3.1
• Turn on 2FA and added my yubikey