[Fixed] Upgrade from 14.3.0 to 14.10.0 fails with background migration error

[mikehaertl]

This is an FYI issue just in case someone has the same problem. It's similar to #2409 but with an upgrade from 14.3.0 to 14.10.0.

The error message was:

rake aborted!
StandardError: An error has occurred, this and all later migrations canceled:

Expected batched background migration for the given configuration to be marked as 'finished', but it is 'active':#11{:job_class_name=>"ProjectNamespaces::BackfillProjectNamespaces", :table_name=>:projects, :col
umn_name=>:id, :job_arguments=>[nil, "up"]}

Finalize it manualy by running

#011sudo gitlab-rake gitlab:background_migrations:finalize[ProjectNamespaces::BackfillProjectNamespaces,projects,id,'[null,"up"]']

For more information, check the documentation

#11https://docs.gitlab.com/ee/user/admin_area/monitoring/background_migrations.html#database-migrations-failing-because-of-batched-background-migration-not-finished

To apply the suggested fix (using docker-compose where I have a gitlab container):

• Stop the container
• Run docker-compose run --rm gitlab app:rake gitlab:background_migrations:finalize[ProjectNamespaces::BackfillProjectNamespaces,projects,id,'[null\,"up"]']

That fixed it for me. The command may be a little different but can be copied from the error message.

[VolanDeVovan]

Have same issue in cloud installation (helm)
But i cant solve it by manually run

git@gitlab-toolbox-8684fc585b-wb859:/$  gitlab-rake gitlab:background_migrations:finalize[ProjectNamespaces::BackfillProjectNamespaces,projects,id,'[null\,"up"]']
rake aborted!
Gitlab::Database::BackgroundMigration::BatchedMigrationRunner::FailedToFinalize: Gitlab::Database::BackgroundMigration::BatchedMigrationRunner::FailedToFinalize
/srv/gitlab/lib/gitlab/database/background_migration/batched_migration_runner.rb:76:in `finalize'
/srv/gitlab/lib/gitlab/database/background_migration/batched_migration_runner.rb:10:in `finalize'
/srv/gitlab/lib/tasks/gitlab/background_migrations.rake:70:in `finalize_migration'
/srv/gitlab/lib/tasks/gitlab/background_migrations.rake:18:in `block (3 levels) in <main>'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/sentry-ruby-core-5.1.1/lib/sentry/rake.rb:26:in `execute'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/rake-13.0.6/exe/rake:27:in `<top (required)>'
/srv/gitlab/bin/bundle:5:in `load'
/srv/gitlab/bin/bundle:5:in `<main>'
Tasks: TOP => gitlab:background_migrations:finalize
(See full trace by running task with --trace)

[Novoleg]

VolanDeVovan Thank you, your decision helped me.

[paolomainardi]

Running the command from a terminal, does not solve the issue.
It's something related to an open issue here: https://gitlab.com/gitlab-org/gitlab/-/issues/360377#note_935153082

cc @sachilles this is quite critical

[kkimurak]

We must wait for background migration at 14.9.x - this is clearly described in docs : https://docs.gitlab.com/ee/update/#1490

Do we need more kindful warning or else?

[paolomainardi]

@kkimurak yes but there are some cases (like mine) where for some reasons that specific migration fail in background too without any real evidenze, unless you check it here: https://gitlab.sparkfabrik.com/admin/background_migrations
I've also tried to run it manually but it fails and it is quite hard to debug, moreover as this procedure is quite uncommon would be better to advise the user to check that before migrate to 14.10, i lost hours to revert back to 14.9.

[kkimurak]

Hmm that too sad..

For example, there is a code to prompt user to upgrade to version 8.0.5-1 before upgrading to 8.1.0 or higher. How about running rake task to check migration status as well, before upgrading to 14.10.x ?

[sachilles]

Then, from my point of view, the adjustment of the release information would make sense. Or do you have any other suggestions?

[sachilles]

Ooooh dear, that sounds like a lot of warnings and related CI runs.

[paolomainardi]

Added here some info: https://gitlab.com/gitlab-org/gitlab/-/issues/360377#note_936850551

[paolomainardi]

I guess to have find the reason (but not the solution):

{
    "severity": "ERROR",
    "time": "2022-05-05T17:42:11.663Z",
    "correlation_id": "2efa593fe0d072a9fa1c8572d96124d0",
    "exception.class": "ActiveRecord::StatementInvalid",
    "exception.message": "PG::WrongObjectType: ERROR:  \"index_projects_on_id_partial_for_visibility\" is not a GIN index\n",
    "exception.backtrace": [
        "lib/gitlab/database/load_balancing/connection_proxy.rb:126:in `block in write_using_load_balancer'",
        "lib/gitlab/database/load_balancing/load_balancer.rb:112:in `block in read_write'",
        "lib/gitlab/database/load_balancing/load_balancer.rb:172:in `retry_with_backoff'",
        "lib/gitlab/database/load_balancing/load_balancer.rb:110:in `read_write'",
        "lib/gitlab/database/load_balancing/connection_proxy.rb:125:in `write_using_load_balancer'",
        "lib/gitlab/database/load_balancing/connection_proxy.rb:95:in `method_missing'",
        "lib/gitlab/background_migration/project_namespaces/backfill_project_namespaces.rb:63:in `block in cleanup_gin_index'",
        "lib/gitlab/background_migration/project_namespaces/backfill_project_namespaces.rb:62:in `each'",
        "lib/gitlab/background_migration/project_namespaces/backfill_project_namespaces.rb:62:in `cleanup_gin_index'",
        "lib/gitlab/background_migration/project_namespaces/backfill_project_namespaces.rb:41:in `block in backfill_project_namespaces'",
        "lib/gitlab/background_migration/project_namespaces/backfill_project_namespaces.rb:36:in `each'",
        "lib/gitlab/background_migration/project_namespaces/backfill_project_namespaces.rb:36:in `each_slice'",
        "lib/gitlab/background_migration/project_namespaces/backfill_project_namespaces.rb:36:in `backfill_project_namespaces'",
        "lib/gitlab/background_migration/project_namespaces/backfill_project_namespaces.rb:21:in `perform'",
        "lib/gitlab/database/background_migration/batched_migration_wrapper.rb:45:in `execute_batch'",
        "lib/gitlab/database/background_migration/batched_migration_wrapper.rb:23:in `perform'",
        "lib/gitlab/database/background_migration/batched_migration_runner.rb:30:in `run_migration_job'",
        "app/workers/database/batched_background_migration/single_database_worker.rb:66:in `run_active_migration'",
        "app/workers/database/batched_background_migration/single_database_worker.rb:54:in `block (2 levels) in perform'",
        "app/workers/database/batched_background_migration/single_database_worker.rb:77:in `with_exclusive_lease'",
        "app/workers/database/batched_background_migration/single_database_worker.rb:46:in `block in perform'",
        "lib/gitlab/database/shared_model.rb:23:in `using_connection'",
        "app/workers/database/batched_background_migration/single_database_worker.rb:43:in `perform'",
        "lib/gitlab/database/load_balancing/sidekiq_server_middleware.rb:26:in `call'",
        "lib/gitlab/sidekiq_middleware/duplicate_jobs/strategies/until_executing.rb:16:in `perform'",
        "lib/gitlab/sidekiq_middleware/duplicate_jobs/duplicate_job.rb:58:in `perform'",
        "lib/gitlab/sidekiq_middleware/duplicate_jobs/server.rb:8:in `call'",
        "lib/gitlab/application_context.rb:88:in `block in use'",
        "lib/gitlab/application_context.rb:88:in `use'",
        "lib/gitlab/sidekiq_middleware/worker_context.rb:11:in `wrap_in_optional_context'",
        "lib/gitlab/sidekiq_middleware/worker_context/server.rb:17:in `block in call'",
        "lib/gitlab/application_context.rb:88:in `block in use'",
        "lib/gitlab/application_context.rb:88:in `use'",
        "lib/gitlab/application_context.rb:41:in `with_context'",
        "lib/gitlab/sidekiq_middleware/worker_context/server.rb:15:in `call'",
        "lib/gitlab/sidekiq_status/server_middleware.rb:7:in `call'",
        "lib/gitlab/sidekiq_versioning/middleware.rb:9:in `call'",
        "lib/gitlab/sidekiq_middleware/query_analyzer.rb:7:in `block in call'",
        "lib/gitlab/database/query_analyzer.rb:46:in `within'",
        "lib/gitlab/sidekiq_middleware/query_analyzer.rb:7:in `call'",
        "lib/gitlab/sidekiq_middleware/admin_mode/server.rb:14:in `call'",
        "lib/gitlab/sidekiq_middleware/instrumentation_logger.rb:9:in `call'",
        "lib/gitlab/sidekiq_middleware/batch_loader.rb:7:in `call'",
        "lib/gitlab/sidekiq_middleware/extra_done_log_metadata.rb:7:in `call'",
        "lib/gitlab/sidekiq_middleware/request_store_middleware.rb:10:in `block in call'",
        "lib/gitlab/with_request_store.rb:17:in `enabling_request_store'",
        "lib/gitlab/with_request_store.rb:10:in `with_request_store'",
        "lib/gitlab/sidekiq_middleware/request_store_middleware.rb:9:in `call'",
        "lib/gitlab/sidekiq_middleware/server_metrics.rb:74:in `block in call'",
        "lib/gitlab/sidekiq_middleware/server_metrics.rb:97:in `block in instrument'",
        "lib/gitlab/metrics/background_transaction.rb:33:in `run'",
        "lib/gitlab/sidekiq_middleware/server_metrics.rb:97:in `instrument'",
        "lib/gitlab/sidekiq_middleware/server_metrics.rb:73:in `call'",
        "lib/gitlab/sidekiq_middleware/monitor.rb:10:in `block in call'",
        "lib/gitlab/sidekiq_daemon/monitor.rb:49:in `within_job'",
        "lib/gitlab/sidekiq_middleware/monitor.rb:9:in `call'",
        "lib/gitlab/sidekiq_middleware/size_limiter/server.rb:13:in `call'",
        "lib/gitlab/sidekiq_logging/structured_logger.rb:21:in `call'"
    ],
    "exception.sql": "/*application:sidekiq,correlation_id:2efa593fe0d072a9fa1c8572d96124d0,jid:84b97022cbd7e17a8ffe5a16,endpoint_id:Database::BatchedBackgroundMigrationWorker,db_config_name:main*/ select gin_clean_pending_list($1)",
    "user.username": null,
    "tags.program": "sidekiq",
    "tags.locale": "en",
    "tags.feature_category": "database",
    "tags.correlation_id": "2efa593fe0d072a9fa1c8572d96124d0",
    "extra.sidekiq": {
        "retry": false,
        "queue": "cronjob:database_batched_background_migration",
        "version": 0,
        "queue_namespace": "cronjob",
        "args": [],
        "class": "Database::BatchedBackgroundMigrationWorker",
        "jid": "84b97022cbd7e17a8ffe5a16",
        "created_at": 1651772531.4898553,
        "meta.caller_id": "Cronjob",
        "correlation_id": "2efa593fe0d072a9fa1c8572d96124d0",
        "meta.feature_category": "database",
        "worker_data_consistency": "always",
        "idempotency_key": "resque:gitlab:duplicate:cronjob:database_batched_background_migration:592d9619e1997b640b70ce6a22f6713bc7793bb7a4e342b7380d90b691fcd6ae",
        "enqueued_at": 1651772531.4918294
    },
    "extra.batched_job_id": 129
}

@sachilles @kkimurak it sparks you some ideas ?

[sachilles]

Hmm that too sad..

For example, there is a code to prompt user to upgrade to version 8.0.5-1 before upgrading to 8.1.0 or higher. How about running rake task to check migration status as well, before upgrading to 14.10.x ?

I did not know this test at all. Thanks for sharing.