chef-server-ctl upgrade fails at postgresql migrations (using sqitch)

[mathiasjakobsen]

When chef-server-ctl upgrade-ing, the following happens in the the [private-chef-upgrade] step..

bundler: failed to load command: ./bin/partybus (./bin/partybus)
RuntimeError: Partybus failed to execute command 'sqitch --engine pg --db-name opscode_chef --db-host 127.0.0.1 --db-port 5432 --db-user --top-dir /opt/opscode/embedded/service/opscode-erchef/schema deploy @2.5.3 --verify' exitstatus: 2

Trying to run the command manually shows that because no --db-user is specified, it reads the next arguments wrong, and interprets the --top-dir as binaries.

When removing --db-user, it fails with fe_sendauth: no password supplied.

See here for full log:

ok: down: bookshelf: 5416s, normally up
ok: down: nginx: 5579s, normally up
ok: down: oc_bifrost: 5415s, normally up
ok: down: oc_id: 5415s, normally up
ok: down: opscode-chef-mover: 5428s, normally up
ok: down: opscode-erchef: 5415s, normally up
ok: down: opscode-expander: 5414s, normally up
ok: down: opscode-expander-reindexer: 5576s, normally up
ok: down: opscode-pushy-server: 5575s, normally up
ok: down: opscode-reporting: 5575s, normally up
ok: down: opscode-solr4: 5413s, normally up
ok: down: postgresql: 0s, normally up
ok: down: rabbitmq: 1s, normally up
ok: down: redis_lb: 0s, normally up
ok: run: postgresql: (pid 24248) 0s
[private-chef-upgrade] - Latest Migration Available: 1.30
[private-chef-upgrade] - Migrations to Run: [#<Partybus::MigrationFile:0x000000011d6928 @path="/opt/opscode/embedded/upgrades/001/020_multi_key_schema_migration.rb", @major=1, @minor=20>, #<Partybus::MigrationFile:0x000000011d8c00 @path="/opt/opscode/embedded/upgrades/001/021_key_schema_migration_2.rb", @major=1, @minor=21>, #<Partybus::MigrationFile:0x000000011d9088 @path="/opt/opscode/embedded/upgrades/001/022_cbv_type_addition.rb", @major=1, @minor=22>, #<Partybus::MigrationFile:0x000000011d84d0 @path="/opt/opscode/embedded/upgrades/001/023_policyfile_schema_and_permissions.rb", @major=1, @minor=23>, #<Partybus::MigrationFile:0x000000011da528 @path="/opt/opscode/embedded/upgrades/001/024_cleanup_local_cache_path.rb", @major=1, @minor=24>, #<Partybus::MigrationFile:0x000000011d81d8 @path="/opt/opscode/embedded/upgrades/001/025_user_read_access.rb", @major=1, @minor=25>, #<Partybus::MigrationFile:0x000000011dbec8 @path="/opt/opscode/embedded/upgrades/001/026_global_admins_rename.rb", @major=1, @minor=26>, #<Partybus::MigrationFile:0x000000011dacf8 @path="/opt/opscode/embedded/upgrades/001/027_node_policyfile_fields.rb", @major=1, @minor=27>, #<Partybus::MigrationFile:0x000000011d99c0 @path="/opt/opscode/embedded/upgrades/001/028_server_admins_global_group.rb", @major=1, @minor=28>, #<Partybus::MigrationFile:0x000000011d9768 @path="/opt/opscode/embedded/upgrades/001/029_server_admins_existing_user_read_access.rb", @major=1, @minor=29>, #<Partybus::MigrationFile:0x000000011db568 @path="/opt/opscode/embedded/upgrades/001/030_actor_keys_access_group.rb", @major=1, @minor=30>]
[private-chef-upgrade] - Current Migration Version: 1.19
[private-chef-upgrade] - Starting Migration 1.20
[private-chef-upgrade] - 	Running Command: sqitch --engine pg --db-name opscode_chef --db-host 127.0.0.1 --db-port 5432 --db-user --top-dir /opt/opscode/embedded/service/opscode-erchef/schema deploy @2.5.3 --verify with options {:env=>{"PGPASSWORD"=>nil}}
Bareword "opt" not allowed while "strict subs" in use at (eval 469) line 1, <DATA> line 1.
Bareword "opscode" not allowed while "strict subs" in use at (eval 469) line 1, <DATA> line 1.
Bareword "embedded" not allowed while "strict subs" in use at (eval 469) line 1, <DATA> line 1.
Bareword "service" not allowed while "strict subs" in use at (eval 469) line 1, <DATA> line 1.
Bareword "opscode_erchef" not allowed while "strict subs" in use at (eval 469) line 1, <DATA> line 1.
Bareword "schema" not allowed while "strict subs" in use at (eval 469) line 1, <DATA> line 1.

bundler: failed to load command: ./bin/partybus (./bin/partybus)
RuntimeError: Partybus failed to execute command 'sqitch --engine pg --db-name opscode_chef --db-host 127.0.0.1 --db-port 5432 --db-user --top-dir /opt/opscode/embedded/service/opscode-erchef/schema deploy @2.5.3 --verify' exitstatus: 2
  /opt/opscode/embedded/service/partybus/lib/partybus/dsl_runner.rb:14:in `instance_eval'
  /opt/opscode/embedded/service/partybus/lib/partybus/migration_api/v1.rb:97:in `run_command'
  /opt/opscode/embedded/service/partybus/lib/partybus/migration_api/v1.rb:113:in `run_sqitch'
  (eval):6:in `block in run'
  /opt/opscode/embedded/service/partybus/lib/partybus/migration_api/v1.rb:14:in `instance_eval'
  /opt/opscode/embedded/service/partybus/lib/partybus/migration_api/v1.rb:14:in `initialize'
  /opt/opscode/embedded/service/partybus/lib/partybus/dsl_runner.rb:25:in `new'
  /opt/opscode/embedded/service/partybus/lib/partybus/dsl_runner.rb:25:in `define_upgrade'
  (eval):1:in `run'
  /opt/opscode/embedded/service/partybus/lib/partybus/dsl_runner.rb:14:in `instance_eval'
  /opt/opscode/embedded/service/partybus/lib/partybus/dsl_runner.rb:14:in `run'
  /opt/opscode/embedded/service/partybus/lib/partybus/migrations.rb:41:in `run_migration'
  /opt/opscode/embedded/service/partybus/bin/partybus:62:in `block in do_upgrade'
  /opt/opscode/embedded/service/partybus/bin/partybus:58:in `each'
  /opt/opscode/embedded/service/partybus/bin/partybus:58:in `do_upgrade'
  /opt/opscode/embedded/service/partybus/bin/partybus:79:in `block in <top (required)>'
  /opt/opscode/embedded/service/partybus/bin/partybus:70:in `open'
  /opt/opscode/embedded/service/partybus/bin/partybus:70:in `<top (required)>'

[mathiasjakobsen]

The hacky solution is to edit /opt/opscode/embedded/service/partybus/lib/partybus/migration_api/v1.rb.

def run_sqitch(target, service, opts = {})
        options = default_opts_for_service(service).merge(opts)
        command = <<-EOM.gsub(/\s+/," ").strip!
          sqitch --engine pg
            --db-name #{options[:database]}
            --db-host #{Partybus.config.postgres['vip']}
            --db-port #{Partybus.config.postgres['port']}
            --db-user EXPLICITLY SET USERNAME
            --top-dir /opt/opscode/embedded/service/#{options[:path]}
            deploy #{target} --verify
        EOM
        run_command(command, env: {"PGPASSWORD" => EXPLICITLY SET PASSWORD})
      end

[stevendanna]

From my reading of the code, it looks like what possibly could have been the case is that /etc/opscode/chef-server-running.json somehow didn't have the db_superuser info in it.

[srenatus]

@mathiasjakobsen has this been resolved? 😃

[mathiasjakobsen]

@srenatus I actually don't know ¯\(ツ)/¯ As I mentioned above I used a hacky solution to get around the problem, and havn't done an update since. But we can close the issue, and I'll reopen if I encounter it again.

[srenatus]

@mathiasjakobsen thanks for cleaning up! 👍

[bucm01]

I'm having the exact same issue, but the fix shown above is not working. I assume the user and password and single quoted strings?

[mathiasjakobsen]

@bucm01 as I recall, the username was not quoted, but the password was double quoted.