Skip to content

Restoring to an un-configured appliance fails due to a missing license file #449

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 3 commits into from
Nov 8, 2018
Merged

Restoring to an un-configured appliance fails due to a missing license file #449

merged 3 commits into from
Nov 8, 2018

Conversation

taz
Copy link
Member

@taz taz commented Nov 8, 2018 •
edited
Loading

When restoring to an un-configured appliance, ghe-restore is unable to progress past the ghe-restore-storage step. The restore of storage data fails because the storage-cluster-restore-routes script on the appliance is unable to connect to the MySQL database.

"keiko_auth_recv: 1045 Access denied for user 'github'@'localhost' (using password: YES)"
ghe-restore verbose output 
$ bin/ghe-restore -c -v -f -s /opt/backup-utils/data/20181108T152539 x.x.x.x
Checking for leaked keys in the backup snapshot that is being restored ...
* No leaked keys found
Connect x.x.x.x:122 OK (v2.15.1)
Starting restore of x.x.x.x:122 with backup-utils v2.15.0 from snapshot 20181108T152539
Stopping cron and github-timerd ...
Restoring settings ...
 --> Importing settings data /data/user/common/github.conf...
Error: no license file found (/data/user/common/enterprise.ghl)
Restoring license ...
Restoring management console password ...
Restoring CA certificates ...
 --> Importing custom CA certificates...
 --> Installing CA certificate to /usr/local/share/ca-certificates/Let's_Encrypt_Authority_X3_0A0141420000015385736A0B85ECA708.crt...
 --> Updating CA certificates...
 --> Done.
Restoring UUID ...
Restoring MySQL database ...
 --> Importing MySQL data...
Restoring Redis database ...
 --> Importing redis data...
 --> Starting redis...
Restoring Git repositories and Gists ...
building file list ... done
./
c/
c/nw/
c/nw/c4/
c/nw/c4/ca/
c/nw/c4/ca/42/
c/nw/c4/ca/42/1/
c/nw/c4/ca/42/1/1.git/
c/nw/c4/ca/42/1/1.git/HEAD
c/nw/c4/ca/42/1/1.git/audit_log
c/nw/c4/ca/42/1/1.git/config
c/nw/c4/ca/42/1/1.git/description
c/nw/c4/ca/42/1/1.git/dgit-state
c/nw/c4/ca/42/1/1.git/dgit-state.flock
c/nw/c4/ca/42/1/1.git/hooks -> /data/github/current/lib/git-core/fi-template/hooks
c/nw/c4/ca/42/1/1.git/language-stats.cache
c/nw/c4/ca/42/1/1.git/info/
c/nw/c4/ca/42/1/1.git/info/nwo
c/nw/c4/ca/42/1/1.git/logs/
c/nw/c4/ca/42/1/1.git/logs/HEAD
c/nw/c4/ca/42/1/1.git/logs/refs/
c/nw/c4/ca/42/1/1.git/logs/refs/heads/
c/nw/c4/ca/42/1/1.git/logs/refs/heads/master
c/nw/c4/ca/42/1/1.git/objects/
c/nw/c4/ca/42/1/1.git/objects/c9/
c/nw/c4/ca/42/1/1.git/objects/c9/0692700d1ac0fb4580b3fa1d6e9b6aaccc09a8
c/nw/c4/ca/42/1/1.git/objects/ee/
c/nw/c4/ca/42/1/1.git/objects/ee/0656d545fa9be234f1d27e69ce820fc6fcf359
c/nw/c4/ca/42/1/1.git/objects/f9/
c/nw/c4/ca/42/1/1.git/objects/f9/e261d5a038403639c4bb06387479efa955166f
c/nw/c4/ca/42/1/1.git/objects/info/
c/nw/c4/ca/42/1/1.git/objects/pack/
c/nw/c4/ca/42/1/1.git/refs/
c/nw/c4/ca/42/1/1.git/refs/heads/
c/nw/c4/ca/42/1/1.git/refs/heads/master
c/nw/c4/ca/42/1/1.git/refs/tags/
info/
info/nw-layout
info/svn-v4-upgraded

sent 4192 bytes  received 516 bytes  3138.67 bytes/sec
total size is 2467  speedup is 0.52
Restoring GitHub Pages ...
building file list ... done
./
.dpages-layout
0/
1/
2/
3/
4/
5/
6/
7/

sent 251 bytes  received 96 bytes  694.00 bytes/sec
total size is 0  speedup is 0.00
Restoring SSH authorized keys ...
 --> Importing SSH authorized keys...
Restoring storage data ...
/data/github/d8fb9827682f1bd883bef46a1b5ebb95a49a313c/vendor/internal-gems/activerecord-githubmysql2-adapter/lib/active_record/connection_adapters/githubmysql2_adapter.rb:598:in `initialize': keiko_auth_recv: 1045 Access denied for user 'github'@'localhost' (using password: YES) (Keiko::MysqlError)
	from /data/github/d8fb9827682f1bd883bef46a1b5ebb95a49a313c/vendor/internal-gems/activerecord-githubmysql2-adapter/lib/active_record/connection_adapters/githubmysql2_adapter.rb:598:in `new'
	from /data/github/d8fb9827682f1bd883bef46a1b5ebb95a49a313c/vendor/internal-gems/activerecord-githubmysql2-adapter/lib/active_record/connection_adapters/githubmysql2_adapter.rb:598:in `block in build_keiko'
	from /data/github/d8fb9827682f1bd883bef46a1b5ebb95a49a313c/lib/github/config/stats.rb:163:in `time'
	from /data/github/d8fb9827682f1bd883bef46a1b5ebb95a49a313c/vendor/internal-gems/activerecord-githubmysql2-adapter/lib/active_record/connection_adapters/githubmysql2_adapter.rb:582:in `build_keiko'
	from /data/github/d8fb9827682f1bd883bef46a1b5ebb95a49a313c/vendor/internal-gems/activerecord-githubmysql2-adapter/lib/active_record/connection_adapters/githubmysql2_adapter.rb:526:in `connect'
	from /data/github/d8fb9827682f1bd883bef46a1b5ebb95a49a313c/vendor/internal-gems/activerecord-githubmysql2-adapter/lib/active_record/connection_adapters/githubmysql2_adapter.rb:711:in `execute_raw'
	from /data/github/d8fb9827682f1bd883bef46a1b5ebb95a49a313c/vendor/internal-gems/activerecord-githubmysql2-adapter/lib/active_record/connection_adapters/githubmysql2_adapter.rb:669:in `block in execute_with_retry'
	from /data/github/d8fb9827682f1bd883bef46a1b5ebb95a49a313c/vendor/internal-gems/activerecord-githubmysql2-adapter/lib/active_record/connection_adapters/githubmysql2_adapter.rb:764:in `query_with_retry'
	from /data/github/d8fb9827682f1bd883bef46a1b5ebb95a49a313c/vendor/internal-gems/activerecord-githubmysql2-adapter/lib/active_record/connection_adapters/githubmysql2_adapter.rb:668:in `execute_with_retry'
	from /data/github/d8fb9827682f1bd883bef46a1b5ebb95a49a313c/vendor/internal-gems/activerecord-githubmysql2-adapter/lib/active_record/connection_adapters/githubmysql2_adapter.rb:254:in `block in execute'
	from /data/github/d8fb9827682f1bd883bef46a1b5ebb95a49a313c/vendor/internal-gems/activerecord-githubmysql2-adapter/lib/active_record/connection_adapters/githubmysql2_adapter.rb:757:in `execute_with_scanner'
	from /data/github/d8fb9827682f1bd883bef46a1b5ebb95a49a313c/vendor/internal-gems/activerecord-githubmysql2-adapter/lib/active_record/connection_adapters/githubmysql2_adapter.rb:253:in `execute'
	from /data/github/d8fb9827682f1bd883bef46a1b5ebb95a49a313c/vendor/internal-gems/activerecord-githubmysql2-adapter/lib/active_record/connection_adapters/githubmysql2_adapter.rb:651:in `block (2 levels) in select'
	from /data/github/d8fb9827682f1bd883bef46a1b5ebb95a49a313c/vendor/internal-gems/activerecord-githubmysql2-adapter/lib/active_record/connection_adapters/githubmysql2_adapter.rb:764:in `query_with_retry'
	from /data/github/d8fb9827682f1bd883bef46a1b5ebb95a49a313c/vendor/internal-gems/activerecord-githubmysql2-adapter/lib/active_record/connection_adapters/githubmysql2_adapter.rb:650:in `block in select'
	from /data/github/d8fb9827682f1bd883bef46a1b5ebb95a49a313c/vendor/internal-gems/activerecord-githubmysql2-adapter/lib/active_record/connection_adapters/githubmysql2_adapter.rb:757:in `execute_with_scanner'
	from /data/github/d8fb9827682f1bd883bef46a1b5ebb95a49a313c/vendor/internal-gems/activerecord-githubmysql2-adapter/lib/active_record/connection_adapters/githubmysql2_adapter.rb:649:in `select'
	from /data/github/d8fb9827682f1bd883bef46a1b5ebb95a49a313c/vendor/gems/2.4.4/ruby/2.4.0/gems/activerecord-5.2.1.acbe726/lib/active_record/connection_adapters/abstract/database_statements.rb:57:in `select_all'
	from /data/github/d8fb9827682f1bd883bef46a1b5ebb95a49a313c/vendor/gems/2.4.4/ruby/2.4.0/gems/activerecord-5.2.1.acbe726/lib/active_record/connection_adapters/abstract/query_cache.rb:101:in `select_all'
	from /data/github/d8fb9827682f1bd883bef46a1b5ebb95a49a313c/vendor/gems/2.4.4/ruby/2.4.0/gems/github-ds-0.2.9/lib/github/sql.rb:242:in `block in results'
	from /data/github/d8fb9827682f1bd883bef46a1b5ebb95a49a313c/vendor/gems/2.4.4/ruby/2.4.0/gems/github-ds-0.2.9/lib/github/sql.rb:411:in `enforce_timezone'
	from /data/github/d8fb9827682f1bd883bef46a1b5ebb95a49a313c/vendor/gems/2.4.4/ruby/2.4.0/gems/github-ds-0.2.9/lib/github/sql.rb:229:in `results'
	from /data/github/d8fb9827682f1bd883bef46a1b5ebb95a49a313c/lib/github/storage/allocator.rb:81:in `get_hosts'
	from /data/github/d8fb9827682f1bd883bef46a1b5ebb95a49a313c/script/storage-cluster-restore-routes:25:in `main'
	from /data/github/d8fb9827682f1bd883bef46a1b5ebb95a49a313c/script/storage-cluster-restore-routes:69:in `<main>'

We first see something going wrong when the message Error: no license file found (/data/user/common/enterprise.ghl) is shown. This corresponds with an early ghe-single-config-apply termination recorded in the ghe-config.log on the appliance:

$ cat /data/user/common/ghe-config.log
Current version information:
RELEASE_VERSION="2.15.1"
RELEASE_PLATFORM="ami"
RELEASE_BUILD_ID="dc163e46e7"
RELEASE_BUILD_DATE="2018-10-26T10:53:58Z"
Error: no license file found (/data/user/common/enterprise.ghl)

 

It appears the error is a result of the license file being restored after ghe-restore-settings imports the new config and executes ghe-single-config-apply --phase-1. As the ghe-single-config-apply run is incomplete, the GitHub environment is not properly populated with SQL credentials.

This PR alters the order of operations to introduce the license file to the appliance before attempting to import and apply the configuration.

I also noted while reproducing this issue that if the hostname is specified as the first argument to ghe-restore, that all subsequent flags are ignored. I've included a fix to work around that use case as well.

/cc @github/backup-utils

@taz taz self-assigned this Nov 8, 2018
@taz taz requested a review from lildude November 8, 2018 13:24
lildude
lildude approved these changes Nov 8, 2018
View reviewed changes
Copy link
Member

@lildude lildude left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice catch. LGTM.

@taz taz merged commit 22349b4 into master Nov 8, 2018
@taz taz deleted the taz/fix-settings-restore-and-attributes branch November 9, 2018 01:11
@lildude lildude mentioned this pull request Nov 13, 2018
Merged
dooleydevin added a commit that referenced this pull request Jul 25, 2023
@dooleydevin
Merge pull request #449 from github/enterprise-3.8-backport-443-kyfas…
99171e5 
…t-create-current-encryption-key-backup-3.7.0+

Backport 443 for 3.8: Kyfast create current encryption key backup 3.7.0+
@dooleydevin dooleydevin mentioned this pull request Jul 26, 2023
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
1 more reviewer

@lildude lildude lildude approved these changes

Reviewers whose approvals may not affect merge requirements
Assignees

@taz taz

Labels
bug restore
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@taz @lildude