Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Return error if trying to open secondary on missing or inaccessible primary #8200

Closed

Conversation

riversand963
Copy link
Contributor

If the primary's CURRENT file is missing or inaccessible, the secondary should not hang
trying repeatedly to switch to the next MANIFEST.

Test plan:
make check

@facebook-github-bot
Copy link
Contributor

@riversand963 has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

@facebook-github-bot
Copy link
Contributor

@riversand963 has updated the pull request. You must reimport the pull request before landing.

@facebook-github-bot
Copy link
Contributor

@riversand963 has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

@facebook-github-bot
Copy link
Contributor

@riversand963 has updated the pull request. You must reimport the pull request before landing.

@facebook-github-bot
Copy link
Contributor

@riversand963 has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

@facebook-github-bot
Copy link
Contributor

@riversand963 has updated the pull request. You must reimport the pull request before landing.

@facebook-github-bot
Copy link
Contributor

@riversand963 has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

Copy link
Contributor

@jay-zhuang jay-zhuang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

Comment on lines +5814 to +5816
s = Status::TryAgain(
"The primary may have switched to a new MANIFEST and deleted the old "
"one.");
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we retry it, instead of reporting to user? Maybe the secondary should take a snapshot of manifest/CURRENT file at the beginning.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the secondary checks for the presence of MANIFEST via FileExists() and then tries to open it, there is always a possibility that the primary deletes the MANIFEST in between, thus the secondary will always likely hit a missing MANIFEST file. Similarly, after checking the presence of MANIFEST but before taking a hard-link, it is possible that the MANIFEST is deleted. We should always handle this situation.
Retry is possible, but I think we should have an interface for application to set retry policy. Since we do not have the interface yet, and it is beyond the scope of this PR. Future change of adding retry logic will be a behavior/contract change for TryCatchUpWithPrimary().

@facebook-github-bot
Copy link
Contributor

@riversand963 has updated the pull request. You must reimport the pull request before landing.

@facebook-github-bot
Copy link
Contributor

@riversand963 has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

@facebook-github-bot
Copy link
Contributor

@riversand963 merged this pull request in 2e53881.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

DB::OpenAsSecondary hangs if primary DB not available
3 participants