HDDS-3965. SCM failed to start up for duplicated pipeline detected. #1210
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
cc @fapifta |
arp7
approved these changes
Jul 17, 2020
There was a problem hiding this comment.
LGTM, however I would be more comfortable if @nandakumar131 also reviews since I am not familiar with this code and might have missed some subtle point.
nandakumar131
approved these changes
Jul 17, 2020
errose28
pushed a commit
to errose28/ozone
that referenced
this pull request
Jul 20, 2020
* master: HDDS-3984. Support filter and search the columns in recon UI (apache#1218) HDDS-3806. Support recognize aws v2 Authorization header. (apache#1098) HDDS-3955. Unable to list intermediate paths on keys created using S3G. (apache#1196) HDDS-3741. Reload old OM state if Install Snapshot from Leader fails (apache#1129) HDDS-3965. SCM failed to start up for duplicated pipeline detected. (apache#1210) HDDS-3855. Add upgrade smoketest (apache#1142) HDDS-3964. Ratis config key mismatch (apache#1204) HDDS-3612. Allow mounting bucket under other volume (apache#1104) HDDS-3926. OM Token Identifier table should use in-house serialization. (apache#1182) HDDS-3824: OM read requests should make SCM#refreshPipeline outside BUCKET_LOCK (apache#1164) HDDS-3966. Disable flaky TestOMRatisSnapshots
errose28
pushed a commit
to errose28/ozone
that referenced
this pull request
Jul 20, 2020
* HDDS-3869: HDDS-3984. Support filter and search the columns in recon UI (apache#1218) HDDS-3806. Support recognize aws v2 Authorization header. (apache#1098) HDDS-3955. Unable to list intermediate paths on keys created using S3G. (apache#1196) HDDS-3741. Reload old OM state if Install Snapshot from Leader fails (apache#1129) HDDS-3965. SCM failed to start up for duplicated pipeline detected. (apache#1210)
errose28
pushed a commit
to errose28/ozone
that referenced
this pull request
Jul 20, 2020
* master: HDDS-3984. Support filter and search the columns in recon UI (apache#1218) HDDS-3806. Support recognize aws v2 Authorization header. (apache#1098) HDDS-3955. Unable to list intermediate paths on keys created using S3G. (apache#1196) HDDS-3741. Reload old OM state if Install Snapshot from Leader fails (apache#1129) HDDS-3965. SCM failed to start up for duplicated pipeline detected. (apache#1210)
errose28
pushed a commit
to errose28/ozone
that referenced
this pull request
Jul 21, 2020
errose28
pushed a commit
to errose28/ozone
that referenced
this pull request
Jul 21, 2020
* add-deleted-block-table: (63 commits) Make block iterator tests use deleted blocks table, and remove the now unused #deleted# Replace uses of #deleted# key prefix with access to new deleted blocks table Add deleted blocks table to base level DB wrappers Have block deleting service test look for #deleted# keys in metadata table Move block delete to correct table and remove debugging print statement Import schema version when importing container data from export HDDS-3984. Support filter and search the columns in recon UI (apache#1218) HDDS-3806. Support recognize aws v2 Authorization header. (apache#1098) HDDS-3955. Unable to list intermediate paths on keys created using S3G. (apache#1196) HDDS-3741. Reload old OM state if Install Snapshot from Leader fails (apache#1129) Move new key value block iterator implementation and tests to new interface Fix checkstyle violations HDDS-3965. SCM failed to start up for duplicated pipeline detected. (apache#1210) Update comments Add comments on added helper method Remove seekToLast() from iterator interface, implementation, and tests Add more robust unit test with alternating key matches All unit tests pass after allowing keys with deleted and deleting prefixes to be made HDDS-3855. Add upgrade smoketest (apache#1142) HDDS-3964. Ratis config key mismatch (apache#1204) ...
ChenSammi
pushed a commit
that referenced
this pull request
Jul 22, 2020
timmylicheng
pushed a commit
that referenced
this pull request
Aug 6, 2020
rakeshadr
pushed a commit
to rakeshadr/hadoop-ozone
that referenced
this pull request
Sep 3, 2020
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
After the patch from HDDS-3925, SCM failed to start up with an exception stating that a 'duplicated pipeline has been detected'.
RCA
On investigating this along with @nandakumar131, the issue was found to be in the implementation of the RocksDBIterator wrapper that we have over the native RocksIterator. While calling next(), the current value is returned, and the pointer is moved ahead. Hence, the removeFromDB method actually deletes the next entry from the DB every time! During the last removeFromDB call invoked after the last "next()" call, the current pointer is pointing to garbage. This means that when SCM starts up with this schema change, the removeFromDB actually deletes the next pipeline from the DB everytime, thereby leaving the "first" pipeline entry in old format undeleted. On restart again, this causes a duplicate pipeline exception. I will attach detailed logs from the problematic state and the fix.
What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-3965
How was this patch tested?
Manually tested.
Added unit tests.
Logs
Logs which show the issue.txt
Logs with fix.txt