Recursive checker implementation. #225

ptabor · 2020-06-26T11:02:30Z

Recursive checker confirms database consistency with respect to b-tree
key order constraints:

keys on pages must be sorted
keys on children pages are between 2 consecutive keys on parent
branch page).

This PR moves checker logic to separate tx_checker file as the logic is already substantial (200).
It does not modify the original logic apart of running recursivelyCheckPages.

The reason to create this stronger checker is data corruption issue in etcd I was investigating that led to following branch state:

000000000000014d5f0000000000000000: <pgid=618>
000000000000014d5f0000000000000000: <pgid=778>
00000000013a29135f0000000000000000: <pgid=343>

...

Such state - according to the previous checker was correct.
The new checker reports:

key (0: 000000000000014d5f0000000000000000) on the branch page(618) needs to be < than key of the next element reachable from the ancestor (000000000000014d5f0000000000000000). Pages stack: [56 618]

mndrix · 2020-08-29T13:20:25Z

I'm not from etcd, but I've incorporated parts of this patch into my private fork. Thanks.

ahrtr · 2022-11-30T07:45:19Z

@ptabor Let's resurrect this PR. Could you please rebase this PR to resolve conflict?

ptabor · 2022-12-01T11:24:04Z

[scheduling this on my TODO queue]

ptabor · 2022-12-08T09:25:59Z

Resurrected. But still need to add a test.

ptabor · 2023-01-04T09:10:59Z

The PR has tests and resolved conflicts.

cmd/bbolt/main.go

internal/tests/tx_check_test.go

tx_check.go

ahrtr · 2023-01-05T02:36:51Z

tx_check.go

+			maxKeyInSubtree = tx.recursivelyCheckPagesInternal(elem.pgid, elem.key(), maxKey, pagesStack, keyToString, ch)
+			runningMin = maxKeyInSubtree
+		}
+		return


You need to define maxKey outside the for loop

Suggested change

return

return maxKey

Why not return maxKeyInSubtree that is implicit ?

Right, we should return maxKeyInSubtree. For the rightest element in a branch page, the maxKey in the page it links to should be the largest key in the tree rooted from current branch page.

tx_check.go

ahrtr · 2023-01-12T23:22:21Z

internal/tests/tx_check_test.go

+	xRay := surgeon.NewXRay(db.Path())
+
+	path1, err := xRay.FindPathsToKey([]byte("0451"))
+	require.NoError(t, err, "Cannot find page that contains key:'0451'")


nit: better to lower case the first letter when a message is wrapped in any other utilities (github.com/stretchr/testify in this case) instead of being outputted directly. This comment applies to the following similar messages as well.

Suggested change

require.NoError(t, err, "Cannot find page that contains key:'0451'")

require.NoError(t, err, "cannot find page that contains key:'0451'")

ahrtr · 2023-01-12T23:25:12Z

internal/tests/tx_check_test.go

+
+import (
+	"fmt"
+	"github.com/stretchr/testify/require"


Please reorder this import item.

ahrtr · 2023-01-12T23:31:09Z

tx_check.go

+//   - Are in right ordering relationship to their parents.
+//     `pagesStack` is expected to contain IDs of pages from the tree root to `pgid` for the clean debugging message.
+func (tx *Tx) recursivelyCheckPagesInternal(
+	pgid pgid, minKeyClosed, maxKeyOpen []byte, pagesStack []pgid,


pgid pgid... the name is the same string as the type. I know it works, but I'd suggest to avoid it to make it clearer and less confusion.

How about change it to something like pid pgid ?

Done pgId pgid --> pid sounds too much like process id.

ahrtr · 2023-01-13T01:12:07Z

tx_check.go

+			maxKeyInSubtree = tx.recursivelyCheckPagesInternal(elem.pgid, elem.key(), maxKey, pagesStack, keyToString, ch)
+			runningMin = maxKeyInSubtree
+		}
+		return


Right, we should return maxKeyInSubtree. For the rightest element in a branch page, the maxKey in the page it links to should be the largest key in the tree rooted from current branch page.

ahrtr · 2023-01-13T01:12:39Z

tx_check.go

+		for i := range p.leafPageElements() {
+			elem := p.leafPageElement(uint16(i))
+			if i == 0 && runningMin != nil && compareKeys(runningMin, elem.key()) > 0 {
+				ch <- fmt.Errorf("The first key[%d]=(hex)%s on leaf page(%d) needs to be >= the key in the ancestor (%s). Stack: %v",


Suggested change

ch <- fmt.Errorf("The first key[%d]=(hex)%s on leaf page(%d) needs to be >= the key in the ancestor (%s). Stack: %v",

ch <- fmt.Errorf("the first key[%d]=(hex)%s on leaf page(%d) needs to be >= the key in the ancestor (%s). Stack: %v",

ahrtr · 2023-01-13T01:17:35Z

In (*Tx) recursivelyCheckPages, the case p.flags&branchPageFlag != 0 should have almost the same logic as case p.flags&leafPageFlag != 0. The only difference is that it needs to recursively check the child page each branch element links to.

So the existing logic in case p.flags&branchPageFlag != 0 isn't completed as compared to the case p.flags&leafPageFlag != 0.

I would suggest to implement a common function or method to be shared by both cases.

Signed-off-by: Piotr Tabor <ptab@google.com>

Recursive checker confirms database consistency with respect to b-tree key order constraints: - keys on pages must be sorted - keys on children pages are between 2 consecutive keys on parent branch page). Signed-off-by: Piotr Tabor <ptab@google.com>

Signed-off-by: Piotr Tabor <ptab@google.com>

ahrtr

LGTM

ahrtr · 2023-01-28T22:33:14Z

tx_check.go

+// because of caching. This overhead can be removed if running on a read-only
+// transaction, however, it is not safe to execute other writer transactions at
+// the same time.
+func (tx *Tx) Check(kvStringer KVStringer) <-chan error {


This is a breaking change. It changed the signature of method Check. Previously there was no parameter. FYI. etcd-io/etcd@87c0da8

The proposals:

keep it as it's, and add a breaking change note when we release v1.3.7;

revert the change, and add a separate method something like CheckWithStringer(kvString KVString), and use a default KVStringer (e.g. bolt.HexKVStringer) when calling Check();

allow users to pass a nil KVStringer, and we use a default one in that case.

"2" seems to be the safest solution, because it has no any impact on users.

ptabor mentioned this pull request Jun 26, 2020

Is being dependent on the Check method enough for detecting boltdb corruption? #174

Open

ptabor mentioned this pull request Jul 22, 2020

Index out of range on etcd single node start-up #229

Open

ptabor force-pushed the extend_checker2 branch 3 times, most recently from 8d1bd97 to bdade3c Compare December 8, 2022 09:25

ptabor force-pushed the extend_checker2 branch 6 times, most recently from 0d5dd9a to 6bc4f3a Compare December 20, 2022 15:13

ptabor mentioned this pull request Dec 20, 2022

Factor out (bolt)testing library. #360

Merged

ptabor force-pushed the extend_checker2 branch 2 times, most recently from f58b8ea to 6ee6169 Compare December 20, 2022 16:48

This was referenced Dec 20, 2022

Extract from bbolt/CLI library for low-level inspection and modifications #361

Merged

Enhance check command to detect duplicated keys in each bucket #301

Closed

ptabor force-pushed the extend_checker2 branch 2 times, most recently from 8f17a73 to 6e95e9f Compare January 4, 2023 09:08

ptabor requested a review from ahrtr January 4, 2023 09:10

ptabor force-pushed the extend_checker2 branch from 6e95e9f to 0b0bb5f Compare January 4, 2023 11:08

ahrtr reviewed Jan 5, 2023

View reviewed changes

ptabor force-pushed the extend_checker2 branch from 0b0bb5f to 49e4182 Compare January 12, 2023 16:39

ahrtr reviewed Jan 13, 2023

View reviewed changes

Mechanical move of Check functions to tx_check.go file.

0ccb16d

Signed-off-by: Piotr Tabor <ptab@google.com>

ptabor added 6 commits January 13, 2023 19:14

Recursive checker implementation.

0c8d75d

Recursive checker confirms database consistency with respect to b-tree key order constraints: - keys on pages must be sorted - keys on children pages are between 2 consecutive keys on parent branch page). Signed-off-by: Piotr Tabor <ptab@google.com>

Tests for recursive checker (working on a corrupted files).

710c33f

Signed-off-by: Piotr Tabor <ptab@google.com>

Recursive checker: Final touches.

ee27a54

Signed-off-by: Piotr Tabor <ptab@google.com>

Address review comments.

f16e252

Signed-off-by: Piotr Tabor <ptab@google.com>

Rename: pgid pgid => pgId pgid to avoid confusion.

80edaf1

Signed-off-by: Piotr Tabor <ptab@google.com>

Refactor common code within key-order checker.

eb0deb9

Signed-off-by: Piotr Tabor <ptab@google.com>

ptabor force-pushed the extend_checker2 branch from 73d3338 to eb0deb9 Compare January 13, 2023 18:14

ahrtr approved these changes Jan 13, 2023

View reviewed changes

ptabor merged commit 51370e4 into etcd-io:master Jan 14, 2023

ahrtr added this to the 1.3.7 milestone Jan 16, 2023

ahrtr reviewed Jan 28, 2023

View reviewed changes

ahrtr mentioned this pull request Jan 28, 2023

Revert the signature of tx.Check and add tx.CheckWithStringer #395

Merged

unmarshall mentioned this pull request Sep 6, 2023

[Enhancement] Upgrade etcd version in backup-restore gardener/etcd-backup-restore#662

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Recursive checker implementation. #225

Recursive checker implementation. #225

ptabor commented Jun 26, 2020 •

edited

mndrix commented Aug 29, 2020

ahrtr commented Nov 30, 2022

ptabor commented Dec 1, 2022

ptabor commented Dec 8, 2022

ptabor commented Jan 4, 2023

ahrtr Jan 5, 2023 •

edited

ptabor Jan 12, 2023

ahrtr Jan 13, 2023

ahrtr Jan 12, 2023

ptabor Jan 13, 2023

ahrtr Jan 12, 2023

ptabor Jan 13, 2023

ahrtr Jan 12, 2023

ptabor Jan 13, 2023

ahrtr Jan 13, 2023

ahrtr Jan 13, 2023

ptabor Jan 13, 2023

ahrtr commented Jan 13, 2023

ahrtr left a comment

ahrtr Jan 28, 2023 •

edited

	require.NoError(t, err, "Cannot find page that contains key:'0451'")
	require.NoError(t, err, "cannot find page that contains key:'0451'")

	ch <- fmt.Errorf("The first key[%d]=(hex)%s on leaf page(%d) needs to be >= the key in the ancestor (%s). Stack: %v",
	ch <- fmt.Errorf("the first key[%d]=(hex)%s on leaf page(%d) needs to be >= the key in the ancestor (%s). Stack: %v",

Recursive checker implementation. #225

Recursive checker implementation. #225

Conversation

ptabor commented Jun 26, 2020 • edited

mndrix commented Aug 29, 2020

ahrtr commented Nov 30, 2022

ptabor commented Dec 1, 2022

ptabor commented Dec 8, 2022

ptabor commented Jan 4, 2023

ahrtr Jan 5, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ahrtr commented Jan 13, 2023

ahrtr left a comment

Choose a reason for hiding this comment

ahrtr Jan 28, 2023 • edited

Choose a reason for hiding this comment

ptabor commented Jun 26, 2020 •

edited

ahrtr Jan 5, 2023 •

edited

ahrtr Jan 28, 2023 •

edited