Skip to content

bitmap indices for logs - #1124

Merged
AlexeyAkhunov merged 170 commits into
masterfrom
logs_index2
Sep 28, 2020
Merged

bitmap indices for logs #1124
AlexeyAkhunov merged 170 commits into
masterfrom
logs_index2

Conversation

@AskAlexSharov

@AskAlexSharov AskAlexSharov commented Sep 18, 2020

Copy link
Copy Markdown
Collaborator
  • Indices: addr -> bitmap(blockN) and topic -> bitmap(blockN) allow make complicated queries with all-optional parameters: aka "give me logs of addres X and has topic B or has topic C and between blocks D and E"
  • Indices bucket size: 13GB, 100M records (80M accounts, 20M topics).
  • Some indices have low selectivity and have big size - 1Mb. Implemented "hot-cold" sharding - small hot shard (where happen most of changes) and big cold shard (where hot shard merged when it getting big enough). Cold shards have limited size also and never merge between each-other.

@AskAlexSharov AskAlexSharov changed the title [do not merge] bitmap indices for logs bitmap indices for logs Sep 22, 2020
@AskAlexSharov AskAlexSharov changed the title bitmap indices for logs [wip] bitmap indices for logs Sep 23, 2020
@AskAlexSharov AskAlexSharov changed the title [wip] bitmap indices for logs bitmap indices for logs Sep 25, 2020
Comment thread cmd/rpcdaemon/commands/get_receipts.go Outdated
begin = uint32(crit.FromBlock.Uint64())
}
end := int64(latest)
end = uint32(latest)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do you convert it to uint32 and then convert to uint64 a bit later?

if addrBitmap == nil {
addrBitmap = m
} else {
addrBitmap = gocroaring.Or(addrBitmap, m)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do you use Or here and FastOr in another place?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Or - accept 2 bitmaps, FastOr - many, for 2 it works same as Or. so, no real reason here.

Comment thread cmd/rpcdaemon/commands/get_receipts.go Outdated
}

for _, blockNToMatch := range blockNumbers.ToArray() {
binary.BigEndian.PutUint32(blockNToMatchBytes, blockNToMatch)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you think this needs to be removed?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, removed

@AlexeyAkhunov
AlexeyAkhunov merged commit e02d6ac into master Sep 28, 2020
@AlexeyAkhunov
AlexeyAkhunov deleted the logs_index2 branch February 3, 2021 19:07
battlmonstr pushed a commit that referenced this pull request Sep 14, 2023
cffls pushed a commit to cffls/erigon that referenced this pull request Sep 5, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants