New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Event filtering is slow #15091

Closed
jannikluhn opened this Issue Sep 5, 2017 · 4 comments

Comments

Projects
None yet
2 participants
@jannikluhn

jannikluhn commented Sep 5, 2017

System information

Geth
Version: 1.6.7-stable
Git Commit: ab5646c
Architecture: amd64
Protocol Versions: [63 62]
Network Id: 1
Go Version: go1.8.1
Operating System: linux

Issue

Getting logs from a large block range is slow, even if the filter doesn't match anything:

time curl -X POST --data '{"id":8,"jsonrpc":"2.0","method":"eth_getLogs","params":[{"fromBlock":"0x0","toBlock":"0xf0be2", "address": "0x33990122638b9132ca29c723bdf037f1a891a70d"}]}' -H "Content-Type: application/json" http://127.0.0.1:8545 >> /dev/null

takes 27s instead of 0.02s promised in this stackexchange answer. Tested on Ropsten.

@karalabe

This comment has been minimized.

Show comment
Hide comment
@karalabe

karalabe Sep 5, 2017

Member

There's a PR pending review/merge that replaces the entire event filtering to a completely different mechanism #14631. It should be orders of magnitude faster. Feel free to take it to a spin an report on actual numbers. I'd be curious for real world tests.

Member

karalabe commented Sep 5, 2017

There's a PR pending review/merge that replaces the entire event filtering to a completely different mechanism #14631. It should be orders of magnitude faster. Feel free to take it to a spin an report on actual numbers. I'd be curious for real world tests.

@karalabe

This comment has been minimized.

Show comment
Hide comment
@karalabe

karalabe Sep 5, 2017

Member

Just did a fresh sync on Ropsten with the bloombits PR and tried your command:

$ time curl -s -X POST --data '{"id":8,"jsonrpc":"2.0","method":"eth_getLogs","params":[{"fromBlock":"0x0","toBlock":"0xf0be2", "address": "0x33990122638b9132ca29c723bdf037f1a891a70d"}]}' -H "Content-Type: application/json" http://127.0.0.1:8545 >> /dev/null

real	0m0.168s
user	0m0.004s
sys	0m0.000s

$ time curl -s -X POST --data '{"id":8,"jsonrpc":"2.0","method":"eth_getLogs","params":[{"fromBlock":"0x0","toBlock":"0xf0be2", "address": "0x33990122638b9132ca29c723bdf037f1a891a70d"}]}' -H "Content-Type: application/json" http://127.0.0.1:8545 >> /dev/null

real	0m0.012s
user	0m0.004s
sys	0m0.000s

Running the a full chain filtering for all ENS .eth events on Ropsten:

$ time curl -s -X POST --data '{"id":1,"jsonrpc":"2.0","method":"eth_getLogs","params":[{"fromBlock":"0x0","toBlock":"latest", "address": "0xC68De5B43C3d980B0C110A77a5F78d3c4c4d63B4"}]}' http://127.0.0.1:8545 >> /dev/null

real	0m2.203s
user	0m0.008s
sys	0m0.024s

$ time curl -s -X POST --data '{"id":1,"jsonrpc":"2.0","method":"eth_getLogs","params":[{"fromBlock":"0x0","toBlock":"latest", "address": "0xC68De5B43C3d980B0C110A77a5F78d3c4c4d63B4"}]}' http://127.0.0.1:8545 > /dev/null

real	0m0.658s
user	0m0.000s
sys	0m0.008s
Member

karalabe commented Sep 5, 2017

Just did a fresh sync on Ropsten with the bloombits PR and tried your command:

$ time curl -s -X POST --data '{"id":8,"jsonrpc":"2.0","method":"eth_getLogs","params":[{"fromBlock":"0x0","toBlock":"0xf0be2", "address": "0x33990122638b9132ca29c723bdf037f1a891a70d"}]}' -H "Content-Type: application/json" http://127.0.0.1:8545 >> /dev/null

real	0m0.168s
user	0m0.004s
sys	0m0.000s

$ time curl -s -X POST --data '{"id":8,"jsonrpc":"2.0","method":"eth_getLogs","params":[{"fromBlock":"0x0","toBlock":"0xf0be2", "address": "0x33990122638b9132ca29c723bdf037f1a891a70d"}]}' -H "Content-Type: application/json" http://127.0.0.1:8545 >> /dev/null

real	0m0.012s
user	0m0.004s
sys	0m0.000s

Running the a full chain filtering for all ENS .eth events on Ropsten:

$ time curl -s -X POST --data '{"id":1,"jsonrpc":"2.0","method":"eth_getLogs","params":[{"fromBlock":"0x0","toBlock":"latest", "address": "0xC68De5B43C3d980B0C110A77a5F78d3c4c4d63B4"}]}' http://127.0.0.1:8545 >> /dev/null

real	0m2.203s
user	0m0.008s
sys	0m0.024s

$ time curl -s -X POST --data '{"id":1,"jsonrpc":"2.0","method":"eth_getLogs","params":[{"fromBlock":"0x0","toBlock":"latest", "address": "0xC68De5B43C3d980B0C110A77a5F78d3c4c4d63B4"}]}' http://127.0.0.1:8545 > /dev/null

real	0m0.658s
user	0m0.000s
sys	0m0.008s
@karalabe

This comment has been minimized.

Show comment
Hide comment
@karalabe

karalabe Sep 6, 2017

Member

The PR was merged into master. Feel free to grab a fresh build and try it out yourself. Note, master does a few database upgrades, so you'll probably need to wait for that to finish (or resync from zero).

Member

karalabe commented Sep 6, 2017

The PR was merged into master. Feel free to grab a fresh build and try it out yourself. Note, master does a few database upgrades, so you'll probably need to wait for that to finish (or resync from zero).

@karalabe karalabe closed this Sep 6, 2017

@jannikluhn

This comment has been minimized.

Show comment
Hide comment
@jannikluhn

jannikluhn Sep 6, 2017

Awesome, now it's literally a thousand times better (down to 0.03s). Thanks a lot!

jannikluhn commented Sep 6, 2017

Awesome, now it's literally a thousand times better (down to 0.03s). Thanks a lot!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment