Handle removed logs in filter changes and add geth compatibility field #8796

sorpaas · 2018-06-05T09:47:21Z

Fixes #6880. This PR adds two things:

Add removed field, equivalent to log.type == "removed", for geth compatibility.
Handle reorg removed logs in filter. We do this by keeping track of the current polling block hash. Whenever log changes are requested, we check whether this block hash is on the canon chain. If not, going through the parent hashes of the tracked block hash to find the canon chain, before getting a normal filter. Light client removed logs are disabled, as I think it's expensive to fetch this information.

Light client removed logs fetching is disabled. It looks expensive.

sorpaas · 2018-06-07T07:10:29Z

I updated this to address #6880 (comment), handling removed logs in filter changes.

…rpaas/geth-removed

andresilva

Overall LGTM, just some minor grumbles.

andresilva · 2018-06-12T15:52:06Z

rpc/src/v1/helpers/poll_filter.rs

@@ -30,7 +30,7 @@ pub enum PollFilter {
 	/// Hashes of all transactions which client was notified about.
 	PendingTransaction(Vec<H256>),
 	/// Number of From block number, pending logs and log filter itself.


Can you add documentation for this new parameter: "last seen block hash"?

andresilva · 2018-06-12T16:01:36Z

rpc/src/v1/impls/eth_filter.rs

@@ -52,6 +52,9 @@ pub trait Filterable {

 	/// Get a reference to the poll manager.
 	fn polls(&self) -> &Mutex<PollManager<PollFilter>>;
+
+	/// Get removed logs within route from the given block to the nearest canon block, not including the canon block. Also returns how many logs have been traversed.
+	fn canon_logs(&self, block_hash: H256, filter: &EthcoreFilter) -> (Vec<Log>, u64);


Can we rename this to something like removed_logs or reorged_logs? Something other than canon_logs since we're returning logs that are not canon.

andresilva · 2018-06-12T16:10:06Z

rpc/src/v1/impls/eth_filter.rs

@@ -176,6 +213,10 @@ impl<T: Filterable + Send + Sync + 'static> EthFilter for T {
 					filter.from_block = BlockId::Number(*block_number);
 					filter.to_block = BlockId::Latest;

+					// retrieve reorg logs
+					let (mut reorg, reorg_len) = last_block_hash.map_or_else(|| (Vec::new(), 0), |h| self.canon_logs(h, &filter));
+					*block_number -= reorg_len as u64;


This doesn't affect the filter that's used below when getting pending logs. And it's overridden afterwards with current + 1.

My mistake. Filter from_block should be set after we alter block_number.

…d (#8796) * Add removed geth compatibility field in log * Fix mocked tests * Add field block hash in PollFilter * Store last block hash info for log filters * Implement canon route * Use canon logs for fetching reorg logs Light client removed logs fetching is disabled. It looks expensive. * Make sure removed flag is set * Address grumbles

* master: Handle removed logs in filter changes and add geth compatibility field (#8796) fixed ipc leak, closes #8774 (#8876) scripts: remove md5 checksums (#8884) hardware_wallet/Ledger `Sign messages` + some refactoring (#8868)

#8796) * Add removed geth compatibility field in log * Fix mocked tests * Add field block hash in PollFilter * Store last block hash info for log filters * Implement canon route * Use canon logs for fetching reorg logs Light client removed logs fetching is disabled. It looks expensive. * Make sure removed flag is set * Address grumbles

* `duration_ns: u64 -> duration: Duration` (#8457) * duration_ns: u64 -> duration: Duration * format on millis {:.2} -> {} * Keep all enacted blocks notify in order (#8524) * Keep all enacted blocks notify in order * Collect is unnecessary * Update ChainNotify to use ChainRouteType * Fix all ethcore fn defs * Wrap the type within ChainRoute * Fix private-tx and sync api * Fix secret_store API * Fix updater API * Fix rpc api * Fix informant api * Eagerly cache enacted/retracted and remove contain_enacted/retracted * Fix indent * tests: should use full expr form for struct constructor * Use into_enacted_retracted to further avoid copy * typo: not a function * rpc/tests: ChainRoute -> ChainRoute::new * Handle removed logs in filter changes and add geth compatibility field (#8796) * Add removed geth compatibility field in log * Fix mocked tests * Add field block hash in PollFilter * Store last block hash info for log filters * Implement canon route * Use canon logs for fetching reorg logs Light client removed logs fetching is disabled. It looks expensive. * Make sure removed flag is set * Address grumbles * Fixed AuthorityRound deadlock on shutdown, closes #8088 (#8803) * CI: Fix docker tags (#8822) * scripts: enable docker builds for beta and stable * scripts: docker latest should be beta not master * scripts: docker latest is master * ethcore: fix ancient block error msg handling (#8832) * Disable parallel verification and skip verifiying already imported txs. (#8834) * Reject transactions that are already in pool without verifying them. * Avoid verifying already imported transactions. * Fix concurrent access to signer queue (#8854) * Fix concurrent access to signer queue * Put request back to the queue if confirmation failed * typo: fix docs and rename functions to be more specific `request_notify` does not need to be public, and it's renamed to `notify_result`. `notify` is renamed to `notify_message`. * Change trace info "Transaction" -> "Request" * Don't allocate in expect_valid_rlp unless necessary (#8867) * don't allocate via format! in case there's no error * fix test? * fixed ipc leak, closes #8774 (#8876) * Add new ovh bootnodes and fix port for foundation bootnode 3.2 (#8886) * Add new ovh bootnodes and fix port for foundation bootnode 3.2 * Remove old bootnodes. * Remove duplicate 1118980bf48b0a3640bdba04e0fe78b1add18e1cd99bf22d53daac1fd9972ad650df52176e7c7d89d1114cfef2bc23a2959aa54998a46afcf7d91809f0855082 * Block 0 is valid in queries (#8891) Early exit for block nr 0 leads to spurious error about pruning: `…your node is running with state pruning…`. Fixes #7547, #8762 * Add ETC Cooperative-run load balanced parity node (#8892) * Minor fix in chain supplier and light provider (#8906) * fix chain supplier increment * fix light provider block_headers * Check whether we need resealing in miner and unwrap has_account in account_provider (#8853) * Remove unused Result wrap in has_account * Check whether we need to reseal for external transactions * Fix reference to has_account interface * typo: missing ) * Refactor duplicates to prepare_and_update_sealing * Fix build * Allow disabling local-by-default for transactions with new config entry (#8882) * Add tx_queue_allow_unknown_local config option - Previous commit messages: dispatcher checks if we have the sender account Add `tx_queue_allow_unknown_local` to MinerOptions Add `tx_queue_allow_unknown_local` to config fix order in MinerOptions to match Configuration add cli flag for tx_queue_allow_unknown_local Update refs to `tx_queue_allow_unknown_local` Add tx_queue_allow_unknown_local to config test revert changes to dispatcher Move tx_queue_allow_unknown_local to `import_own_transaction` Fix var name if statement should return the values derp de derp derp derp semicolons Reset dispatch file to how it was before fix compile issues + change from FLAG to ARG add test and use `into` import MinerOptions, clone the secret Fix tests? Compiler/linter issues fixed Fix linter msg - case of constants IT LIVES refactor to omit yucky explict return update comments Fix based on diff AccountProvider.has_account method * Refactor flag name + don't change import_own_tx behaviour fix arg name Note: force commit to try and get gitlab tests working again 😠 * Add fn to TestMinerService * Avoid race condition from trusted sources - refactor the miner tests a bit to cut down on code reuse - add `trusted` param to dispatch_transaction and import_claimed_local_transaction Add param to `import_claimed_local_transaction` Fix fn sig in tests

…rp_sync_on_light_client * 'master' of https://github.com/paritytech/parity: (29 commits) Block 0 is valid in queries (openethereum#8891) fixed osx permissions (openethereum#8901) Atomic create new files with permissions to owner in ethstore (openethereum#8896) Add ETC Cooperative-run load balanced parity node (openethereum#8892) Add support for --chain tobalaba (openethereum#8870) fix some warns on nightly (openethereum#8889) Add new ovh bootnodes and fix port for foundation bootnode 3.2 (openethereum#8886) SecretStore: service pack 1 (openethereum#8435) Handle removed logs in filter changes and add geth compatibility field (openethereum#8796) fixed ipc leak, closes openethereum#8774 (openethereum#8876) scripts: remove md5 checksums (openethereum#8884) hardware_wallet/Ledger `Sign messages` + some refactoring (openethereum#8868) Check whether we need resealing in miner and unwrap has_account in account_provider (openethereum#8853) docker: Fix alpine build (openethereum#8878) Remove mac os installers etc (openethereum#8875) README.md: update the list of dependencies (openethereum#8864) Fix concurrent access to signer queue (openethereum#8854) Tx permission contract improvement (openethereum#8400) Limit the number of transactions in pending set (openethereum#8777) Use sealing.enabled to emit eth_mining information (openethereum#8844) ...

Add removed geth compatibility field in log

7a4fbd7

sorpaas added A0-pleasereview 🤓 Pull request needs code review. M6-rpcapi 📣 RPC API. labels Jun 5, 2018

sorpaas added this to the 1.12 milestone Jun 5, 2018

5chdn added the B0-patchthis label Jun 5, 2018

tomusdrw mentioned this pull request Jun 5, 2018

Parity doesn't return removed flag on filter changes #6880

Closed

Fix mocked tests

e92d69c

debris added A8-looksgood 🦄 Pull request is reviewed well. and removed A0-pleasereview 🤓 Pull request needs code review. labels Jun 5, 2018

debris approved these changes Jun 5, 2018

View reviewed changes

5chdn mentioned this pull request Jun 5, 2018

Release next-beta 1.11 #8190

Closed

57 tasks

sorpaas added 6 commits June 7, 2018 14:03

Add field block hash in PollFilter

0c029f1

Store last block hash info for log filters

b5002bf

Implement canon route

92ed0e8

Use canon logs for fetching reorg logs

e7ac8e4

Light client removed logs fetching is disabled. It looks expensive.

Merge branch 'sorpaas/filter-removed' into sorpaas/geth-removed

d751314

Make sure removed flag is set

cacb283

sorpaas added A0-pleasereview 🤓 Pull request needs code review. and removed A8-looksgood 🦄 Pull request is reviewed well. labels Jun 7, 2018

sorpaas changed the title ~~Add removed geth compatibility field in log~~ Handle removed logs in filter changes and add geth compatibility field Jun 7, 2018

sorpaas added 2 commits June 7, 2018 22:56

Merge branch 'master' of https://github.com/paritytech/parity into so…

b31a419

…rpaas/geth-removed

Merge branch 'master' of https://github.com/paritytech/parity into so…

9affc9c

…rpaas/geth-removed

andresilva suggested changes Jun 12, 2018

View reviewed changes

andresilva added A5-grumble 🔥 Pull request has minor issues that must be addressed before merging. and removed A0-pleasereview 🤓 Pull request needs code review. labels Jun 12, 2018

Address grumbles

9c6d1c6

5chdn added A0-pleasereview 🤓 Pull request needs code review. and removed A5-grumble 🔥 Pull request has minor issues that must be addressed before merging. labels Jun 13, 2018

andresilva approved these changes Jun 13, 2018

View reviewed changes

andresilva added A8-looksgood 🦄 Pull request is reviewed well. and removed A0-pleasereview 🤓 Pull request needs code review. labels Jun 13, 2018

5chdn merged commit b37b3cd into master Jun 13, 2018

5chdn deleted the sorpaas/geth-removed branch June 13, 2018 11:39

andresilva mentioned this pull request Jun 18, 2018

[beta] Backports #8916

Merged

18 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Handle removed logs in filter changes and add geth compatibility field #8796

Handle removed logs in filter changes and add geth compatibility field #8796

sorpaas commented Jun 5, 2018 •

edited

sorpaas commented Jun 7, 2018

andresilva left a comment

andresilva Jun 12, 2018

andresilva Jun 12, 2018

andresilva Jun 12, 2018

sorpaas Jun 13, 2018

Handle removed logs in filter changes and add geth compatibility field #8796

Handle removed logs in filter changes and add geth compatibility field #8796

Conversation

sorpaas commented Jun 5, 2018 • edited

sorpaas commented Jun 7, 2018

andresilva left a comment

Choose a reason for hiding this comment

andresilva Jun 12, 2018

Choose a reason for hiding this comment

andresilva Jun 12, 2018

Choose a reason for hiding this comment

andresilva Jun 12, 2018

Choose a reason for hiding this comment

sorpaas Jun 13, 2018

Choose a reason for hiding this comment

sorpaas commented Jun 5, 2018 •

edited