Weights, fees and storage bloat

[sander2]

We need to improve our weights and fees, as well as how we handle storage bloat. We need to do this to prevent DoS attacks. This includes:

• Charging for storage. A lot of calls add items to storage but we don't charge based on size
• We need to remove old state when it becomes irrelevant
• We need to implement existential deposits
  • Perhaps also for vaults - vaults that have almost no collateral can be removed
• We need to parametrize the weights of functions better so that we address their actual complexity (instead of using a fixed value)
• We need to re-run the benchmarks on a machine that represent the typical node hardware
• Functions that need to improve their weight determination include all functions that deal with bitcoin transactions. We might want to use pre-parsed structs rather than raw bytes to make that easier.
• We need to address unbounded iteration in e.g. execute_issue
• We should re-evaluate the amount we charge per byte in the extrinsic
• We should re-evaluate how we determine xcm fees
• Some of our dependencies use configurable weights. We currently use the default but we may want to override them
• Pallets should not use #[pallet::without_storage_info]. See my stackoverflow question
  • All storage items should implement MaxEncodedLen
• We should start using dispatch classes. See https://docs.substrate.io/build/tx-weights-fees/
• Attach #[pallet::whitelist_storage] to storage items that are read every block
• Look into base weight
• External pallets that we should reconsider config of:
  • pallet_transaction_payment

[gregdhill]

See also:

• Don't parse, verify #605
• Set ExistentialDeposits to non-zero values #481
• Reap storage entries for old block headers and requests #482
• fix: use RocksDbWeight for DbWeight #710
• Automatically benchmark runtimes #1009
• Improve benchmarking and accurately measure weights #1043

[sander2]

From SR-labs:

Extrinsics with missing storage deposits/fees could clutter the blockchain storage

Summary

Storage deposit fees are missing for multiple crates: example btc-relay, clients-info and democracy. An attacker could clutter the storage by calling extrinsics that save data into the blockchain database multiple times. A similar issue was also reported on the previous audit about missing storage deposits.

Issue details

The following extrinsic are affected:

• initialize extrinsic performs insert into storage as _store_block_header with following functions which in turns calls helpers for insertion into storage:

• create_and_store_blockchain

• extend_blockchain

• Self::reorganize_chains(&blockchain)?;

• register_as_candidate extrinsic performs insert into storage through <LastAuthoredBlock>::insert(

• propose performs deposit without any storage deposits as DepositOf::::insert(index, (depositors, value));

• fast_track_referendum extrinsic performs insert into storage through ReferendumInfoOf::::insert(ref_index, ReferendumInfo::Ongoing(status));

• vote extrinsic performs insert into storage through try_vote as ReferendumInfoOf::::insert(ref_index, ReferendumInfo::Ongoing(status));

• remove_vote extrinsic performs insert into storage through try_remove_vote as ReferendumInfoOf::::insert(ref_index, ReferendumInfo::Ongoing(status));

• increase_unlock_height through deposit for insert into storage <Locked>::insert(who, new_locked.clone());

• add_market performs insert into storage as Markets::::insert(asset_id, market.clone()); and ExchangeRate::::insert(asset_id, initial_exchange_rate);