Skip to content
This repository has been archived by the owner on Nov 15, 2023. It is now read-only.

Add extrinsic to improve position in a bag of bags-list #9829

Merged
merged 42 commits into from
Dec 8, 2021
Merged
Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
42 commits
Select commit Hold shift + click to select a range
c414583
pallet-bags-list: Add `put_in_front_of` extrinsic
emostov Sep 17, 2021
331df1b
Test & Benchmarks
emostov Sep 21, 2021
e7c016c
Merge remote-tracking branch 'origin' into zeke-change-bag-position
emostov Sep 21, 2021
2401b2b
Respect line width
emostov Sep 21, 2021
60743cc
Remove List::put_in_fron_of tests; Remove AlreadyHigher error
emostov Sep 21, 2021
6bb232f
The dispatch origin for this call must be ...
emostov Sep 21, 2021
45f4a73
Add some periods
emostov Sep 21, 2021
f04a64c
Add back test to list module: put_in_front_of_exits_early_if_bag_not_…
emostov Sep 21, 2021
ce0fcfb
Merge remote-tracking branch 'origin' into zeke-change-bag-position
emostov Sep 21, 2021
aa0aaeb
add test tests::pallet::heavier_is_head_lighter_is_not_terminal
emostov Sep 22, 2021
fa1a33d
Cater for edge case of heavier being head
emostov Sep 23, 2021
21aef98
Add ExtBuilder::add_aux_data; try to make some tests use simpler data
emostov Sep 23, 2021
7433cf6
Update frame/bags-list/src/list/tests.rs
emostov Sep 23, 2021
1760815
make insert_at_unchecked infallible
emostov Sep 23, 2021
fcc29d2
Merge branch 'zeke-change-bag-position' of https://github.com/parityt…
emostov Sep 23, 2021
afff2a9
Make it permissioned - only callable by heavier
emostov Sep 23, 2021
b75457f
Add test cases for insert_at_unchecked
emostov Sep 23, 2021
fd1daeb
Move counter update to insert_at; fix comments
emostov Sep 25, 2021
f5a854f
Try merge origin master
emostov Oct 31, 2021
a12f094
Merge remote-tracking branch 'origin' into zeke-change-bag-position
emostov Nov 6, 2021
d5f82ac
Merge remote-tracking branch 'origin' into zeke-change-bag-position
emostov Nov 8, 2021
f3f992c
Address some feedback
emostov Nov 8, 2021
927a383
Make voteweight constructed with parameter_types
emostov Nov 10, 2021
387ab0d
Always set vote weight for Ids in build
emostov Nov 10, 2021
7d5a1ca
Add skip_genesis_ids
emostov Nov 10, 2021
2add927
Do not pass weight fn to List put_in_front_of
emostov Nov 10, 2021
fe120d6
Merge remote-tracking branch 'origin' into zeke-change-bag-position
emostov Nov 17, 2021
d135b8f
Remove remants of CounterForListNodes
emostov Nov 17, 2021
7530d1a
fmt
emostov Nov 17, 2021
cd39dc9
Merge branch 'master' of https://github.com/paritytech/substrate into…
Nov 17, 2021
fd751fc
cargo run --quiet --release --features=runtime-benchmarks --manifest-…
Nov 17, 2021
f5b0224
Delete messed up weights file
emostov Nov 17, 2021
91317bd
Some place holder stuff so we can run bench in CI
emostov Nov 17, 2021
6af64ce
Fix
emostov Nov 18, 2021
3382286
Merge branch 'master' of https://github.com/paritytech/substrate into…
Nov 18, 2021
56ea8cc
cargo run --quiet --release --features=runtime-benchmarks --manifest-…
Nov 18, 2021
a96e495
Merge remote-tracking branch 'origin' into zeke-change-bag-position
emostov Dec 2, 2021
77fd777
Update frame/bags-list/src/list/mod.rs
emostov Dec 6, 2021
aa4d896
fmt
emostov Dec 7, 2021
944dc85
Log + debug assert when refetching an Id
emostov Dec 7, 2021
ae188ae
Merge remote-tracking branch 'origin' into zeke-change-bag-position
emostov Dec 7, 2021
2eb86e5
Merge remote-tracking branch 'origin' into zeke-change-bag-position
emostov Dec 7, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
38 changes: 38 additions & 0 deletions frame/bags-list/src/benchmarks.rs
Original file line number Diff line number Diff line change
Expand Up @@ -134,6 +134,44 @@ frame_benchmarking::benchmarks! {
]
);
}

put_in_front_of {
// The most expensive case for `put_in_front_of`:
//
// - both heavier's `prev` and `next` are nodes that will need to be read and written.
// - `lighter` is the bag's `head`, so the bag will need to be read and written.
thiolliere marked this conversation as resolved.
Show resolved Hide resolved

let bag_thresh = T::BagThresholds::get()[0];

// insert the nodes in order
let lighter: T::AccountId = account("lighter", 0, 0);
assert_ok!(List::<T>::insert(lighter.clone(), bag_thresh));

let heavier_prev: T::AccountId = account("heavier_prev", 0, 0);
assert_ok!(List::<T>::insert(heavier_prev.clone(), bag_thresh));

let heavier: T::AccountId = account("heavier", 0, 0);
assert_ok!(List::<T>::insert(heavier.clone(), bag_thresh));

let heavier_next: T::AccountId = account("heavier_next", 0, 0);
assert_ok!(List::<T>::insert(heavier_next.clone(), bag_thresh));

T::VoteWeightProvider::set_vote_weight_of(&lighter, bag_thresh - 1);
T::VoteWeightProvider::set_vote_weight_of(&heavier, bag_thresh);

assert_eq!(
List::<T>::iter().map(|n| n.id().clone()).collect::<Vec<_>>(),
vec![lighter.clone(), heavier_prev.clone(), heavier.clone(), heavier_next.clone()]
);

let caller = whitelisted_caller();
}: _(SystemOrigin::Signed(caller), lighter.clone(), heavier.clone())
verify {
assert_eq!(
List::<T>::iter().map(|n| n.id().clone()).collect::<Vec<_>>(),
vec![heavier, lighter, heavier_prev, heavier_next]
)
}
}

use frame_benchmarking::impl_benchmark_test_suite;
Expand Down
35 changes: 35 additions & 0 deletions frame/bags-list/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -178,6 +178,21 @@ pub mod pallet {
Rebagged(T::AccountId, VoteWeight, VoteWeight),
}

#[pallet::error]
#[cfg_attr(test, derive(PartialEq))]
pub enum Error<T> {
/// Attempted to place node in front of a node in another bag.
NotInSameBag,
/// Id not found in list.
IdNotFound,
/// An Id does not have a greater vote weight than another Id.
NotHeavier,
/// Heavier weight Id is already in a higher position than a lesser weight Id.
AlreadyHigherPosition,
emostov marked this conversation as resolved.
Show resolved Hide resolved
/// Bag could not be found. This is a system logic error that should never happen.
BagNotFound,
emostov marked this conversation as resolved.
Show resolved Hide resolved
}

#[pallet::call]
impl<T: Config> Pallet<T> {
/// Declare that some `dislocated` account has, through rewards or penalties, sufficiently
Expand All @@ -195,6 +210,26 @@ pub mod pallet {
let _ = Pallet::<T>::do_rebag(&dislocated, current_weight);
Ok(())
}

/// Move `heavier` directly in front of `lighter`.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Given

heavy -> ... -> ... -> light

I as a 3rd person can pay to make this

... -> ... -> heavy -> light

which is unfair to heavy node.

given this I think this should not be permissionless, although it would be a pity. WDYT? the scenario above is def uncool.

Copy link
Contributor Author

@emostov emostov Sep 23, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I need to think about this more - but one approach would be to heavily restrict usage and say heavy can only go in front of light if light is the head. Significantly less helpful since once a node with a weight equal to the bag threhsold is at the head we can no longer do anything

In the direction of having it as a permissioned extrinsic within the context of Polkadot I don't really see a justification for adding this as a permissioned extrinsic - do you ?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Significantly less helpful since once a node with a weight equal to the bag threhsold is at the head we can no longer do anything

yeah, not a fan.

In the context of Polkadot I don't really see a justification for adding this as a permissioned extrinsic

I def. do think there's a use case for the permissioned version as well. A user within a bag is incentivized to call this themselves if they have a better position in the bag.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stating the obvious, but the fundamental issue here is we don't know relative position of any non-terminal nodes - not sure if there is any way of getting around that without adding more data to either the nodes or some storage item that keeps track of node index - both of which sound like too much added complexity at this stage

Copy link
Contributor Author

@emostov emostov Sep 23, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I def. do think there's a use case for the permissioned version as well. A user within a bag is incentivized to call this themselves if they have a better position in the bag.

EDIT sorry i was understanding this wrong - I thought you meant ROOT origin by permissioned . need to sleep

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree it should be permissioned and probably only callable by the heavy node

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since this is going to be permissioned it will mean it is only callable by the stash address - so we should make sure this is callable through at least some proxy, or maybe work in a way to call it via the controller.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds better than having it permissionless.

///
/// Only works if
emostov marked this conversation as resolved.
Show resolved Hide resolved
/// - both nodes are within the same bag,
/// - and `heavier` has a greater `VoteWeight` than `lighter`.
#[pallet::weight(T::WeightInfo::put_in_front_of())]
pub fn put_in_front_of(
origin: OriginFor<T>,
heavier: T::AccountId,
lighter: T::AccountId,
) -> DispatchResult {
ensure_signed(origin)?;
List::<T>::put_in_front_of(
&heavier,
&lighter,
Box::new(T::VoteWeightProvider::vote_weight),
emostov marked this conversation as resolved.
Show resolved Hide resolved
)
.map_err(Into::into)
}
}

#[pallet::hooks]
Expand Down
71 changes: 71 additions & 0 deletions frame/bags-list/src/list/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -384,6 +384,77 @@ impl<T: Config> List<T> {
})
}

/// Put `heavier_id` to the position directly in front of `lighter_id`. Both ids must be in
/// same bag and the `weight_of` `lighter_id` must be less than that of `heavier_id`
emostov marked this conversation as resolved.
Show resolved Hide resolved
pub(crate) fn put_in_front_of(
lighter_id: &T::AccountId,
heavier_id: &T::AccountId,
weight_of: Box<dyn Fn(&T::AccountId) -> VoteWeight>,
) -> Result<(), crate::pallet::Error<T>> {
use crate::pallet;
use frame_support::ensure;

let lighter_node = Node::<T>::get(&lighter_id).ok_or(pallet::Error::IdNotFound)?;
let mut heavier_node = Node::<T>::get(&heavier_id).ok_or(pallet::Error::IdNotFound)?;

ensure!(lighter_node.bag_upper == heavier_node.bag_upper, pallet::Error::NotInSameBag);
ensure!(
heavier_node.next.as_ref() != Some(lighter_id),
pallet::Error::AlreadyHigherPosition
);
kianenigma marked this conversation as resolved.
Show resolved Hide resolved

// this is the most expensive check, so we do it last
ensure!(weight_of(&heavier_id) > weight_of(&lighter_id), pallet::Error::NotHeavier);

// we need to write the bag to storage if its head and/or tail is updated
if lighter_node.is_terminal() || heavier_node.is_terminal() {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here you are assuming that the lighter node could only ever be a head, and the heavier node could only ever be a tail, while I don't see that being guaranteed. Can you elaborate all the edge cases that you considered, and why you think this is enough?

Copy link
Contributor

@kianenigma kianenigma Sep 22, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

here's an idea to break this down into smaller bite-size pieces:

  1. For removing, we simply use the current fn remove(). After all the checks, we immediately remove heavier. The logic for this function should already update the Bag head/tail if heavier.is_terminal().
  2. For inserting, we write a private helper function as such:
fn insert_at(after: Option<Node>) {
	if let Some(x) = after {
		// new logic, insert a new node on the prev of `after`  
	} else {
		// insert at tail: delegate to whatever we do now
	}
}

This will require the least amount of added logic, which helps with verifying the correctness of the PR (assuming that none of the old code was buggy).


If we keep the current code, here's a mental model to check it

  1. none are terminal
  2. lighter is head, heavier is not terminal (common case)
  3. lighter is tail, heavier is not terminal (mistake case)
  4. heavier is tail, lighter is not terminal (common case)
  5. heavier is head, lighter is not terminal (mistake case)
  6. lighter is head, heavier is tail (common case)
  7. lighter is tail, heavier is head (mistake case)

and for any case that contains a non-terminal, we have to consider if they are adjacent or not (in case 5 it totally makes a different).

I'm going to now start thinking about each case and how it is handled.

Copy link
Contributor

@kianenigma kianenigma Sep 22, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My analysis shows case 5 is not handled correctly, when the non-terminal lighter is not adjacent to the heavier one.

#[test]
fn heavier_is_head_lighter_is_not_terminal() {
	ExtBuilder::default().add_ids(vec![(5, 1000)]).build_and_execute(|| {
		// given
		assert_eq!(List::<Runtime>::get_bags(), vec![(10, vec![1]), (1_000, vec![2, 3, 4, 5])]);
		// when
		BagsList::put_in_front_of(Origin::signed(0), 2, 4).unwrap();
		// then.. :(
		assert_eq!(List::<Runtime>::get_bags(), vec![(10, vec![1]), (1_000, vec![3, 2, 4, 5])]);
	});
}

I'd like to see the fix, but my gut feeling is that rewriting it using smaller functions is easier. This if conditions_in_which_we_might_update_bags {} is a bit hard to verify. If we first remove, then update everything, and then insert, and update everything again, there would probably be a few extra (IMO negligible) lookups to the overlay, but the (non-negligible) db access would be the same.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

let mut bag = Bag::<T>::get(lighter_node.bag_upper).ok_or_else(|| {
debug_assert!(false, "bag that should exist cannot be found");
crate::log!(warn, "bag that should exist cannot be found");
pallet::Error::BagNotFound
})?;
debug_assert!(bag.iter().count() > 1);

if bag.head.as_ref() == Some(lighter_id) {
debug_assert!(lighter_node.next.is_some(), "lighter node must have next if head");
bag.head = Some(heavier_id.clone());
}

// re-assign bag tail if lighter is the tail
emostov marked this conversation as resolved.
Show resolved Hide resolved
if bag.tail.as_ref() == Some(heavier_id) {
debug_assert!(heavier_node.prev.is_some(), "heavier node must have prev if tail");
bag.tail = heavier_node.prev.clone();
}

// within this block we know the bag must have been updated, so we update storage
bag.put()
}

// cut heavier out of the list, updating its neighbors
heavier_node.excise();

// re-fetch `lighter_node` from storage since it may have been updated when heavier node was
// excised
let mut lighter_node = Node::<T>::get(&lighter_id).ok_or(pallet::Error::IdNotFound)?;

// connect `heavier_node` to its new `prev`
if let Some(mut prev) = lighter_node.prev() {
prev.next = Some(heavier_id.clone());
prev.put()
}
heavier_node.prev = lighter_node.prev;

// connect `heavier_node` and `lighter`
heavier_node.next = Some(lighter_id.clone());
lighter_node.prev = Some(heavier_id.clone());

// write the updated nodes to storage
lighter_node.put();
heavier_node.put();

Ok(())
}

/// Sanity check the list.
///
/// This should be called from the call-site, whenever one of the mutating apis (e.g. `insert`)
Expand Down
Loading