Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor(core): apply numerated to pages #3861

Merged
merged 27 commits into from
Apr 16, 2024
Merged
Show file tree
Hide file tree
Changes from 19 commits
Commits
Show all changes
27 commits
Select commit Hold shift + click to select a range
a0275ac
initial refactoring (pass pre-commit)
grishasobol Mar 29, 2024
8470c6f
remove some exess changes
grishasobol Mar 31, 2024
b1bbac8
refactoring
grishasobol Mar 31, 2024
b603921
remove exsess from benches
grishasobol Mar 31, 2024
4866641
refactor numerated using deps
grishasobol Mar 31, 2024
bc43c80
add issues
grishasobol Apr 1, 2024
9d3efa4
append more tests for alloc
grishasobol Apr 2, 2024
aceec87
fix test
grishasobol Apr 2, 2024
6754d54
add testing for pages in core
grishasobol Apr 3, 2024
e9a71ee
add tests for lazy-pages pages
grishasobol Apr 3, 2024
7cb06b2
PR fixes
grishasobol Apr 3, 2024
c8ace93
fix lazy-pages tests
grishasobol Apr 4, 2024
cdd6090
update gsdk
grishasobol Apr 4, 2024
fa5799c
fix test for windows
grishasobol Apr 4, 2024
dfe0cbe
Merge branch 'master' into gsobol-pages
grishasobol Apr 4, 2024
3c303ed
Merge branch 'master' into gsobol-pages
grishasobol Apr 4, 2024
45dad08
Merge branch 'master' into gsobol-pages
grishasobol Apr 9, 2024
551e4f1
Merge branch 'master' into gsobol-pages
grishasobol Apr 11, 2024
dd0cfff
PR fixes
grishasobol Apr 12, 2024
1999492
try to change try runtime version
grishasobol Apr 14, 2024
2ce7a13
add comment for SIZES_AMOUNT
grishasobol Apr 14, 2024
4f5472b
pub use numerated::numerated
grishasobol Apr 14, 2024
c38b0e1
Revert "try to change try runtime version"
grishasobol Apr 14, 2024
19b67e3
Merge branch 'master' into gsobol-pages
grishasobol Apr 14, 2024
d48112a
Merge branch 'master' into gsobol-pages
grishasobol Apr 15, 2024
aa071a0
fix after merge
grishasobol Apr 16, 2024
e2afb51
PR fixes
grishasobol Apr 16, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

3 changes: 2 additions & 1 deletion common/numerated/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -16,11 +16,12 @@ derive_more.workspace = true
scale-info = { workspace = true, features = ["derive"] }
parity-scale-codec = { workspace = true, features = ["derive"] }
log = { workspace = true, optional = true }
proptest = { workspace = true, optional = true }

[dev-dependencies]
env_logger.workspace = true
proptest.workspace = true
log.workspace = true

[features]
mock = ["log"]
mock = ["log", "proptest"]
5 changes: 4 additions & 1 deletion common/numerated/src/interval.rs
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,10 @@

//! [`Interval`] implementations.

use crate::{numerated::Numerated, Bound, IntervalIterator};
use crate::{
iterators::IntervalIterator,
numerated::{Bound, Numerated},
};
use core::{
fmt::{self, Debug, Formatter},
ops::{Range, RangeFrom, RangeFull, RangeInclusive, RangeTo, RangeToInclusive},
Expand Down
4 changes: 2 additions & 2 deletions common/numerated/src/iterators.rs
Original file line number Diff line number Diff line change
Expand Up @@ -19,8 +19,8 @@
//! [`IntervalIterator`], [`VoidsIterator`], [`DifferenceIterator`] implementations.

use crate::{
interval::{IncorrectRangeError, NewWithLenError, TryFromRangeError},
Interval, Numerated,
interval::{IncorrectRangeError, Interval, NewWithLenError, TryFromRangeError},
numerated::Numerated,
};
use core::{
fmt::{self, Debug, Display, Formatter},
Expand Down
14 changes: 4 additions & 10 deletions common/numerated/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -28,16 +28,10 @@
extern crate alloc;

pub mod interval;
mod iterators;
mod numerated;
mod tree;

pub use crate::{
interval::Interval,
iterators::{DifferenceIterator, IntervalIterator, VoidsIterator},
numerated::{Bound, Numerated, OptionBound},
tree::IntervalsTree,
};
pub mod iterators;
pub mod numerated;
pub mod tree;

pub use num_traits;

#[cfg(any(feature = "mock", test))]
Expand Down
88 changes: 83 additions & 5 deletions common/numerated/src/mock.rs
Original file line number Diff line number Diff line change
Expand Up @@ -18,9 +18,20 @@

//! Mock for crate property testing and also can be used in other crates for their numerated types impls.

use crate::{Bound, IntervalIterator, IntervalsTree, Numerated};
use crate::{
iterators::IntervalIterator,
numerated::{Bound, Numerated},
tree::IntervalsTree,
};
use alloc::{collections::BTreeSet, fmt::Debug, vec::Vec};
use core::ops::Range;
use num_traits::{bounds::UpperBounded, One, Zero};
use proptest::{
collection,
prelude::{any, Arbitrary},
prop_oneof,
strategy::{BoxedStrategy, Strategy},
};

/// Mock function for any [`Numerated`] implementation testing.
pub fn test_numerated<T>(x: T, y: T)
Expand Down Expand Up @@ -118,10 +129,7 @@ fn btree_set_voids<T: Numerated>(set: &BTreeSet<T>, interval: IntervalIterator<T
}

/// Mock function for [`IntervalsTree`] testing for any [`Numerated`] implementation.
pub fn test_tree<T: Numerated + UpperBounded + Debug>(
initial: BTreeSet<T>,
actions: Vec<TreeAction<T>>,
) {
pub fn test_tree<T: Numerated + Debug>(initial: BTreeSet<T>, actions: Vec<TreeAction<T>>) {
let mut tree: IntervalsTree<T> = initial.iter().copied().collect();
let mut expected: BTreeSet<T> = tree.points_iter().collect();
assert_eq!(expected, initial);
Expand Down Expand Up @@ -152,3 +160,73 @@ pub fn test_tree<T: Numerated + UpperBounded + Debug>(
assert_eq!(expected, tree.points_iter().collect());
}
}

impl<T> Arbitrary for IntervalIterator<T>
where
T: Numerated + Arbitrary + 'static,
{
type Parameters = ();
type Strategy = BoxedStrategy<Self>;

fn arbitrary_with(_args: Self::Parameters) -> Self::Strategy {
any::<(T, T)>()
.prop_map(|(p1, p2)| (p1.min(p2)..=p1.max(p2)).try_into().unwrap())
.boxed()
}
}

impl<T> Arbitrary for TreeAction<T>
where
T: Numerated + Arbitrary + 'static,
{
type Parameters = Range<usize>;
type Strategy = BoxedStrategy<Self>;

fn arbitrary_with(range: Self::Parameters) -> Self::Strategy {
prop_oneof![
any::<IntervalIterator<T>>().prop_map(TreeAction::Insert),
any::<IntervalIterator<T>>().prop_map(TreeAction::Remove),
any::<IntervalIterator<T>>().prop_map(TreeAction::Voids),
collection::btree_set(any::<T>(), range).prop_map(TreeAction::Difference),
]
.boxed()
}
}

impl<T> Arbitrary for IntervalAction<T>
where
T: Numerated + Arbitrary + 'static,
T::Bound: Debug,
{
type Parameters = ();
type Strategy = BoxedStrategy<Self>;

fn arbitrary_with(_args: Self::Parameters) -> Self::Strategy {
let start = any::<Option<T>>();
let end = any::<Option<T>>();
(start, end)
.prop_map(|(start, end)| {
let start: T::Bound = start.into();
let end: T::Bound = end.into();
match (start.unbound(), end.unbound()) {
(_, None) => IntervalAction::Correct(start, end),
(Some(s), Some(e)) if s <= e => IntervalAction::Correct(start, end),
(Some(_), Some(_)) => IntervalAction::Incorrect(start, end),
(None, Some(_)) => IntervalAction::Incorrect(start, end),
}
})
.boxed()
}
}

/// Tree actions strategy.
pub fn tree_actions<T: Numerated + Arbitrary + 'static>(
tree_size_range: Range<usize>,
actions_amount_range: Range<usize>,
) -> impl Strategy<Value = (BTreeSet<T>, Vec<TreeAction<T>>)> {
let action = TreeAction::arbitrary_with(tree_size_range.clone());
(
collection::btree_set(any::<T>(), tree_size_range),
collection::vec(action, actions_amount_range),
)
}
2 changes: 1 addition & 1 deletion common/numerated/src/numerated.rs
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ use num_traits::{One, PrimInt, Unsigned};
///
/// 2) When `inner` field max value is always smaller than `inner` type max value, then we can use this variant:
/// ```
/// use numerated::Bound;
/// use numerated::numerated::Bound;
grishasobol marked this conversation as resolved.
Show resolved Hide resolved
///
/// /// `inner` is a value from 0 to 99.
/// struct Number { inner: u32 }
Expand Down
63 changes: 6 additions & 57 deletions common/numerated/src/tests.rs
Original file line number Diff line number Diff line change
Expand Up @@ -18,59 +18,8 @@

//! Property testing for Numerated, Interval and IntervalsTree.

use crate::{
mock::{self, IntervalAction, TreeAction},
Bound, IntervalIterator, Numerated,
};
use alloc::{collections::BTreeSet, fmt::Debug, vec::Vec};
use num_traits::bounds::{LowerBounded, UpperBounded};
use proptest::{
arbitrary::{any, Arbitrary},
prop_oneof, proptest,
strategy::Strategy,
test_runner::Config as ProptestConfig,
};

struct Generator<T>(T);

impl<T> Generator<T>
where
T: Numerated + Arbitrary + Debug + LowerBounded + UpperBounded,
T::Bound: Debug,
{
fn rand_interval() -> impl Strategy<Value = IntervalIterator<T>> {
any::<(T, T)>().prop_map(|(p1, p2)| (p1.min(p2)..=p1.max(p2)).try_into().unwrap())
}

fn interval_action() -> impl Strategy<Value = IntervalAction<T>> {
let start = any::<Option<T>>();
let end = any::<Option<T>>();
(start, end).prop_map(|(start, end)| {
let start: T::Bound = start.into();
let end: T::Bound = end.into();
match (start.unbound(), end.unbound()) {
(_, None) => IntervalAction::Correct(start, end),
(Some(s), Some(e)) if s <= e => IntervalAction::Correct(start, end),
(Some(_), Some(_)) => IntervalAction::Incorrect(start, end),
(None, Some(_)) => IntervalAction::Incorrect(start, end),
}
})
}

fn rand_set() -> impl Strategy<Value = BTreeSet<T>> {
proptest::collection::btree_set(any::<T>(), 0..1000)
}

fn tree_actions() -> impl Strategy<Value = Vec<TreeAction<T>>> {
let action = prop_oneof![
Self::rand_interval().prop_map(TreeAction::Insert),
Self::rand_interval().prop_map(TreeAction::Remove),
Self::rand_interval().prop_map(TreeAction::Voids),
Self::rand_set().prop_map(TreeAction::Difference),
];
proptest::collection::vec(action, 10..20)
}
}
use crate::mock::{self, IntervalAction};
use proptest::{arbitrary::any, proptest, test_runner::Config as ProptestConfig};

proptest! {
#![proptest_config(ProptestConfig::with_cases(10_000))]
Expand All @@ -81,7 +30,7 @@ proptest! {
}

#[test]
fn proptest_interval_i16(action in Generator::<i16>::interval_action()) {
fn proptest_interval_i16(action in any::<IntervalAction::<i16>>()) {
mock::test_interval(action);
}

Expand All @@ -91,7 +40,7 @@ proptest! {
}

#[test]
fn proptest_interval_u16(action in Generator::<u16>::interval_action()) {
fn proptest_interval_u16(action in any::<IntervalAction::<u16>>()) {
mock::test_interval(action);
}
}
Expand All @@ -100,12 +49,12 @@ proptest! {
#![proptest_config(ProptestConfig::with_cases(128))]

#[test]
fn proptest_tree_i16(actions in Generator::<i16>::tree_actions(), initial in Generator::<i16>::rand_set()) {
fn proptest_tree_i16((initial, actions) in mock::tree_actions::<i16>(0..1000, 10..20)) {
mock::test_tree(initial, actions);
}

#[test]
fn proptest_tree_u16(actions in Generator::<u16>::tree_actions(), initial in Generator::<u16>::rand_set()) {
fn proptest_tree_u16((initial, actions) in mock::tree_actions::<i16>(0..1000, 10..20)) {
mock::test_tree(initial, actions);
}
}
8 changes: 6 additions & 2 deletions common/numerated/src/tree.rs
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,11 @@

//! [`IntervalsTree`] implementation.

use crate::{DifferenceIterator, Interval, IntervalIterator, Numerated, VoidsIterator};
use crate::{
interval::Interval,
iterators::{DifferenceIterator, IntervalIterator, VoidsIterator},
numerated::Numerated,
};
use alloc::{collections::BTreeMap, fmt, fmt::Debug, vec::Vec};
use core::{fmt::Formatter, ops::RangeInclusive};
use num_traits::{CheckedAdd, Zero};
Expand All @@ -37,7 +41,7 @@ use scale_info::{
///
/// # Examples
/// ```
/// use numerated::{IntervalsTree, Interval};
/// use numerated::{tree::IntervalsTree, interval::Interval};
/// use std::collections::BTreeSet;
/// use std::ops::RangeInclusive;
///
Expand Down
11 changes: 4 additions & 7 deletions common/src/benchmarking.rs
Original file line number Diff line number Diff line change
Expand Up @@ -18,10 +18,7 @@

use super::*;

use gear_core::{
pages::{PageNumber, WasmPage},
program::MemoryInfix,
};
use gear_core::{pages::WasmPage, program::MemoryInfix};
use gear_wasm_instrument::parity_wasm::{self, elements::*};
use sp_io::hashing::blake2_256;
use sp_runtime::traits::Zero;
Expand All @@ -46,7 +43,7 @@ pub fn create_module(num_pages: WasmPage) -> parity_wasm::elements::Module {
Section::Import(ImportSection::with_entries(vec![ImportEntry::new(
"env".into(),
"memory".into(),
External::Memory(MemoryType::new(num_pages.raw(), None)),
External::Memory(MemoryType::new(num_pages.into(), None)),
)])),
Section::Function(FunctionSection::with_entries(vec![Func::new(0)])),
Section::Export(ExportSection::with_entries(vec![ExportEntry::new(
Expand Down Expand Up @@ -102,7 +99,7 @@ pub fn generate_wasm(num_pages: WasmPage) -> Result<Vec<u8>, &'static str> {
FuncBody::new(
vec![Local::new(1, ValueType::I32)],
Instructions::new(vec![
Instruction::I32Const(num_pages.raw() as i32),
Instruction::I32Const(u32::from(num_pages) as i32),
breathx marked this conversation as resolved.
Show resolved Hide resolved
Instruction::Call(0),
Instruction::SetLocal(0),
Instruction::End,
Expand All @@ -118,7 +115,7 @@ pub fn generate_wasm(num_pages: WasmPage) -> Result<Vec<u8>, &'static str> {
pub fn set_program<ProgramStorage, BlockNumber>(
program_id: ProgramId,
code: Vec<u8>,
static_pages: WasmPage,
static_pages: WasmPagesAmount,
) where
ProgramStorage: super::ProgramStorage<BlockNumber = BlockNumber>,
BlockNumber: Zero + Copy + Saturating,
Expand Down
4 changes: 2 additions & 2 deletions common/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -58,7 +58,7 @@ use gear_core::{
ids::{CodeId, MessageId, ProgramId},
memory::PageBuf,
message::DispatchKind,
pages::{GearPage, WasmPage},
pages::{GearPage, WasmPage, WasmPagesAmount},
program::MemoryInfix,
reservation::GasReservationMap,
};
Expand Down Expand Up @@ -289,7 +289,7 @@ pub struct ActiveProgram<BlockNumber: Copy + Saturating> {
pub gas_reservation_map: GasReservationMap,
pub code_hash: H256,
pub code_exports: BTreeSet<DispatchKind>,
pub static_pages: WasmPage,
pub static_pages: WasmPagesAmount,
pub state: ProgramState,
pub expiration_block: BlockNumber,
}
Expand Down
11 changes: 4 additions & 7 deletions common/src/paused_program_storage.rs
Original file line number Diff line number Diff line change
Expand Up @@ -18,15 +18,10 @@

use super::{program_storage::MemoryMap, *};
use crate::storage::{MapStorage, ValueStorage};
use gear_core::{
code::MAX_WASM_PAGE_AMOUNT,
pages::{GEAR_PAGE_SIZE, WASM_PAGE_SIZE},
};
use gear_core::code::MAX_WASM_PAGES_AMOUNT;
use sp_core::MAX_POSSIBLE_ALLOCATION;
use sp_io::hashing;

const SPLIT_COUNT: u16 = (WASM_PAGE_SIZE / GEAR_PAGE_SIZE) as u16 * MAX_WASM_PAGE_AMOUNT / 2;

pub type SessionId = u32;

// The entity helps to calculate hash of program's data and memory pages.
Expand All @@ -40,7 +35,9 @@ impl From<(BTreeSet<WasmPage>, H256, MemoryMap)> for Item {
fn from(
(allocations, code_hash, mut memory_pages): (BTreeSet<WasmPage>, H256, MemoryMap),
) -> Self {
let remaining_pages = memory_pages.split_off(&GearPage::from(SPLIT_COUNT));
// TODO: consider memory pages batch size #1381
let split_page = WasmPage::from(MAX_WASM_PAGES_AMOUNT / 2);
grishasobol marked this conversation as resolved.
Show resolved Hide resolved
let remaining_pages = memory_pages.split_off(&split_page.to_page());
Self {
data: (allocations, code_hash, memory_pages),
remaining_pages,
Expand Down
Loading
Loading