Skip to content

Commit

Permalink
Merge pull request #10 from bieli/update-benchmarks-test-and-readme
Browse files Browse the repository at this point in the history 
update benchmarks and README
  • Loading branch information
@bieli
bieli committed Mar 2, 2024
2 parents 97cd119 + 732c0d8 commit 31332f1
Show file tree
Hide file tree
Showing 3 changed files with 152 additions and 21 deletions.
1 change: 1 addition & 0 deletions Cargo.toml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@ rust_decimal_macros = "1.31"

[dev-dependencies]
criterion = "0.3"
#rand = "0.8.4"

[[bench]]
name = "my_benchmark"
Expand Down
97 changes: 79 additions & 18 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -10,10 +10,10 @@ This project was written in `RUST language` and it is an implementation of IoTex
IoText data protocol specification is here: https://github.com/bieli/IoText-data-protocol


## Development STDOUT (parsing IoText data protocol from String)
## Development details


### Example library usage code (parse String with IoText row and prepare String from IoText row)
### Example library code usage (parsing String with IoText row and preparing String from IoText MSG_EXAMPLE row)

```rust
use std::*;
Expand All @@ -32,6 +32,24 @@ fn main() {
}
```

## Benchmark synthetics tests - guaranty of linear inc. parsing time depends IoText metrics count

Table with tests results:

| IoText message size (bytes) | IoText msg. metrics count | avg. parsing time (µs) |
| --- | --- | --- |
| 206 | 10 | 2.5 |
| 432 | 20 | 4.5 |
| 638 | 30 | 6.5 |
| 844 | 40 | 8.5 |
| 1050 | 50 | 10.5 |
| 2080 | 100 | 20.5 |
| 4160 | 200 | 41.5 |

Tests were prepared on machine with Intel(R) Core(TM) i7-8850H CPU @ 2.60GHz 6-cores processor and 32GB RAM. With `load average: 1,44, 2,49, 3,51` (info from `upime` cmd) and approx. 80% RAM used. Linux kernel 5.15 and distribution Linux Ubuntu 20.04.6 LTS.

More detailed statistics from `cargo bench` running you can see in bottom section `Example cargo bench outputs`.

### Input example IoText data row
```bash
t|3900237526042,d|device_name_001,m|val_water_001=i:1234,m|val_water_002=i:15,m|bulb_state=b:1,m|connector_state=b:0,m|temp_01=d:34.4,m|temp_02=d:36.4,m|temp_03=d:10.4,m|pwr=d:12.231,m|current=d:1.429,m|current_battery=d:1.548,m|status_txt=t:in_progress
Expand Down Expand Up @@ -127,6 +145,65 @@ iotext_data_row: IoTextDataRow {
}
```

### Example cargo bench outputs
```bash

$ cargo bench

parse_iotext_str - 10 metrics
time: [1.9704 µs 1.9910 µs 2.0142 µs]
change: [-1.0093% +2.9404% +7.3783%] (p = 0.19 > 0.05)
No change in performance detected.
Found 13 outliers among 100 measurements (13.00%)
9 (9.00%) high mild
4 (4.00%) high severe

parse_iotext_str - 20 metrics
time: [3.8728 µs 3.9334 µs 4.0187 µs]
change: [-1.9420% +0.3434% +2.5942%] (p = 0.78 > 0.05)
No change in performance detected.
Found 11 outliers among 100 measurements (11.00%)
4 (4.00%) high mild
7 (7.00%) high severe

parse_iotext_str - 30 metrics
time: [5.9316 µs 6.0091 µs 6.0924 µs]
change: [+2.2407% +3.9826% +5.6299%] (p = 0.00 < 0.05)
Performance has regressed.
Found 7 outliers among 100 measurements (7.00%)
6 (6.00%) high mild
1 (1.00%) high severe

parse_iotext_str - 40 metrics
time: [8.4750 µs 8.6151 µs 8.7691 µs]
change: [-2.1735% -0.1968% +1.6887%] (p = 0.85 > 0.05)
No change in performance detected.
Found 2 outliers among 100 measurements (2.00%)
2 (2.00%) high mild

parse_iotext_str - 50 metrics
time: [10.914 µs 11.076 µs 11.264 µs]
change: [+3.2785% +5.5452% +7.6130%] (p = 0.00 < 0.05)
Performance has regressed.
Found 3 outliers among 100 measurements (3.00%)
2 (2.00%) high mild
1 (1.00%) high severe

parse_iotext_str - 100 metrics
time: [20.456 µs 20.593 µs 20.764 µs]
change: [-1.5163% +0.1975% +1.7861%] (p = 0.82 > 0.05)
No change in performance detected.
Found 8 outliers among 100 measurements (8.00%)
2 (2.00%) high mild
6 (6.00%) high severe

parse_iotext_str - 200 metrics
time: [40.849 µs 41.396 µs 42.023 µs]
Found 6 outliers among 100 measurements (6.00%)
4 (4.00%) high mild
2 (2.00%) high severe
```


## Run unit tests

Expand All @@ -145,21 +222,5 @@ test tests::test_extract_metric_value_type_decimal ... ok
test tests::test_extract_metric_value_type_text ... ok

test result: ok. 5 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s


```

## Run benchmark tests

```bash

$ cargo bench

...

parse_iotext_str time: [1.9652 µs 2.0015 µs 2.0388 µs]
Found 19 outliers among 100 measurements (19.00%)
8 (8.00%) high mild
11 (11.00%) high severe

```
75 changes: 72 additions & 3 deletions benches/my_benchmark.rs
View file
Original file line number Diff line number Diff line change
@@ -1,13 +1,82 @@
use criterion::{criterion_group, criterion_main, Criterion};
use iotext_rs::IoTextData;
use iotext_rs::IoTextDataRow;
/*
use rand::Rng;
pub fn gen_rnd_str() -> String {
const CHARSET: &[u8] = b"ABCDEFGHIJKLMNOPQRSTUVWXYZ\
abcdefghijklmnopqrstuvwxyz\
0123456789_";
const OUTPUT_LENGTH: usize = 30;
let mut rng = rand::thread_rng();
let gen_str: String = (0..OUTPUT_LENGTH)
.map(|_| {
let idx = rng.gen_range(0..CHARSET.len());
CHARSET[idx] as char
})
.collect();
gen_str
}
pub fn gen_next_10_metrics() -> String {
format!("|{gen_rnd_str()}=i:12345678,m|{gen_rnd_str()}=i:15,m|bulb_state2=b:1,m|connector_state2=b:0,m|temp_02=d:341.14,m|temp_022=d:316.4,m|temp_032=d:20.4,m|pwr2=d:32.231,m|current2=d:4.429,m|current_battery2=d:1.548".to_owned())
}
*/

pub fn criterion_benchmark(c: &mut Criterion) {
let data_obj: IoTextDataRow = IoTextDataRow::default();
const MSG_EXAMPLE: &str = "t|3900237526042,d|device_name_001,m|val_water_001=i:1234,m|val_water_002=i:15,m|bulb_state=b:1,m|connector_state=b:0,m|temp_01=d:34.4,m|temp_02=d:36.4,m|temp_03=d:10.4,m|pwr=d:12.231,m|current=d:1.429,m|current_battery=d:1.548";
const MSG_EXAMPLE_10_METRICS_FIRST: &str = "t|3900237526042,d|device_name_001,m|val_water_001=i:1234,m|val_water_002=i:15,m|bulb_state=b:1,m|connector_state=b:0,m|temp_01=d:34.4,m|temp_02=d:36.4,m|temp_03=d:10.4,m|pwr=d:12.231,m|current=d:1.429,m|current_battery=d:1.548";

const MSG_EXAMPLE_10_METRICS_2: &str = "|val_water_002=i:12345678,m|val_water_0022=i:15,m|bulb_state2=b:1,m|connector_state2=b:0,m|temp_02=d:341.14,m|temp_022=d:316.4,m|temp_032=d:20.4,m|pwr2=d:32.231,m|current2=d:4.429,m|current_battery2=d:1.548";

const MSG_EXAMPLE_10_METRICS_3: &str = "|val_water_003=i:12345678,m|val_water_0023=i:15,m|bulb_state3=b:1,m|connector_state3=b:0,m|temp_03=d:341.14,m|temp_023=d:316.4,m|temp_033=d:20.4,m|pwr3=d:32.231,m|current3=d:4.429,m|current_battery3=d:1.548";

const MSG_EXAMPLE_10_METRICS_4: &str = "|val_water_004=i:12345678,m|val_water_0024=i:15,m|bulb_state4=b:1,m|connector_state4=b:0,m|temp_04=d:341.14,m|temp_024=d:316.4,m|temp_034=d:20.4,m|pwr4=d:32.231,m|current4=d:4.429,m|current_battery4=d:1.548";

let msg_example_20_metrics: String =
format!("{MSG_EXAMPLE_10_METRICS_FIRST}{MSG_EXAMPLE_10_METRICS_2}");

let msg_example_30_metrics: String = format!(
"{MSG_EXAMPLE_10_METRICS_FIRST}{MSG_EXAMPLE_10_METRICS_2}{MSG_EXAMPLE_10_METRICS_3}"
);

let msg_example_40_metrics: String = format!("{MSG_EXAMPLE_10_METRICS_FIRST}{MSG_EXAMPLE_10_METRICS_2}{MSG_EXAMPLE_10_METRICS_3}{MSG_EXAMPLE_10_METRICS_4}");

let msg_example_50_metrics: String = format!("{MSG_EXAMPLE_10_METRICS_FIRST}{MSG_EXAMPLE_10_METRICS_2}{MSG_EXAMPLE_10_METRICS_3}{MSG_EXAMPLE_10_METRICS_4}{MSG_EXAMPLE_10_METRICS_4}");

let msg_example_100_metrics: String = format!("{MSG_EXAMPLE_10_METRICS_FIRST}{MSG_EXAMPLE_10_METRICS_2}{MSG_EXAMPLE_10_METRICS_3}{MSG_EXAMPLE_10_METRICS_4}{MSG_EXAMPLE_10_METRICS_4}{MSG_EXAMPLE_10_METRICS_2}{MSG_EXAMPLE_10_METRICS_3}{MSG_EXAMPLE_10_METRICS_4}{MSG_EXAMPLE_10_METRICS_4}{MSG_EXAMPLE_10_METRICS_4}");

let msg_example_200_metrics: String = format!("{MSG_EXAMPLE_10_METRICS_FIRST}{MSG_EXAMPLE_10_METRICS_2}{MSG_EXAMPLE_10_METRICS_3}{MSG_EXAMPLE_10_METRICS_4}{MSG_EXAMPLE_10_METRICS_4}{MSG_EXAMPLE_10_METRICS_2}{MSG_EXAMPLE_10_METRICS_3}{MSG_EXAMPLE_10_METRICS_4}{MSG_EXAMPLE_10_METRICS_4}{MSG_EXAMPLE_10_METRICS_4}{MSG_EXAMPLE_10_METRICS_2}{MSG_EXAMPLE_10_METRICS_3}{MSG_EXAMPLE_10_METRICS_4}{MSG_EXAMPLE_10_METRICS_4}{MSG_EXAMPLE_10_METRICS_2}{MSG_EXAMPLE_10_METRICS_3}{MSG_EXAMPLE_10_METRICS_4}{MSG_EXAMPLE_10_METRICS_4}{MSG_EXAMPLE_10_METRICS_4}{MSG_EXAMPLE_10_METRICS_4}");

c.bench_function("parse_iotext_str - 10 metrics", |b| {
b.iter(|| data_obj.parse_iotext_str(&MSG_EXAMPLE_10_METRICS_FIRST))
});

c.bench_function("parse_iotext_str - 20 metrics", |b| {
b.iter(|| data_obj.parse_iotext_str(&msg_example_20_metrics))
});

c.bench_function("parse_iotext_str - 30 metrics", |b| {
b.iter(|| data_obj.parse_iotext_str(&msg_example_30_metrics))
});

c.bench_function("parse_iotext_str - 40 metrics", |b| {
b.iter(|| data_obj.parse_iotext_str(&msg_example_40_metrics))
});

c.bench_function("parse_iotext_str - 50 metrics", |b| {
b.iter(|| data_obj.parse_iotext_str(&msg_example_50_metrics))
});

c.bench_function("parse_iotext_str - 100 metrics", |b| {
b.iter(|| data_obj.parse_iotext_str(&msg_example_100_metrics))
});

c.bench_function("parse_iotext_str", |b| {
b.iter(|| data_obj.parse_iotext_str(MSG_EXAMPLE))
c.bench_function("parse_iotext_str - 200 metrics", |b| {
b.iter(|| data_obj.parse_iotext_str(&msg_example_200_metrics))
});
}

Expand Down

0 comments on commit 31332f1

Please sign in to comment.