m0crate kv(index) test fixes. #461

somnathbghule · 2021-02-24T10:28:20Z

There are improvement areas such as,

Correctness in output logs (for all log levels) of m0crate KV test run.
	a. Following parameters are not being used by m0crate kv test but still displayed in the output logs.
		Before Fix:
			info: average size:          65536
			info: maximal size:          1048576
			info: block size:            0
		After Fix:
			No need to display this if KV test run is running.
	b. Following parameter shows wrong output.
		Before Fix:
			info: number of operations:  1000 (OP_COUT: 32 was passed. always shows the same value.). 
		After Fix:
			info: number of operations:  32
	c. If we read following output it looks like percentage of ops but its actually 
            count of number of operations remaining. 
		Before Fix:
			trace: dix: prcnt: [5, 4, 0, 0]		
		After Fix: 
			trace: dix: ops remaining: [5, 4, 0, 0]
	d.  Result should shows details about the test run,
            Before Fix:
                            result: all, 0.133912
            After Fix:
                           result: Total, 0.133912 s
	                          Avg time per ops.
	                          PUT, <time taken by put ops> s
	                          GET, <time taken by get ops> s and so on

Implementation of specifying the specific KEY and VALUE sizes in kv test run, sizes can be number (int) or random.

    key_size=sizeof(struct m0_fid)
    value_size=RECORD_SIZE-key_size;
    here RECORD_SIZE is tunable parameter available in .yaml file.
    There is no way/option available to set specific size of key and value.

welcome · 2021-02-24T10:28:22Z

Thanks for your contribution in opening this pull request! Now you can be rewarded with a CORTX sticker by requesting cortx sticker
In the meantime, please check out the contributing guidelines and explore other ways you can get involved.

cla-bot · 2021-02-24T10:28:22Z

Thank you for your pull request and welcome to our community. We could not parse the GitHub identity of the following contributors: Somnath Ghule.
This is most likely caused by a git client misconfiguration; please make sure to:

check if your git client is configured with an email to sign commits git config --list | grep email
If not, set it up using git config --global user.email email@example.com
Make sure that the git commit email is configured in your GitHub account settings, see https://github.com/settings/emails

cortx-admin · 2021-02-24T10:32:09Z

Can one of the admins verify this patch?

cla-bot · 2021-02-25T03:44:56Z

Thank you for your pull request and welcome to our community. We could not parse the GitHub identity of the following contributors: Somnath Ghule.
This is most likely caused by a git client misconfiguration; please make sure to:

check if your git client is configured with an email to sign commits git config --list | grep email
If not, set it up using git config --global user.email email@example.com
Make sure that the git commit email is configured in your GitHub account settings, see https://github.com/settings/emails

cla-bot · 2021-02-25T12:51:46Z

Thank you for your pull request and welcome to our community. We could not parse the GitHub identity of the following contributors: Somnath Ghule.
This is most likely caused by a git client misconfiguration; please make sure to:

check if your git client is configured with an email to sign commits git config --list | grep email
If not, set it up using git config --global user.email email@example.com
Make sure that the git commit email is configured in your GitHub account settings, see https://github.com/settings/emails

cla-bot · 2021-02-26T10:17:10Z

Thank you for your pull request and welcome to our community. We could not parse the GitHub identity of the following contributors: Somnath Ghule.
This is most likely caused by a git client misconfiguration; please make sure to:

check if your git client is configured with an email to sign commits git config --list | grep email
If not, set it up using git config --global user.email email@example.com
Make sure that the git commit email is configured in your GitHub account settings, see https://github.com/settings/emails

cla-bot · 2021-02-26T14:03:54Z

Thank you for your pull request and welcome to our community. We could not parse the GitHub identity of the following contributors: Somnath Ghule.
This is most likely caused by a git client misconfiguration; please make sure to:

check if your git client is configured with an email to sign commits git config --list | grep email
If not, set it up using git config --global user.email email@example.com
Make sure that the git commit email is configured in your GitHub account settings, see https://github.com/settings/emails

cla-bot · 2021-02-26T14:35:24Z

Thank you for your pull request and welcome to our community. We could not parse the GitHub identity of the following contributors: Somnath Ghule.
This is most likely caused by a git client misconfiguration; please make sure to:

check if your git client is configured with an email to sign commits git config --list | grep email
If not, set it up using git config --global user.email email@example.com
Make sure that the git commit email is configured in your GitHub account settings, see https://github.com/settings/emails

somnathbghule · 2021-02-26T14:40:53Z

Result after fix:

info: done workload 0
info: ---------------------------------------
info: starting workload 0
info: workload type index/3
info: random seed: 0
info: number of threads: 1
info: dix: End of operations.
result: total_s, 0.231018, avg_time_per_op_ns, 7452178, key_size_bytes, 1024, value_size_bytes, 32768, ops, 31
result: put_s, 0.118350, avg_time_per_op_ns, 6228926, ops, 19
result: get_s, 0.044113, avg_time_per_op_ns, 5514173, ops, 8
result: del_s, 0.004043, avg_time_per_op_ns, 4043037, ops, 1
result: next_s, 0.026017, avg_time_per_op_ns, 8672168, ops, 3
Total: time=[0:231017544] ([0:007452178] per op) ops=31
PUT: [0:118349598] ([0:006228926] per op) ops=19
GET: [0:044113390] ([0:005514173] per op) ops=8
DEL: [0:004043037] ([0:004043037] per op) ops=1
NEXT: [0:026016505] ([0:008672168] per op) ops=3
info: done workload 0
info: ---------------------------------------

cla-bot · 2021-03-01T03:06:25Z

Thank you for your pull request and welcome to our community. We could not parse the GitHub identity of the following contributors: Somnath Ghule, root.
This is most likely caused by a git client misconfiguration; please make sure to:

check if your git client is configured with an email to sign commits git config --list | grep email
If not, set it up using git config --global user.email email@example.com
Make sure that the git commit email is configured in your GitHub account settings, see https://github.com/settings/emails

cla-bot · 2021-03-01T04:54:40Z

Thank you for your pull request and welcome to our community. We could not parse the GitHub identity of the following contributors: Somnath Ghule.
This is most likely caused by a git client misconfiguration; please make sure to:

check if your git client is configured with an email to sign commits git config --list | grep email
If not, set it up using git config --global user.email email@example.com
Make sure that the git commit email is configured in your GitHub account settings, see https://github.com/settings/emails

cla-bot · 2021-03-01T05:02:27Z

Thank you for your pull request and welcome to our community. We could not parse the GitHub identity of the following contributors: Somnath Ghule.
This is most likely caused by a git client misconfiguration; please make sure to:

check if your git client is configured with an email to sign commits git config --list | grep email
If not, set it up using git config --global user.email email@example.com
Make sure that the git commit email is configured in your GitHub account settings, see https://github.com/settings/emails

cla-bot · 2021-03-01T05:26:48Z

Thank you for your pull request and welcome to our community. We could not parse the GitHub identity of the following contributors: Somnath Ghule.
This is most likely caused by a git client misconfiguration; please make sure to:

check if your git client is configured with an email to sign commits git config --list | grep email
If not, set it up using git config --global user.email email@example.com
Make sure that the git commit email is configured in your GitHub account settings, see https://github.com/settings/emails

cla-bot · 2021-03-01T05:35:20Z

Thank you for your pull request and welcome to our community. We could not parse the GitHub identity of the following contributors: somnathbghule.
This is most likely caused by a git client misconfiguration; please make sure to:

check if your git client is configured with an email to sign commits git config --list | grep email
If not, set it up using git config --global user.email email@example.com
Make sure that the git commit email is configured in your GitHub account settings, see https://github.com/settings/emails

motr/m0crate/crate_index.c

kaustubh-d · 2021-03-01T06:02:14Z

motr/m0crate/crate_index.c

@@ -308,6 +308,8 @@ struct cr_idx_w {
 	struct cr_time_measure_ctx	exec_time_ctx;
 	size_t				exec_time;
 	enum cr_op_selector		op_selector;
+	m0_time_t                       kv_op_acc_time[CRATE_OP_NR];


how about name exec_time_per_kv_op_type?

Add comment
// see enum cr_opcode, 4 kv op types for now

you may want to have struct cr_idx_w_results and move measurements outside workload descriptor.

example

struct cr_idx_ops_result { const char * c_op_type; // label int op_type; // comes from cr_opcode::CRATE_OP_PUT m0_time_t total_time_all_ops_s; m0_time_t time_per_op_ns; } struct cr_idx_w_results { m0_time_t total_time; // entire workload cr_idx_ops_result ops_result[CRATE_OP_NR]; // per op result for each put, get, del, next }

Fixed as per discussion.

kaustubh-d · 2021-03-01T06:04:07Z

motr/m0crate/crate_index.c

@@ -308,6 +308,8 @@ struct cr_idx_w {
 	struct cr_time_measure_ctx	exec_time_ctx;
 	size_t				exec_time;
 	enum cr_op_selector		op_selector;
+	m0_time_t                       kv_op_acc_time[CRATE_OP_NR];
+	int				kv_op_count[CRATE_OP_NR];


total_ops_count_per_kv_op_type

kaustubh-d · 2021-03-01T06:10:24Z

motr/m0crate/crate_index.c

+		w.wit->key_size, w.wit->value_size, w.nr_ops_total);
+
+	if (w.kv_op_count[CRATE_OP_PUT])	
+		fprintf(stdout, "result: put_s, %f, avg_time_per_op_ns,  %"PRIu64", ops, %d\n", 


ops should not have _s for secs, like put_s

use key as total_time_s

kaustubh-d · 2021-03-01T06:12:41Z

motr/m0crate/crate_index.c

+		t->elapsed, m0_time_nanoseconds(t->test_time) / w.nr_ops_total,
+		w.wit->key_size, w.wit->value_size, w.nr_ops_total);
+
+	if (w.kv_op_count[CRATE_OP_PUT])	


Simply to reduce fprintf/if-else ladder as discussed.

stale · 2021-03-15T05:50:42Z

This issue/pull request has been marked as needs attention as it has been left pending without new activity for 4 days. Tagging @huanghua78 @mukundkanekar for appropriate assignment. Sorry for the delay & Thank you for contributing to CORTX. We will get back to you as soon as possible.

kaustubh-d · 2021-03-15T08:12:35Z

motr/m0crate/tests/test1.yaml

+    WORKLOAD_SEED: tstamp
+    NUM_KVP: 8
+    NXRECORDS: default # int or default
+    RECORD_SIZE: 32 # int [units] or random


since we are updating these files, lets update to use latest params as key size and value size and just deprecate RECORD_SIZE

same for other sample yaml files.

yes, we can remove the "RECORD_SIZE" from these sample yaml config files.

Removed RECORD_SIZE and MAX_RSIZE from sample yaml config files.

Removed unwanted info from kv test run o/p, added KEY_SIZE support. Added m0crate index value_size changes. Cleaned record size changes and added changes for max key and value. added changes for total op time and time per ops in ns. added changes for total op time and time per ops in ns. Signed-off-by: somnathbghule <somnath.b.ghule@seagate.com>

Signed-off-by: somnathbghule <somnath.b.ghule@seagate.com>

huanghua78 · 2021-03-18T03:31:58Z

ok to test

welcome · 2021-03-18T05:06:18Z

Thanks for your contribution to CORTX! 🎉

kaustubh-d · 2021-03-20T04:56:21Z

Thanks @huanghua78 for your support in getting this reviewed and merged! Thanks @somnathbghule for the improvements!

* Removed unwanted info from kv test run o/p, added KEY_SIZE support. * Added m0crate index value_size changes. * Cleaned record size changes and added changes for max key and value. * Added changes for total op time and time per ops in ns. * Removed RECORD_SIZE and MAX_RSIZE from tests/*.yaml files. Signed-off-by: somnathbghule <somnath.b.ghule@seagate.com>

somnathbghule changed the title ~~[IN-PROGRESS] m0crate kv test fixes.~~ [IN-PROGRESS] m0crate kv(index) test fixes. Feb 25, 2021

somnathbghule marked this pull request as ready for review February 26, 2021 14:35

somnathbghule requested review from atulsdeshmukh2312 and huanghua78 as code owners February 26, 2021 14:35

somnathbghule changed the title ~~[IN-PROGRESS] m0crate kv(index) test fixes.~~ m0crate kv(index) test fixes. Feb 26, 2021

somnathbghule force-pushed the sg/m0crate-fixes branch from 2a49a0a to 531b9cd Compare March 1, 2021 04:54

somnathbghule force-pushed the sg/m0crate-fixes branch from 531b9cd to 6eec6ed Compare March 1, 2021 05:02

somnathbghule force-pushed the sg/m0crate-fixes branch from 6eec6ed to a27a003 Compare March 1, 2021 05:26

somnathbghule force-pushed the sg/m0crate-fixes branch from a27a003 to 8a5acb0 Compare March 1, 2021 05:35

kaustubh-d reviewed Mar 1, 2021

View reviewed changes

motr/m0crate/crate_index.c Outdated Show resolved Hide resolved

kaustubh-d reviewed Mar 1, 2021

View reviewed changes

motr/m0crate/crate_index.c Show resolved Hide resolved

kaustubh-d reviewed Mar 1, 2021

View reviewed changes

motr/m0crate/crate_index.c Outdated Show resolved Hide resolved

kaustubh-d reviewed Mar 1, 2021

View reviewed changes

stale bot added the needs-attention label Mar 15, 2021

somnathbghule force-pushed the sg/m0crate-fixes branch from 1439443 to 0dacfed Compare March 15, 2021 07:09

stale bot removed the needs-attention label Mar 15, 2021

kaustubh-d reviewed Mar 15, 2021

View reviewed changes

huanghua78 approved these changes Mar 15, 2021

View reviewed changes

somnathbghule and others added 17 commits March 17, 2021 05:33

Incorporated review comments, added formatting changes.

aba0cbc

Signed-off-by: somnathbghule <somnath.b.ghule@seagate.com>

Fixed README.md format.

57bcd3c

Signed-off-by: somnathbghule <somnath.b.ghule@seagate.com>

Incorporated review commnets, fixed formatting stuff.

c09246d

Signed-off-by: somnathbghule <somnath.b.ghule@seagate.com>

Update README.md.

6f5567d

Signed-off-by: somnathbghule <somnath.b.ghule@seagate.com>

Fixed README.md and result capture.

df3c17b

Signed-off-by: somnathbghule <somnath.b.ghule@seagate.com>

Update README.md.

9d609ba

Signed-off-by: somnathbghule <somnath.b.ghule@seagate.com>

Update README.md list item indentation.

711144d

Signed-off-by: somnathbghule <somnath.b.ghule@seagate.com>

Incorporated review comments, added formatting changes.

e815633

Signed-off-by: somnathbghule <somnath.b.ghule@seagate.com>

README.md update.

d01d768

Signed-off-by: somnathbghule <somnath.b.ghule@seagate.com>

Fix README.md.

a7fe0ef

Signed-off-by: somnathbghule <somnath.b.ghule@seagate.com>

README.md: remove extraneous new line after list item.

f968e89

Signed-off-by: somnathbghule <somnath.b.ghule@seagate.com>

Fixed key and value random gen issue.

0d07d7c

Signed-off-by: somnathbghule <somnath.b.ghule@seagate.com>

Added backward compatibility changes.

a99ad90

Signed-off-by: somnathbghule <somnath.b.ghule@seagate.com>

Update README.md.

e46a811

Signed-off-by: somnathbghule <somnath.b.ghule@seagate.com>

Fixed default max key and value.

eb2dc6a

Signed-off-by: somnathbghule <somnath.b.ghule@seagate.com>

Removed RECORD_SIZE and MAX_RSIZE from tests/*.yaml files.

e3c8132

Signed-off-by: somnathbghule <somnath.b.ghule@seagate.com>

somnathbghule force-pushed the sg/m0crate-fixes branch from 0dacfed to e3c8132 Compare March 17, 2021 11:34

huanghua78 merged commit 7a734e7 into Seagate:main Mar 18, 2021

imvenkip mentioned this pull request Nov 2, 2021

Improvements in m0crate utility (KV test run). #450

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

m0crate kv(index) test fixes. #461

m0crate kv(index) test fixes. #461

somnathbghule commented Feb 24, 2021 •

edited

Loading

welcome bot commented Feb 24, 2021

cla-bot bot commented Feb 24, 2021

cortx-admin commented Feb 24, 2021

cla-bot bot commented Feb 25, 2021

cla-bot bot commented Feb 25, 2021

cla-bot bot commented Feb 26, 2021

cla-bot bot commented Feb 26, 2021

cla-bot bot commented Feb 26, 2021

somnathbghule commented Feb 26, 2021

cla-bot bot commented Mar 1, 2021

cla-bot bot commented Mar 1, 2021

cla-bot bot commented Mar 1, 2021

cla-bot bot commented Mar 1, 2021

cla-bot bot commented Mar 1, 2021

kaustubh-d Mar 1, 2021

kaustubh-d Mar 1, 2021 •

edited

Loading

somnathbghule Mar 2, 2021

kaustubh-d Mar 1, 2021

somnathbghule Mar 2, 2021

kaustubh-d Mar 1, 2021

somnathbghule Mar 2, 2021

kaustubh-d Mar 1, 2021

somnathbghule Mar 2, 2021

stale bot commented Mar 15, 2021

kaustubh-d Mar 15, 2021

kaustubh-d Mar 15, 2021

huanghua78 Mar 15, 2021

somnathbghule Mar 17, 2021

huanghua78 commented Mar 18, 2021

welcome bot commented Mar 18, 2021

kaustubh-d commented Mar 20, 2021

m0crate kv(index) test fixes. #461

m0crate kv(index) test fixes. #461

Conversation

somnathbghule commented Feb 24, 2021 • edited Loading

welcome bot commented Feb 24, 2021

cla-bot bot commented Feb 24, 2021

cortx-admin commented Feb 24, 2021

cla-bot bot commented Feb 25, 2021

cla-bot bot commented Feb 25, 2021

cla-bot bot commented Feb 26, 2021

cla-bot bot commented Feb 26, 2021

cla-bot bot commented Feb 26, 2021

somnathbghule commented Feb 26, 2021

cla-bot bot commented Mar 1, 2021

cla-bot bot commented Mar 1, 2021

cla-bot bot commented Mar 1, 2021

cla-bot bot commented Mar 1, 2021

cla-bot bot commented Mar 1, 2021

Choose a reason for hiding this comment

kaustubh-d Mar 1, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stale bot commented Mar 15, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

huanghua78 commented Mar 18, 2021

welcome bot commented Mar 18, 2021

kaustubh-d commented Mar 20, 2021

somnathbghule commented Feb 24, 2021 •

edited

Loading

kaustubh-d Mar 1, 2021 •

edited

Loading