Added gufi_stats -c direntries-log2-bins #139

johnbent · 2023-10-04T08:52:23Z

Added direntries and dirsize cumulative queries into gufi_stats following the logic of the newly added filesize. Also made 0 and 1 size items show in the 0 and 2 bins respectively. Changed floor to ceil so that 5 shows in the 8 bit and not the 4 bin which follows more consistently with the logical handling of 0, 1, 2, 3, 4, etc.

I tested it with the super simple bash script attached.
test.sh.txt

The output is as follows:

[jbent@hpe1 build]$ bash /tmp/test.sh 
Index building log in /tmp/dir2index_log.63409. Consult if errors
Initial direntries query
0 774
2 1092
4 515
8 518
Create 10 dirs with 0 files each and reindex.
0 784
2 1092
4 515
8 518
Create 10 dirs with 1 files each and reindex.
0 784
2 1102
4 515
8 518
Create 10 dirs with 2 files each and reindex.
0 784
2 1112
4 515
8 518
Create 10 dirs with 3 files each and reindex.
0 784
2 1112
4 525
8 518
Create 10 dirs with 4 files each and reindex.
0 784
2 1112
4 535
8 518
Create 10 dirs with 5 files each and reindex.
0 784
2 1112
4 535
8 528
Initial filesize query
0 2056
2 97
4 143
8 135
Create 10 files with size 0 each and reindex.
0 2066
2 97
4 143
8 135
Create 10 files with size 1 each and reindex.
0 2066
2 107
4 143
8 135
Create 10 files with size 2 each and reindex.
0 2066
2 117
4 143
8 135
Create 10 files with size 3 each and reindex.
0 2066
2 117
4 153
8 135
Create 10 files with size 4 each and reindex.
0 2066
2 117
4 163
8 135
Create 10 files with size 5 each and reindex.
0 2066
2 117
4 163
8 145

calccrypto · 2023-10-04T15:27:24Z

scripts/gufi_stats

+    ['dirsize-log2-bins',     dirsize_log2_bins],
+    ['dirsize-log1024-bins',  dirsize_log1024_bins],
    ['filesize-log2-bins',    filesize_log2_bins],
    ['filesize-log1024-bins', filesize_log1024_bins],
+    ['direntries-log2-bins',  direntries_log2_bins],
+    ['direntries-log1024-bins', direntries_log1024_bins],


group the dir* stuff together and in the order that they appear as above

As discussed, I'll just get rid of dirsize.

calccrypto · 2023-10-04T15:32:56Z

scripts/gufi_stats

@@ -1016,10 +1016,13 @@ def size_bins(args, base, type): # pylint: disable=redefined-builtin
    pinode_create = ['{0} INTEGER'.format(pinode_col)] if args.recursive else []
    pinode = [pinode_col] if args.recursive else []

+    # use log and power to put things into bins. Note that 0 must be handled specially because 2^0 is 1. Also 1 needs to be handled specially because log(N,1) is 0 but it shouldn't be included with 0
+    # so this will return items of size zero in a 0 bin and will return items of size 1 in the next bin
+    # use ceil so that 3 and 4 go into the 4 bin and 5 6 7 8 go into the 8 bin, etc.


which bin ends do we want?

I don't have a preference. I'll back this out and test with your zero-fix.

calccrypto · 2023-10-04T16:27:49Z

scripts/gufi_stats

+def direntries_log2_bins(_config, args, _where):
+    return size_bins(args, 2, 'd', 'totfiles')
+
+def direntries_log1024_bins(_config, args, _where):
+    return size_bins(args, 1024, 'd', 'totfiles')


maybe rename to dirfilecount_* since you are pulling totfiles and not totfiles + totlinks

Can add multiple functions:

dirfilecount for just files

dirlinkcount for just links
Cannot do direntrycount bec need totsubdirs. I'll submit an Issue asking for totsubdirs.

calccrypto · 2023-10-04T16:29:59Z

scripts/gufi_stats

@@ -993,7 +993,7 @@ def uid_size(_config, args, where):
 def gid_size(_config, args, where):
    return uidgid_size(args, 'gid', where)

-def size_bins(args, base, type): # pylint: disable=redefined-builtin
+def size_bins(args, base, type, field='size'): # pylint: disable=redefined-builtin


lets not have default arguments. the only other place there are default arguments is build_where, and they arent even used

calccrypto · 2023-10-04T16:31:34Z

scripts/gufi_stats

+def dirsize_log2_bins(_config, args, _where):
+    return size_bins(args, 2, 'd')
+
+def dirsize_log1024_bins(_config, args, _where):
+    return size_bins(args, 1024, 'd')


what does dirsize tell us?

will remove

johnbent · 2023-10-04T18:08:34Z

Also need to update docs and tests. Follow what was done here:
74a01e9#diff-0f23589fd21049cea7039aa237fb7d0c21c592bb24f81ba4f1f5e48c29c012b9

johnbent · 2023-10-04T18:27:57Z

As discussed, I'll also change 'bits' to 'exponent' as the variable name.

johnbent mentioned this pull request Oct 4, 2023

FEATURE REQUEST: gufi_stats -c dirsize-log2-bins #138

Closed

calccrypto reviewed Oct 4, 2023

View reviewed changes

calccrypto requested changes Oct 4, 2023

View reviewed changes

calccrypto reviewed Oct 4, 2023

View reviewed changes

johnbent closed this Oct 5, 2023

johnbent force-pushed the dirsize_log branch from 5b0e66b to 74a01e9 Compare October 5, 2023 06:45

johnbent deleted the dirsize_log branch October 6, 2023 05:47

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Added gufi_stats -c direntries-log2-bins #139

Added gufi_stats -c direntries-log2-bins #139

johnbent commented Oct 4, 2023

calccrypto Oct 4, 2023 •

edited

Loading

johnbent Oct 4, 2023

calccrypto Oct 4, 2023

johnbent Oct 4, 2023

calccrypto Oct 4, 2023

johnbent Oct 4, 2023

calccrypto Oct 4, 2023

calccrypto Oct 4, 2023

johnbent Oct 4, 2023

johnbent commented Oct 4, 2023

johnbent commented Oct 4, 2023

Added gufi_stats -c direntries-log2-bins #139

Added gufi_stats -c direntries-log2-bins #139

Conversation

johnbent commented Oct 4, 2023

calccrypto Oct 4, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

johnbent commented Oct 4, 2023

johnbent commented Oct 4, 2023

calccrypto Oct 4, 2023 •

edited

Loading