obj: implement statistics boilerplate macros

[pbalcer]

... and add one example statistic:
size in bytes of the curently allocated objects.

Ref: pmem/issues#676

---

This change is 

[krzycz]

Reviewed 32 of 32 files at r1.
Review status: all files reviewed at latest revision, 8 unresolved discussions, some commit checks failed.

---

src/libpmemobj/libpmemobj.vcxproj, line 53 at r1 (raw file):

    <ClCompile Include="stats.c" />
    <ClCompile Include="ringbuf.c" />
    <ClCompile Include="stats.c" />

same file twice?

---

src/libpmemobj/obj.h, line 139 at r1 (raw file):

	uint64_t conversion_flags;

	struct stats_persistent stats_persistent;

So the stats content is limited by the size of pmem_reserved?
What about keeping an OID here and allocate stats dynamically?

---

src/libpmemobj/stats.c, line 73 at r1 (raw file):

	int arg_in = *(int *)arg;

	pop->stats->enabled = arg_in;

1. I wonder if enabling it in runtime makes sense. Perhaps it should be allowed only at pool creation/opening (before any alloc/free opertion). Then, we could initialize statistics with some reliable data (i.e. by scanning the heap for allocated objects).
2. Shouldn't it be a part of persistent state? If the stats are enabled once, probably you want to use them next time the pool is open w/o explicit enabling it again.

---

src/libpmemobj/stats.c, line 88 at r1 (raw file):

/*
 * stats_new -- allocates and initialize statistics instance

initializes

---

src/libpmemobj/stats.c, line 113 at r1 (raw file):

stats_delete(struct stats *s)
{
	Free(s->transient);

pmem_persist(s->persistent, ...);
If these are persistent stats, they need to be flushed at some point. Perhaps it should be done at each STATS_INC/STATS_SUB, but that could be an overkill. OTH, if we do it only at pool close, the stats may get inconsistent in case of any failure.

---

src/libpmemobj/stats.h, line 82 at r1 (raw file):

{\
	int64_t *argv = arg;\
	*argv = (int64_t)pop->stats->type->varname;\

no atomic load here?

---

src/test/obj_ctl_stats/obj_ctl_stats.c, line 41 at r1 (raw file):

int
main(int argc, char *argv[])
{

obj_ctl_stats

---

src/test/obj_ctl_stats/TEST0, line 40 at r1 (raw file):

. ../unittest/unittest.sh

require_test_type short

fs any ?

---

Comments from Reviewable

[pbalcer]

Review status: all files reviewed at latest revision, 8 unresolved discussions, some commit checks failed.

---

src/libpmemobj/libpmemobj.vcxproj, line 53 at r1 (raw file):

Previously, krzycz (Krzysztof Czurylo) wrote…

same file twice?

Done.

---

src/libpmemobj/obj.h, line 139 at r1 (raw file):

Previously, krzycz (Krzysztof Czurylo) wrote…

So the stats content is limited by the size of pmem_reserved?
What about keeping an OID here and allocate stats dynamically?

1. Yes
2. Chicken and egg problem ;) But we can compromise and be always off by one.

---

src/libpmemobj/stats.c, line 73 at r1 (raw file):

Previously, krzycz (Krzysztof Czurylo) wrote…

1. I wonder if enabling it in runtime makes sense. Perhaps it should be allowed only at pool creation/opening (before any alloc/free opertion). Then, we could initialize statistics with some reliable data (i.e. by scanning the heap for allocated objects).
2. Shouldn't it be a part of persistent state? If the stats are enabled once, probably you want to use them next time the pool is open w/o explicit enabling it again.

The majority of statistics should be transient and used to decide how to optimize the use of the library. I do not recommend using these kind of values to make runtime decisions... But we are where we are.

I don't think stats "enabled" should be persistent state - it should be off by default and if you want to debug your application you can enable them through a ctl query. If one wants to ignore recommendations and use them for unintended purposes, one can always use a config file to specify that "stats.enabled = 1;".

---

src/libpmemobj/stats.c, line 88 at r1 (raw file):

Previously, krzycz (Krzysztof Czurylo) wrote…

initializes

Done.

---

src/libpmemobj/stats.c, line 113 at r1 (raw file):

Previously, krzycz (Krzysztof Czurylo) wrote…

pmem_persist(s->persistent, ...);
If these are persistent stats, they need to be flushed at some point. Perhaps it should be done at each STATS_INC/STATS_SUB, but that could be an overkill. OTH, if we do it only at pool close, the stats may get inconsistent in case of any failure.

In the design I've decided not to flush the variables at every store - as you rightly notice, it would be too expensive. We might eventually implement a mechanism for the potential drift that might happen, but that's for the future.

---

src/libpmemobj/stats.h, line 82 at r1 (raw file):

Previously, krzycz (Krzysztof Czurylo) wrote…

no atomic load here?

Done.

---

src/test/obj_ctl_stats/obj_ctl_stats.c, line 41 at r1 (raw file):

Previously, krzycz (Krzysztof Czurylo) wrote…

obj_ctl_stats

Done.

---

src/test/obj_ctl_stats/TEST0, line 40 at r1 (raw file):

Previously, krzycz (Krzysztof Czurylo) wrote…

fs any ?

what does that change? what's the default?
Done.

---

Comments from Reviewable

[krzycz]

---

Reviewed 11 of 11 files at r2.
Review status: all files reviewed at latest revision, 1 unresolved discussion.

---

src/test/obj_ctl_stats/TEST0, line 40 at r1 (raw file):

Previously, pbalcer (Piotr Balcer) wrote…

what does that change? what's the default?
Done.

AFAIR, default is "pmem non-pmem", so w/o "any" the test will be executed twice.

---

Comments from Reviewable

[marcinslusarz]

What do you mean by "are never altered after enabling"?

[pbalcer]

The statistics are not recalculated whenever you enable them. In other words, if you disable statistics, the operations that are performed won't count towards statistic values after you enable them again.

[codecov-io]

Codecov Report

Merging #2459 into master will decrease coverage by 0.08%.
The diff coverage is 51.21%.

@@            Coverage Diff             @@
##           master    #2459      +/-   ##
==========================================
- Coverage   80.03%   79.94%   -0.09%     
==========================================
  Files         121      122       +1     
  Lines       21002    21038      +36     
  Branches     3791     3794       +3     
==========================================
+ Hits        16808    16819      +11     
- Misses       4194     4219      +25

[gaweinbergi]

doc/libpmemobj/pmemobj_ctl_get.3.md, line 326 at r3 (raw file):

The statistics are not
recalculated after enabling, so that any operations that happened between
disabling and enabling are not going to be reflected in the values.

Statistics are not recalculated after enabling; any operations that occur between disabling and re-enabling will not be reflected in subsequent values.

---

Comments from Reviewable

[gaweinbergi]

doc/libpmemobj/pmemobj_ctl_get.3.md, line 326 at r3 (raw file):

might

may

---

Comments from Reviewable

[gaweinbergi]

doc/libpmemobj/pmemobj_ctl_get.3.md, line 337 at r4 (raw file):

might

may

---

Comments from Reviewable

[krzycz]

Reviewed 1 of 1 files at r3, 1 of 2 files at r4.
Review status: all files reviewed at latest revision, 2 unresolved discussions.

---

Comments from Reviewable

[pbalcer]

Review status: 34 of 35 files reviewed at latest revision, 2 unresolved discussions.

---

doc/libpmemobj/pmemobj_ctl_get.3.md, line 326 at r3 (raw file):

Previously, gaweinbergi (Glenn Weinberg) wrote…

might

may

Done.

---

doc/libpmemobj/pmemobj_ctl_get.3.md, line 337 at r4 (raw file):

Previously, gaweinbergi (Glenn Weinberg) wrote…

might

may

Done.

---

Comments from Reviewable

[krzycz]

Reviewed 1 of 1 files at r5.
Review status: all files reviewed at latest revision, 2 unresolved discussions.

---

Comments from Reviewable