stat produces incorrect results when using secondary table

[snystrom]

I'm having an error where stat produces incorrect results depending on how the d4 file was generated.

If generated using the default d4tools create, I get incorrect values for mean and median. However, if generating the file using d4tools create -S, that d4 file produces accurate median and stat measurements. Accessing the values using d4tools view is accurate, however for both types of d4 file, and if I compute a rough estimate of the average from the view scores (without accounting for binning), I get a correct-ish number in both instances.

My computed stats using d4tools view:
Default D4:
- min: 255
- mean (approximate): 273.769
Sparse D4:
- min: 255
- mean (approximate): 273.769

d4tools stats:
default compression
- mean: 63
sparse compression
- mean: 273.5447154471545

[snystrom]

And a reproducible example using d4tools 0.1.16:
issue-12-example.tar.gz

md5sum: 4dbe38988bb191fe093336c3d83d6961 issue-12-example.tar.gz

[snystrom]

Not to be a nuisance, but is this a case of user error, or is this a real bug? This effect is a major blocker for adoption of this format for a few projects.

[arq5x]

I agree, @38 could you look into this ASAP?

[arq5x]

@snystrom just to confirm, this behavior exists on the latest (0.2.18) version as well, correct?

[snystrom]

Yeah I just checked it.

[38]

Hi there, thanks for reporting this issue. It should be fixed by the latest commit. It would be great if you can confirm that. Thanks!

[arq5x]

Great. This fixed it on my laptop.

[snystrom]

Confirm also fixed on my end, thanks a bunch!