Calibration issues

[tim2000s]

Following the issues I saw with the calibrations on logger, below is the content of the requested files:

g5.csv

epochdate	datetime	unfiltered	filtered	trend	calibratedBG	meterbg	slope	yIntercept	slopeError	yError	rSquared	Noise	NoiseSend
1519301888	##########	140064	140544	Flat	131	131	1069	0	0	0	0	0	1
1519302188	##########	140288	140416	Flat	131	null	1069	0	0	0	0	0	1
1519302487	##########	140064	140224	Flat	131	null	1069	0	0	0	0	0	1
1519302788	##########	140160	140096	Flat	131	null	1069	0	0	0	0	0.00014	1
1519303088	##########	139712	140000	Flat	130	null	1069	0	0	0	0	0.00037	1
1519303388	##########	138880	139840	Flat	129		1069	0	0	0	0	0.00083	1
1519303688	##########	137408	139296	Flat	128	null	1069	0	0	0	0	0.00197	1
1519303988	##########	135616	138272	Flat	126	null	1069	0	0	0	0	0.00302	1
1519304288	##########	134400	136800	Flat	125		1069	0	0	0	0	0.00285	1
1519304588	##########	132544	135104	Flat	123	null	1069	0	0	0	0	0.0033	1
1519304888	##########	131392	133440	Flat	122	null	1069	0	0	0	0	0.00302	1
1519305187	##########	130528	132032	Flat	122	null	1069	0	0	0	0	0.00274	1
1519305488	##########	128864	130784	Flat	120	null	1069	0	0	0	0	0.00225	1
1519305788	##########	127680	129632	Flat	119	null	1069	0	0	0	0	0.00186	1
1519306088	##########	125680	128320	Flat	117	null	1069	0	0	0	0	0.00122	1
1519306388	##########	126160	127008	Flat	118	null	1069	0	0	0	0	0.00253	1
1519306688	##########	126896	126128	Flat	118	null	1069	0	0	0	0	0.00446	1
1519306988	##########	133184	126688	Flat	124	null	1069	0	0	0	0	0.0288	1
1519307288	##########	138208	129472	Flat	129	null	1069	0	0	0	0	0.04	1
1519307587	##########	141280	134048	Flat	132	null	1069	0	0	0	0	0.04202	1
1519307888	##########	143936	139104	Flat	134		1069	0	0	0	0	0.03979	1
1519308187	##########	144096	142848	Flat	134	null	1069	0	0	0	0	0.03761	1
1519308488	##########	143968	144576	Flat	134	null	1069	0	0	0	0	0.03628	1
1519308788	##########	145632	144896	Flat	136	null	1069	0	0	0	0	0.03145	1
1519309087	##########	147552	145056	Flat	138	null	1069	0	0	0	0	0.02718	1
1519309388	##########	146976	145728	Flat	137	null	1069	0	0	0	0	0.0225	1
1519309688	##########	149536	147008	Flat	139	null	1069	0	0	0	0	0.02134	1
1519309988	##########	150176	148416	Flat	140	null	1069	0	0	0	0	0.02147	1
1519310288	##########	150688	149568	Flat	140	null	1069	0	0	0	0	0.01404	1
1519310588	##########	152416	150624	Flat	142	null	1069	0	0	0	0	0.00803	1
1519310888	##########	151808	151424	Flat	142	null	1069	0	0	0	0	0.00772	1
1519311189	##########	152000	151936	Flat	142	null	1069	0	0	0	0	0.00625	1
1519311489	##########	152480	152192	Flat	142	null	1069	0	0	0	0	0.0061	1
1519311788	##########	153120	152320	Flat	143	null	1069	0	0	0	0	0.00564	1
1519312088	##########	152128	152480	Flat	142	null	1069	0	0	0	0	0.00676	1
1519312388	##########	151232	152416	Flat	141	null	1069	0	0	0	0	0.00662	1
1519312688	##########	150304	151936	Flat	140		1069	0	0	0	0	0.00704	1
1519312987	##########	150272	151104	Flat	140	null	1069	0	0	0	0	0.00401	1
1519313287	##########	153696	150720	Flat	143	null	1069	0	0	0	0	0.00946	1
1519313587	##########	154528	151424	Flat	144		1069	0	0	0	0	0.00958	1
1519313888	##########	155552	153120	Flat	145		1069	0	0	0	0	0.00876	1
1519314188	##########	155200	154848	Flat	145		1069	0	0	0	0	0.00854	1
1519314487	##########	156608	155872	Flat	146	null	1069	0	0	0	0	0.00911	1
1519314789	##########	159968	156608	Flat	149	null	1069	0	0	0	0	0.01317	1
1519315087	##########	163168	158048	Flat	152	null	1069	0	0	0	0	0.01614	1
1519315387	##########	163456	160352	Flat	152	null	1069	0	0	0	0	0.0143	1
1519315687	##########	166592	163072	Flat	161	171	974	0	0	0	0	0.01371	1
1519315988	##########	168000	165408	FortyFiveUp	172	171	974	0	0	0	0	0.01053	1
1519316288	##########	169376	167200	FortyFiveUp	173	null	974	0	0	0	0	0.00901	1
1519316587	##########	171488	168768	Flat	176	null	974	0	0	0	0	0.00763	1
1519316887	##########	173152	170368	Flat	177	null	974	0	0	0	0	0.00726	1
1519317187	##########	173120	171872	Flat	177	null	974	0	0	0	0	0.00857	1
1519317487	##########	175520	173280	Flat	180	null	974	0	0	0	0	0.00649	1
1519317787	##########	175776	174432	Flat	180	null	974	0	0	0	0	0.00767	1
1519318088	##########	176800	175424	Flat	181	null	974	0	0	0	0	0.00645	1
1519318387	##########	176544	176224	Flat	181	null	974	0	0	0	0	0.00618	1
1519318687	##########	176512	176640	Flat	181	null	974	0	0	0	0	0.00656	1
1519318987	##########	175104	176576	Flat	179	null	974	0	0	0	0	0.0073	1
1519319287	##########	174816	176000	Flat	179	null	974	0	0	0	0	0.00757	1
1519319587	##########	173024	175040	Flat	177	null	974	0	0	0	0	0.00995	1
1519319888	##########	172704	173952	Flat	177	null	974	0	0	0	0	0.00817	1
1519320188	##########	171712	172928	Flat	176	null	974	0	0	0	0	0.00716	1
1519320488	##########	169696	171904	Flat	174	null	974	0	0	0	0	0.00889	1
1519320787	##########	167264	170624	Flat	171	null	974	0	0	0	0	0.00614	1
1519321087	##########	166496	168960	Flat	170	null	974	0	0	0	0	0.00552	1
1519321387	##########	164704	167104	Flat	169	null	974	0	0	0	0	0.00368	1
1519321687	##########	161536	165184	Flat	165	null	974	0	0	0	0	0.00509	1
1519321987	##########	157152	162912	Flat	161	null	974	0	0	0	0	0.00765	1
1519322287	##########	155104	160000	Flat	159	null	974	0	0	0	0	0.00761	1
1519322588	##########	152224	156672	Flat	156	null	974	0	0	0	0	0.00659	1
1519322887	##########	146720	153152	Flat	150	null	974	0	0	0	0	0.01121	1
1519323189	##########	142688	149440	Flat	146	null	974	0	0	0	0	0.00955	1
1519323488	##########	129808	144352	FortyFiveDown	133	null	974	0	0	0	0	0.03253	1
1519323789	##########	128736	137760	FortyFiveDown	132	null	974	0	0	0	0	0.03443	1
1519324087	##########	127056	131264	FortyFiveDown	130	null	974	0	0	0	0	0.03554	1
1519324387	##########	118976	125952	FortyFiveDown	122		974	0	0	0	0	0.03547	1
1519324687	##########	117280	122208	Flat	120	null	974	0	0	0	0	0.03569	1
1519324987	##########	111216	118688	Flat	114	null	974	0	0	0	0	0.03553	1
1519325287	##########	110368	114800	Flat	113	null	974	0	0	0	0	0.04097	1
1519325587	##########	109120	111392	Flat	112		974	0	0	0	0	0.04273	1
1519325887	##########	108272	109104	Flat	111	null	974	0	0	0	0	0.04667	1
1519326187	##########	105184	107856	Flat	107		974	0	0	0	0	0.04639	1
1519326488	##########	103424	106672	Flat	106		974	0	0	0	0	0.0482	1
1519327087	##########	96560	102736	Flat	99		974	0	0	0	0	0.02109	1
1519327387	##########	91680	99456	Flat	94	null	974	0	0	0	0	0.02083	1
1519327687	##########	85776	94960	Flat	88	null	974	0	0	0	0	0.02197	1
1519327987	##########	80928	89408	FortyFiveDown	73	61	779	34917	27	3928	0.97854	0.01581	1
1519328287	##########	77808	83872	FortyFiveDown	64	null	779	34917	27	3928	0.97854	0.01479	1
1519328587	##########	76816	79456	SingleDown	53	null	779	34917	27	3928	0.97854	0.01365	1
1519328887	##########	81120	77376	Flat	59	null	779	34917	27	3928	0.97854	0.0342	1
1519329187	##########	93088	79120	Flat	74	null	779	34917	27	3928	0.97854	0.10314	1
1519329487	##########	113568	86624	SingleUp	87	null	779	34917	27	3928	0.97854	0.19946	1
1519336386	##########	219040	212640	DoubleUp	241	243	747	38358	12	2633	0.98081	0.19946	1
1519336688	##########	221056	216704	DoubleUp	244	244	748	38270	6	1691	0.98643	0.0841	1
1519336988	##########	223392	219872	DoubleUp	247	null	748	38270	6	1691	0.98643	0.07189	1
1519337287	##########	227136	222592	DoubleUp	252	null	748	38270	6	1691	0.98643	0.06057	1
1519337587	##########	226880	224896	Flat	252	null	748	38270	6	1691	0.98643	0.04889	1
1519337887	##########	227200	226592	Flat	252	null	748	38270	6	1691	0.98643	0.039	1
1519338187	##########	225408	227296	Flat	250	null	748	38270	6	1691	0.98643	0.03631	1
1519338487	##########	217280	225920	Flat	239		748	38270	6	1691	0.98643	0.05381	1
1519338787	##########	218528	222816	Flat	240	null	748	38270	6	1691	0.98643	0.05511	1
1519339087	##########	215360	219072	Flat	236	null	748	38270	6	1691	0.98643	0.05784	1
1519339387	##########	211776	215712	Flat	231	null	748	38270	6	1691	0.98643	0.04693	1
1519339687	##########	195872	211744	FortyFiveDown	220	null	748	38270	6	1691	0.98643	0.1094	1
1519339987	##########	131552	198720	DoubleDown	124	null	748	38270	6	1691	0.98643	0.1094	1
1519340286	##########	189408	180352	FortyFiveDown	202	null	748	38270	6	1691	0.98643	0.1094	1
1519340587	##########	157600	164512	SingleDown	180	null	748	38270	6	1691	0.98643	0.50102	3
1519340887	##########	196800	161696	FortyFiveDown	195	null	748	38270	6	1691	0.98643	0.62192	0
1519341187	##########	147904	168960	FortyFiveUp	146	null	748	38270	6	1691	0.98643	0.62192	4
1519341487	##########	138336	169312	DoubleDown	133	null	748	38270	6	1691	0.98643	0.58585	3
1519341787	##########	180512	163488	Flat	190	null	748	38270	6	1691	0.98643	0.58585	3
1519342087	##########	167648	157856	SingleUp	181	null	748	38270	6	1691	0.98643	0.68729	0
1519342388	##########	173376	161152	DoubleUp	181	187	747	37690	5	1709	0.98896	0.69879	0
1519342687	##########	170528	169824	Flat	177	null	747	37690	5	1709	0.98896	0.69944	0
1519342987	##########	167648	174016	Flat	173	null	747	37690	5	1709	0.98896	0.70419	0
1519343286	##########	165248	172384	Flat	170	null	747	37690	5	1709	0.98896	0.64238	0
1519343587	##########	161728	167648	Flat	166	null	747	37690	5	1709	0.98896	0.5777	3
1519343887	##########	159136	163008	Flat	162	null	747	37690	5	1709	0.98896	0.54951	3
1519344187	##########	153536	159424	Flat	155	null	747	37690	5	1709	0.98896	0.43187	3
1519344487	##########	148768	156032	FortyFiveDown	148	null	747	37690	5	1709	0.98896	0.34277	2
1519344786	##########	142304	151744	FortyFiveDown	140	null	747	37690	5	1709	0.98896	0.33326	2
1519345088	##########	136512	146176	FortyFiveDown	132	null	747	37690	5	1709	0.98896	0.06732	1
1519348090	##########	106048	104864	Flat	91		747	37690	5	1709	0.98896	0.02672	1
1519348387	##########	108624	105728	Flat	95	97	750	36963	4	1349	0.98994	0.01481	1
1519348688	##########	111344	107344	Flat	99		750	36963	4	1349	0.98994	0.02395	1
1519348986	##########	114832	109696	Flat	103	null	750	36963	4	1349	0.98994	0.03472	1
1519349287	##########	115408	112432	Flat	104	null	750	36963	4	1349	0.98994	0.03745	1
1519349586	##########	116304	114816	Flat	105	null	750	36963	4	1349	0.98994	0.04022	1
1519349886	##########	115648	116192	Flat	104	null	750	36963	4	1349	0.98994	0.04092	1
1519350186	##########	113904	116208	Flat	102	null	750	36963	4	1349	0.98994	0.03828	1
1519350486	##########	111840	115088	Flat	99	null	750	36963	4	1349	0.98994	0.03643	1
1519363085	##########	249856	134752	Flat	250	135	874	31295	61	18623	0.95513	0.0388	1
1519363386	##########	189440	164064	DoubleDown	180	null	874	31295	61	18623	0.95513	0.0388	1
1519363686	##########	164896	192448	DoubleDown	166	null	874	31295	61	18623	0.95513	0.16299	1
1519363986	##########	149312	199520	DoubleDown	148	118	848	37862	56	18071	0.94517	0.17895	1
1519364286	##########	140928	177248	DoubleDown	134	null	848	37862	56	18071	0.94517	0.18455	1
1519364586	##########	135840	148096	SingleDown	124	null	848	37862	56	18071	0.94517	0.1863	1
1519364886	##########	134528	131040	SingleDown	113	null	848	37862	56	18071	0.94517	0.18656	1
1519365186	##########	133568	128288	FortyFiveDown	112	null	848	37862	56	18071	0.94517	0.18668	1
1519365486	##########	132224	131840	Flat	111	null	848	37862	56	18071	0.94517	0.18683	1
1519365786	##########	130784	134368	Flat	109	null	848	37862	56	18071	0.94517	0.18674	1
1519366086	##########	132832	133952	Flat	111		848	37862	56	18071	0.94517	0.18645	1
1519366386	##########	133120	132416	Flat	112	null	848	37862	56	18071	0.94517	0.16099	1
1519366686	##########	132704	131744	Flat	111		848	37862	56	18071	0.94517	0.12481	1
1519366986	##########	131840	132096	Flat	110	null	848	37862	56	18071	0.94517	0.0786	1
1519367286	##########	131584	132448	Flat	110	null	848	37862	56	18071	0.94517	0.03557	1
1519367586	##########	132480	132320	Flat	111	null	848	37862	56	18071	0.94517	0.01651	1
1519367886	##########	134720	132320	Flat	114	null	848	37862	56	18071	0.94517	0.00958	1
1519368186	##########	136768	133184	Flat	116	null	848	37862	56	18071	0.94517	0.01073	1
1519368486	##########	137152	134912	Flat	117	null	848	37862	56	18071	0.94517	0.00991	1
1519368786	##########	134112	136288	Flat	113	null	848	37862	56	18071	0.94517	0.01431	1
1519369086	##########	133856	136320	Flat	115	147	862	34231	63	18926	0.92196	0.01293	1
1519369386	##########	132672	135040	Flat	114	null	862	34231	63	18926	0.92196	0.0119	1
1519369686	##########	129664	133056	Flat	110	null	862	34231	63	18926	0.92196	0.01606	1
1519369986	##########	128144	131040	Flat	108	null	862	34231	63	18926	0.92196	0.01677	1
1519370286	##########	127296	129216	Flat	107	null	862	34231	63	18926	0.92196	0.01675	1
1519370586	##########	125056	127536	Flat	110	138	893	26262	63	19794	0.91372	0.0183	1
1519370886	##########	125968	126272	Flat	111	138	893	26262	63	19794	0.91372	0.01833	1
1519371186	##########	125920	125584	Flat	111	null	893	26262	63	19794	0.91372	0.01391	1
1519371486	##########	124512	125312	Flat	110	null	893	26262	63	19794	0.91372	0.00909	1
1519371786	##########	123104	125008	Flat	108	null	893	26262	63	19794	0.91372	0.00774	1
1519372086	##########	124512	124480	Flat	110	null	893	26262	63	19794	0.91372	0.00914	1
1519372386	##########	125200	124080	Flat	110	null	893	26262	63	19794	0.91372	0.01023	1
1519372686	##########	126112	124304	Flat	111	null	893	26262	63	19794	0.91372	0.01153	1
1519372986	##########	126736	125232	Flat	112	null	893	26262	63	19794	0.91372	0.00864	1

Calibration-Linear.json

[{"slope":893, "yIntercept":26262, "formula":"calibratedbg=(unfiltered-yIntercept)/slope", "yError":19794, "slopeError":63, "rSquared":0.91372, "numCalibrations":8, "calibrationType":"LeastSquaresRegression"}]

calibrations.csv

140064	131	##########	1519301888	5a8eb46718f7ba5efbccc254
166592	171	##########	1519315687	5a8eead118f7ba5efbcf8123
80928	61	##########	1519327987	5a8f1a2e18f7ba5efbd231a3
219040	243	##########	1519336386	5a8f3a9018f7ba5efbd3f079
221056	244	##########	1519336688	5a8f3c4118f7ba5efbd40613
173376	187	##########	1519342388	5a8f528818f7ba5efbd53502
108624	97	##########	1519348387	5a8f691d18f7ba5efbd65af5
249856	135	##########	1519363085	5a8fa38f18f7ba5efbd95453
149312	118	##########	1519363986	5a8fa6aa18f7ba5efbd98179
133856	147	##########	1519369086	5a8fba6a18f7ba5efbda982f
125056	138	##########	1519370586	5a8fc13318f7ba5efbdaf677

[efidoman]

Thanks for logging this issue. Can you drag-n-drop the entire file here instead of cut-n-paste? I can't read some of the columns in this format as they are #### .

How old was this sensor? From what I can see, it looks like there is some significant variability from the g5 unfiltered value in last 5 calibrations. This level of variability usually indicates an old (or dying) sensor. The unfiltered value comes directly from the g5 sensor and the calibration comes from the meter.
It looks like noise level was high as well which is expected also when meter BG values are so far off the sensor values.

[tynbendad]

also, is it all the same sensor or is there a sensor change in the midst (i don't see one but just want to confirm).

[tynbendad]

was this calibration correctly entered? seems like it should have been closer to 235 than 135?!

249856	135	##########	1519363085	5a8fa38f18f7ba5efbd95453

we probably need to find a way to throw out calibrations that don't make sense.

[tynbendad]

Filtered diverges a ton from unfiltered at that same time, so something odd was going on with the sensor, probably not a good time to calibrate when they diverge:
1519363085 ########## 249856 134752

[efidoman]

Yes, when they diverge like that, in my experience the sensor is on the way out. this brings up an enhancement idea ... shouldn't Logger be able to present this type of divergence to the user? Should noise factor divergence of filtered / unfiltered raw data from the sensor? I think it should.

[alimhassam]

+1 xdrip+ reports something like 'unclear sensor readings' when filtered and unfiltered diverge

[tynbendad]

noise can factor it in, but separately what about calibration - i think we also need to avoid using calibration data like that, since the noise can go away but the calibration will still be very broken.

[efidoman]

Agreed on both points. These changes are definitely something we need to do.

[efidoman]

Where / how should Logger report unclear sensor readings when filtered and unfiltered diverge? Is there a way to get that over to NS in the BG record?

[alimhassam]

I'm not sure but for sure it should report a high level of noise when that's the case. Sometimes the BG is being reported as 38 or 39 in think. Will look it up.

[efidoman]

Looks like xdrip+ uses 38 for "error" ... and xdrip+ uses a hard range of only allowing 39 to 400 to go through.
https://github.com/jamorham/xDrip-plus/blob/d35274d20133c32008a21ed60a209b68bd7ff9bc/wear/src/main/java/com/eveningoutpost/dexdrip/Models/BgReading.java#L538

What does NS show for BG 38 value? Let me test that.

[efidoman]

[efidoman]

That's how 38 shows up in NS. Probably a good error indication.

[tynbendad]

we did disallow calibrations a while ago, when there is a high difference between filtered & unfiltered ("variation"). also fwiw, fixes for noise calculations and additional noise indications when native data is unavailable, are also going on currently...

[efidoman]

Noise refactoring is complete. We are all set for now.