KSampleADTest extrapolation issues (can get p-values greater than 1)

[c123w]

KSampleADTest can easily give wildly incorrect p-values (sometimes greater than 1). For example:

using HypothesisTests
a = [0.4189684779152252,0.5468496855962497,0.259911323159399,0.9056052215860342,0.8234748232573696,0.2082040599799002,0.6524998296887534,0.30796887962238495,0.19344617750107518,0.6190907094780413,0.7978105881966331,0.5016548405492893,0.5692803901434094,0.8841606299457867,0.8676477595923742,0.38457584680642953,0.3123024178738889,0.8358388206393303,0.5952970002936131,0.18386550316331007,0.16417023996707258,0.11258501383765118,0.992007142608603,0.2162375705380748,0.3994502084617857,0.41330536074841295,0.04498832135306796,0.43613735630399453,0.9844431627830721,0.3371547868923841,0.23562416314085843,0.8468715173473391,0.9534597248481105,0.6033155717467349,0.8531095439002176,0.6046247236431788,0.06990970635223115,0.0008454376632107685,0.7126207860829357,0.5041306794627083,0.6807082607002515,0.9100471021835321,0.8503036010484266,0.9734711170569141,0.48334281453064243,0.06240407526437042,0.47595957755833807,0.5569272404818058,0.4282789436045835,0.8502706167498921,0.7391159039350399,0.9877525838587351,0.04883687283552418,0.36744113760528085,0.2644478607144334,0.712120196361852,0.5610116375099059,0.8933243965813906,0.7414518741649023,0.9971723036795488,0.4143782441084436,0.9836597654969848,0.13273348580377986,0.28499123185767106,0.09551928821152078,0.7764369048423534,0.8046562516212368,0.1748607230951562,0.10928154750504548,0.4048213468310049,0.302573127281228,0.8724676862379306,0.7488674575416581,0.13820288314909668,0.027707224672159247,0.8533193652849684,0.6302759366423039,0.6601036062673957,0.5739016001260995,0.5635909220178505,0.5159576355732565,0.6577807764849697,0.16692317165339166,0.8155372119635933,0.8359999483977543,0.42057325280324576,0.586469460053574,0.9890634710924773,0.9604087143315396,0.9285348821238562,0.3146090115699911,0.23665043676665953,0.9260426950345326,0.9724785091326891,0.8610564373233927,0.07668225096823678,0.110997261929064,0.5413955999147215,0.6065753036543795,0.6291172870991586,0.1220336010857892,0.012313183465417321,0.7234544395378384,0.8620236036519484,0.9907149368234969,0.3932556455422964,0.31641239782869857,0.9467475457550569,0.8672423577142647,0.11080528416383051,0.452931455416145,0.9084491310325631,0.6761642702869577,0.0501808627541207,0.9244545726380766,0.5434370793927998,0.7722739846443234,0.6771460328313235,0.10984149054736059,0.03455035408365825,0.6504446539516737,0.2061792349497531,0.3013724029505829,0.6378044284968152,0.37902118295384857,0.44399133124518997,0.33056780523308626,0.5292035411819163,0.15680014449485724,0.9231677979855226,0.9631091596282855,0.14371174295578415,0.7323360067409463,0.35577860561423025,0.21108649171292315,0.6041964994981581,0.40087693432251226,0.2141965976745419,0.037712133692213046,0.39825362504717776,0.39039262189413604,0.5127806573675797,0.6421154504754454,0.7448517776662362,0.3485480121682829,0.9443450745472006,0.14789226870112504,0.9194897771226409,0.5050662822029339,0.39231013591526875,0.8504281783051044,0.7376341034881968,0.5287387009539068,0.056637937489939505,0.017194395589651634,0.6234738524330081,0.45917781766792065,0.16363061211517227,0.5875306575258208,0.7146429457566221,0.4640787869898537,0.24922943137959308,0.3114428247202703,0.22580527983184018,0.6515748993176014,0.08260249266092834,0.17378966877099167,0.4987185391065059,0.44506786103975715,0.29572862603125083,0.16335100297821592,0.5992875780699412,0.3364829233322908,0.7126631957323899,0.46218358938771686,0.3242284241576019,0.07967648559219409,0.03058460601247237,0.02736995154948163,0.22108075540433703,0.2219999803030579,0.7912716615248419,0.6000355496821821,0.9323428729960375,0.3100035710332152,0.7770181180002653,0.9454612365598769,0.7675074506926107,0.32903466001841664,0.9195123541081214,0.023694278775387767,0.7732269890869956,0.23643778064896326,0.86453335250675,0.925627604419569,0.023160662922954156,0.17028525804117378,0.5838237017736096,0.3407534103352534,0.20936930776273144,0.2825909442386809,0.9537405912172912,0.6982076623213516,0.8246863298993956,0.7670351526980743,0.6221159318271692,0.25761892209283577,0.6836496533374379,0.48564226417730505,0.5365274925878925,0.10510071946028376,0.8349016959383768,0.5249714320530305,0.06275039577943109,0.2576129459872234,0.360699686955428,0.094354999244054,0.14750485144698944,0.26320234971018697,0.8920892856945386,0.28771945538179833,0.8382261652843237,0.15312610661151838,0.2306667483741245,0.5636777040976555,0.005897514305174489,0.0010500864027274126,0.1418114561318793,0.055495191629697294,0.604721227143227,0.616718839149881,0.06984281447640805,0.3157282788311284,0.5243649624299367,0.49881700616537383,0.47055285495327825,0.9676073887552561,0.26770346578987114,0.305590416131446,0.18906638864075842,0.9371595808318269,0.7391795428766341,0.6574562369629531,0.0162605599146588,0.934427187769538,0.8046726276651681,0.5546725871368108,0.47910295841048045,0.6592411200889325,0.07862328870230506,0.6155234478787879,0.9510502490406725,0.2862279248003856,0.07273661097112916,0.14732628075626075,0.45804237805466963,0.57274606367436,0.6691260068679981,0.9350407811073493,0.7483186411824494,0.9673796820911607,0.6531746570706936,0.10826718475627772,0.7058829393325188,0.18837917804695525,0.5296930048598238,0.4137107883620934,0.6215754897423327,0.709110713496172,0.7569433034123147,0.15284691980339038,0.2142279940141385,0.04369366711334588,0.41368312385223693,0.5470033260314737,0.6778976271621966,0.6439632071371784,0.6663362266760524,0.8570073695264544,0.06491805384885252,0.9585105504232607,0.8503309581274336,0.758498904688359,0.42719349490034353,0.21617144847593162,0.7278889619440652,0.26180993234845307,0.6408197352305371,0.06097974832103503,0.08312872503240509,0.6780898752811677,0.5010035008570148,0.5396419889479531,0.5748113102609917,0.911199364786335,0.04485731735673748,0.02156760071911923,0.8563789989925004,0.18969220029712508,0.9880206316974269,0.2866657671146833,0.7770592610222922,0.6822232307336618,0.7197234229663354,0.26978523949464983,0.6780755433226091,0.3257382809988687,0.13286984365450327,0.5522218535681112,0.36356646515175695,0.5364865891852983,0.43438638548925845,0.7489062790605372,0.7509368295442114,0.20677445119652638,0.9276969490999616,0.07936740860369818,0.6384018003273133,0.10061378803215448,0.8216568270781228,0.38569823509782,0.4727947764803302,0.24269589417941062,0.2526484103182167,0.9890287294987419,0.8546362577345834,0.31067149117933224,0.6033412899133461,0.05717682688595693,0.0017617498620348115,0.3985460084696124,0.8876723628667136,0.5014085270490358,0.2099789049505394,0.9579282574907415,0.8650672724061095,0.4777105605398544,0.3497926804998843,0.49042946103971596,0.6760314228709647,0.23555623984174434,0.8070776379718816,0.989248857207953,0.24544063459811483,0.3493385193631271,0.7031707895102561,0.9916372743209931,0.6644125503534732,0.08468822089453054,0.7556888235737584,0.062371994460312496,0.290187295846023,0.6903863451272121,0.7711691439939503,0.0793572945816019,0.32597499335583513,0.0025645129749771733,0.45116512797397657,0.1220362661964991,0.7881137925362582,0.5950000184950484,0.16705806813432944,0.8606792119129254,0.8076606495771945,0.9995351524492262,0.7941083170774781,0.2591658115015658,0.8813063655263857,0.26090075016701486,0.8029229027897169,0.008879057965859527,0.13309204002149233,0.23972877834466555,0.23220791763564108,0.43097977696399403,0.35143759495753324,0.6803661450159242,0.832984287052549,0.5205379906505017,0.652752739884664,0.7773713952844112,0.05515767651031278,0.3305630644898221,0.5583457154418237,0.08024993773362255,0.037297384545793566,0.21146571803552727,0.7244259827670807,0.4992245671343536,0.04923944791989321,0.5706024549720725,0.33176855787034776,0.14562777229379797,0.128813503025494,0.5635240285034164,0.581829195372475,0.39149751345621087,0.9587396593278525,0.5679683486925255,0.5046180773452127,0.2274458592259927,0.024446457739114802,0.39059644174936037,0.5657963715154468,0.2625690679499133,0.315609187635183,0.8812226757649135,0.16689887380714596,0.2786071278901856,0.9349387953970285,0.28339702722751836,0.619306301973152,0.164187616355697,0.7545210179528827,0.6789311217457537,0.4340037394555729,0.3377539295924412,0.09614142314040985,0.19188210773096248,0.09154638891859457,0.901014577328012,0.9179594184735218,0.20805134371152567,0.7268252767796932,0.45115455463943777,0.2391919936422231,0.013764517625357131,0.8582802333358717,0.769429452840418,0.267791246605084,0.060303454509072374,0.6374807800889828,0.0958824260797091,0.4347740626557324,0.6928892171432908,0.17135617189514507,0.5173840892005526,0.38890996016938084,0.17202744613286836,0.5774113990400693,0.5070563244150872,0.4139697814108805,0.3524920358225412,0.6658066201972943,0.6333196701117076,0.7756418440423898,0.5349177543685291,0.4911906629710032,0.2632771746527449,0.7731111995821642,0.7929872145390631,0.12025052199344244,0.3666009460098201,0.71306885676261,0.638321407124214,0.04631485851588235,0.8026743947774653,0.35268145194522704,0.7062448143617477,0.8273768139519415,0.3441560188830193,0.29558912469527154,0.1763278493427376,0.6348035939884522,0.6118703020582477,0.8350569698896826,0.7945205320779203,0.34694430060442705,0.13177962559835787,0.6747628242815222,0.38817240931446295,0.7966856854765525,0.028671026809185873,0.5280613640655616,0.41142334151174187,0.20144386564233785,0.39274681593601835,0.17553742944769324,0.9248670259004284,0.44055219330710904,0.6110467475470109,0.01256272202450992,0.022777699363353276,0.11774549499045195,0.6045624560841738,0.2165016370352726,0.3588316121352577,0.14113071985212722,0.9381253154823166,0.9690391423283848,0.13099318138330052,0.8755335582191786,0.27052134367630964,0.8033948068085368,0.10333891133476225,0.4309456634777049,0.9640556389476262,0.4910021558478508,0.4082849969107405,0.9128480305912992]
b = [0.981191532663592,0.1794494676980738,0.07445684757523852,0.717040565071009,0.5904074330286249,0.6009558492137124,0.8167791785788463,0.7566166617066223,0.7516786393841439,0.6052000742894792,0.13739224601503208,0.2225668889036505,0.28222744544445866,0.21381441379922594,0.05457440246688128,0.2812951520100708,0.698430266866416,0.643032261089667,0.810235256833373,0.872514550362159,0.5894561898401505,0.34233063372229244,0.3773978430294671,0.4183459849514082,0.5025561169415751,0.26478022803542944,0.7886543861302013,0.8269625261724973,0.9088722847279438,0.038777083908451804,0.36880554339183713,0.38808394718822803,0.7109248509613002,0.5373317523402572,0.2649721190871339,0.7503625322414988,0.5137935766058848,0.6173975484135046,0.9287081293997788,0.41317695263734744,0.9401475181462025,0.8357669472094071,0.13344020254932487,0.36286494618029663,0.5521920734041434,0.8795987255378066,0.9575535582677099,0.41233207071852473,0.062441848282672296,0.08230703132169381,0.29445871011937275,0.5197861673154343,0.7514695395212605,0.9841885475203251,0.2991373352386568,0.5610352636639424,0.12306385502747319,0.04181900721428544,0.4561976182936929,0.9058425757026343,0.04019624338926531,0.33751349011815535,0.433653012182484,0.3176628595926365,0.6029303084571325,0.0013968270170867658,0.08974628517262762,0.2293382564896771,0.2971858916563106,0.30720799480298555,0.0580769142142199,0.6049365616396258,0.8875025793860383,0.05400781096700369,0.4984995787894473,0.04795031031900443,0.9171725430067283,0.43468514462186225,0.732208032823003,0.24239427911355316,0.8046736792345304,0.9060042251906646,0.707178153837879,0.4909168234942398,0.6303570368028732,0.020392275722391107,0.7701448227181544,0.36129309113115515,0.7791404245557889,0.7750955001680941,0.18464508908211696,0.9789361578452802,0.9129867184875016,0.6121794902831728,0.010686818878296256,0.6495748494847058,0.7614357070516575,0.33990452852476327,0.5544927852002128,0.7605108188285745,0.40344936166697076,0.6444996733245001,0.6471089688512064,0.9528283851870589,0.07410180205425299,0.3321226701230111,0.6181913612593848,0.22711775132986367,0.5548914926676778,0.6030628662457946,0.6761819279769663,0.2448172921695484,0.7104307516882928,0.4772453661007918,0.07720289618609644,0.019287070082233404,0.021706394903487247,0.006856858995857973,0.6266494938107421,0.8696645196311998,0.7428367563900509,0.5694886958442251,0.0315456204405169,0.3851527427735233,0.599085909665011,0.517160901010167,0.13302509040641808,0.484069146074781,0.41257157173965564,0.8418728156929849,0.608561482189385,0.9243985691287835,0.6212190994990126,0.2371376481419989,0.35466167692783523,0.46117992881288594,0.16485667833117823,0.6702546530315889,0.01560843874443063,0.7556586226615853,0.03496152742345893,0.18386174836932923,0.5560272802084592,0.488166472483049,0.757854795140003,0.6656642676826854,0.5914199410663217,0.014153010252113463,0.9746792545789993,0.13113479882631784,0.19546216178479425,0.49434827638890266,0.910971554957815,0.6982532558429713,0.5983862464527907,0.1151654535266815,0.7832658510099628,0.10335171107730678,0.16165625988441779,0.6048998343053349,0.313398970806436,0.5053718771651996,0.05361050961406555,0.6419071183816216,0.17029971463977134,0.7966380163254756,0.7284340497973518,0.355339736176296,0.1746422235888343,0.09217232110285534,0.8758501942505474,0.7144645229507063,0.4610418821346647,0.10236983779618414,0.7635540200240956,0.2822721853197474,0.3999891730302738,0.20864924369424087,0.18714879319527666,0.1955781728542254,0.061570335530285014,0.4292442066044506,0.9106551702404966,0.07970035169088696,0.9211693556947249,0.2560110766003023,0.34206652702273077,0.7572340977274346,0.9463905619843169,0.37078334206393726,0.5529899337214577,0.21075986076072772,0.32200577762773275,0.16228519219244641,0.9832269672659268,0.062366975541621184,0.2996206953465317,0.40588448006204225,0.4763386349208576,0.4205916880758893,0.7847221497330057,0.9405865450960105,0.07795290318877668,0.25167748718056315,0.5355719026312291,0.6453842098925087,0.16508126285870306,0.8846665723984106,0.8805958563527736,0.16287375222192635,0.5280285186673714,0.3160319800781872,0.7853461436945088,0.8053246924555295,0.8289392758315548,0.17024636481720878,0.502978006709117,0.5410969061580393,0.5840771413556667,0.31729592650333216,0.5308839628687554,0.09329377370249037,0.04541294943586993,0.22748413686993119,0.6068238946457452,0.26540067493536323,0.5431844785183315,0.3263510758223991,0.737477010507805,0.8208548092922441,0.8805582342812672,0.1823824418553639,0.6680885955224751,0.09008546807675577,0.13453825430125588,0.23030377244037314,0.2854321927384562,0.12168674726841444,0.8269863781888307,0.054410942760244385,0.17347612922960454,0.47834641552858814,0.36904927550167455,0.8608861420892735,0.818174087088966,0.45041565269012707,0.5568836548850633,0.4328376870819677,0.7015676365933095,0.4091570392008905,0.8657465652156977,0.3444659918036097,0.6695639415316572,0.1740031177422401,0.896710691792427,0.6726702270051867,0.3232599840044448,0.34732848093586766,0.46995250144276013,0.6305516886048643,0.9998106512908334,0.7165061722269608,0.17121309988382882,0.1494741602931169,0.9390415390273577,0.203064285022585,0.3739610238822648,0.8124377801959575,0.4458015296513511,0.25694257900384065,0.04745055830123168,0.8041409666547044,0.3127588830043897,0.4692001099084766,0.2202200903550091,0.10887398348159216,0.7533543804871532,0.045992769531737254,0.241149267297726,0.21866023635304965,0.26250851743821646,0.3835232229200578,0.3445418140566665,0.9985373905741894,0.32422373631127543,0.9735164047588032,0.3386798665720412,0.7892264671709126,0.6577064262773407,0.7307739064333747,0.5066386601667168,0.8329056266413262,0.5222256363818929,0.3093418250166786,0.8502162376335423,0.9032249834041177,0.6576580131486536,0.5317888331973113,0.44761243888174973,0.6023196742408414,0.9121370038439689,0.45854342943698034,0.387634228848901,0.39258564552590647,0.0853985854491972,0.46979935881546253,0.11414473587375729,0.6420760538417014,0.12258621421647664,0.8292633980845021,0.38648124428392205,0.017972377826448893,0.034379264720171276,0.9638813272988369,0.5999673695762939,0.9939451411970774,0.14535749678788346,0.24978999710847205,0.9279493581518616,0.09618976355446107,0.10938179752960031,0.6579713379412404,0.5820484652406896,0.4315092378455758,0.5234258424775984,0.24508653829784932,0.18945890594046344,0.18867591830371255,0.3474418019680039,0.4253005766270066,0.3309818358421015,0.42004807691386636,0.6096128951719082,0.19154630579555798,0.07071421727469374,0.4465070714135486,0.03275269014164661,0.9776336405156714,0.10109740454184579,0.46712661329079475,0.23000027132165135,0.09070813945191913,0.09039450173950292,0.2302430020288635,0.954539978033931,0.0018774644380872285,0.9194807449853832,0.9226997407120836,0.411728431649075,0.9313215691140526,0.8094424940913076,0.5461884281531915,0.7454378623287965,0.24539807367461286,0.5469498077341268,0.5571081059695735,0.41817456866336866,0.7850546763510446,0.2678927984338897,0.11213849209031856,0.8507620076073859,0.05762285980052484,0.8131434213316433,0.42736597921318364,0.5170611018575959,0.6494021621856161,0.5040084228957011,0.40501549502706213,0.03235801594656418,0.4593445963702796,0.0777917656290017,0.06967957884223797,0.6985586388650091,0.7159121302631752,0.26622098934051186,0.9119649011975237,0.9777814702210554,0.031172820979689764,0.6340101827421094,0.23272609376830355,0.7416004265800216,0.6512403373103663,0.09040523951626622,0.9480321752167851,0.14968550868813524,0.413342563419689,0.8010157179669735,0.4839631156493305,0.945368504331838,0.22537685166698274,0.6790114384362569,0.898746035802279,0.6365848449191593,0.872358454671347,0.12119343183121067,0.2204166310054505,0.6801866641051806,0.6934055399755816,0.9767138203031078,0.8672161300701426,0.8802666785554973,0.9399722970869806,0.14755620701719363,0.8086912665216628,0.6826350219922539,0.9826165724668139,0.903243347439419,0.7895581566753493,0.01256644064999457,0.228174696670554,0.4699158479344496,0.38328406855311425,0.48964520300766456,0.9047812482449968,0.30404064888825055,0.8287847346389694,0.8847853622810389,0.7983925142448525,0.179964609228908,0.39189531455601,0.6988438960037613,0.010667752463447977,0.7672957577612916,0.5394406129399765,0.06460760649593866,0.5529117183768304,0.9133143674905844,0.386799335257322,0.42689455354323,0.02281595864612007,0.5023647580181267,0.40797884017382335,0.8605340827843886,0.44861269192314057,0.14970581163298546,0.5780395267806155,0.12309295818713606,0.024114342927236354,0.548299463959913,0.6050415113670633,0.7844296658781209,0.7163298227345263,0.17892646902879927,0.09409696144815793,0.4289999788638006,0.5218505902298844,0.3709348633903826,0.7820242526107428,0.08387508628443441,0.7618644907411163,0.10529223638211604,0.48052435419936046,0.5807426949599552,0.9959152997764489,0.6805795675853035,0.8435904746789369,0.41974040281240543,0.5623344831200459,0.6967515788618159,0.8469891764882023,0.2743599681330988,0.24027759677129912,0.980371724241156,0.3287194995789289,0.0428947185624029,0.6768536423438218,0.5663554348126028,0.6045751623887068,0.8660238228874511,0.8955799941566269,0.6623693262388464,0.9898802910590807,0.004327411665248082,0.6564310531550861,0.7542273910364476,0.9743153640980879,0.8973668081845971,0.20928177256724134,0.1308989208975193,0.37748202691219324,0.7598114175986912,0.7323190735654102,0.2900685507178351,0.11512346467567136,0.30861880833584987,0.6277111255542247,0.5157738336507798,0.8756146921141226,0.3230659012873629,0.8126772663420263,0.5413980921758912,0.053452651203931056,0.19847483708469782,0.27607165373907194,0.6688664523954262,0.18117304352073527,0.11471059280260643,0.6434002659407931,0.5602836945615035,0.28826878189454375]
print(KSampleADTest(a,b))

gives two-sided p-value: 1.2192184260000214 (not significant).

a and b were generated with rand(500)

[ararslan]

A p-value greater than 1 provides strong evidence that there's something wrong.

I think @johnmyleswhite mentioned recently that our Anderson-Darling test doesn't work well in practice. Perhaps he would have some insight here?

[wildart]

Our asymptotic p-value calculation is not very precise for extreme cases. We extrapolate just from 4 points. R package kSamples does much better job with 36 points. But other then that we are up to par.

R> ad.test(a,b,method="exact")


 Anderson-Darling k-sample test.

Number of samples:  2
Sample sizes:  500, 500
Number of ties: 0

Mean of  Anderson-Darling  Criterion: 1
Standard deviation of  Anderson-Darling  Criterion: 0.75993

T.AD = ( Anderson-Darling  Criterion - mean)/sigma

Null Hypothesis: All samples come from a common population.

Based on Nsim = 10000 simulations

               AD   T.AD  asympt. P-value  sim. P-value
version 1: 0.1113 -1.170           0.9999             1
version 2: 0.1100 -1.172           0.9999             1

julia> KSampleADTest(a,b)

k-sample Anderson-Darling test
------------------------------
Population details:
    parameter of interest:   not implemented yet
    value under h_0:         NaN
    point estimate:          NaN

Test summary:
    outcome with 95% confidence: fail to reject h_0
    two-sided p-value:           1.2192184260000232 (not significant)

Details:
    number of samples:        2
    number of observations:   1000

SD of Ak:               0.7599320527202672
Ak statistic:           0.1097192326443887

[johnmyleswhite]

I suspect there are two problems: our implementation of Anderson Darling is imperfect and the test itself just doesn't work very well without very large samples. Someone I respect highly has spoken negatively of the test's accuracy in the past, but I'm struggling to find papers indicating the core issues with the test.

Possibly worth reading for someone interested in looking into this issue: https://www.jstatsoft.org/article/view/v009i02/ad.pdf