The Kolmogorov–Smirnov (KS) test is one the most popular goodness-of-tests for comparing a sample with a hypothesized parametric distribution. Nevertheless, it has often been misused. The standard one-sample KS test applies to independent, continuous data with a hypothesized distribution that is completely specified. It is not uncommon, however, to see in the literature that it was applied to dependent, discrete, or rounded data, with hypothesized distributions containing estimated parameters. For example, it has been “discovered” multiple times that the test is too conservative when the parameters are estimated [e.g., 39]. For the simplest one-sample setting, a comprehensive review of the misuses and demonstration of the mechanisms will be of great value to practitioners.
This project aims to survey the misuses of the KS test, demonstrate their consequences through simulation, and provide remedies as needed. Several specific tasks are to be completed with different levels of challenges: 1) demonstrate that the test is too conservative with estimated parameters and the issue can be fixed with parametric bootstrap; 2) demonstrate that a different null distribution for the KS statistic is needed when the hypothesized distribution is discrete or mixed and how effective the existing software implementations are; 3) demonstrate how discrete data caused by rounding can change the size of the test and how this can be fixed; and 4) demonstrate how serial dependence can change the size of the test and what modification is needed. Completion of three or more tasks may lead to a paper suitable for a journal such as American Statistician.
[39] D. J. Steinskog, D. B. Tjøstheim, and N. G. Kvamstø. A cautionary note on the use of the Kolmogorov–Smirnov test for normality. Monthly Weather Review, 135(3):1151–1157, 2007.