SUMM: Roundup of sandbox exposure, unused files, redundant code, ...

[jbrockmendel]

Collecting these in one place to avoid Issue proliferation; will update as appropriate.

Sandbox

Parts of sandbox that are imported from non-sandbox (cc @ChadFulton if you were wondering how much of the code implicitly relies on sandbox):

• sandbox.stats.multicomp
  • stats.multicomp imports tukeyhsd and MultiComparison (there is only one function defined directly in stats.multicomp)
  • examples.try_tukey_hsd gets tukeyhsd and MultiComparison directly from the sandbox module, could get it from the non-sandbox file.
  • stats.contrast imports contrast_allpairs (also duplicated in sandbox.stats.contrast_tools)
  • stats.tests.test_qsturng uses get_tukeyQcrit
• sandbox.tsa.fftarma.Armafft
  • In several of the places this is imported it is only for the generate_sample method, which is defined on the non-sandbox parent class ArmaProcess. Use ArmaProcess directly.
  • The only remaining place where fftarma is imported is in test_arima_process, where it has two tests for Armafft. These could be moved to a test file in sandbox specific to fftarma.
• sandbox.stats.diagnostic almost all of stats.diagnostic comes from here.
  • The good news is that aside from this pass-through, the only direct import from sandbox.diagnostics is in examples/ex_arch_canada.py for acorr_lm.
• statsmodels.sandbox.regression.predstd is imported in numerous test and example files to get wls_prediction_std (looks like 10 imports total, 13 if we count notebook files)
  • Also in stats.outliers_influence and graphics.regressionplots (i.e. user-facing code)
• sandbox.regression.penalized is imported in several test files for TheilGLS. This includes a test_theil file in regression.tests that sends mixed messages as to the status of this file.
• sandbox.tsa.garch is imported in statsmodels/examples/tsa/ex_arma.py for its Arma class
• sandbox.nonparametric.kernels is imported into 2 files in nonparametric and 3 files in nonparametric.tests.
  • sandbox.nonparametric files are imported into 6 examples/ files
• sandbox.distributions.mv_normal is imported into distributions.mixture_rvs and distributions.tests.test_mixture
  • sandbox.distributions.sppatch is imported into examples/ex_generic_mle_tdist.py
    • sppatch modified scipy objects in-place. Can we avoid this?
• sandbox.tools.cross_val.LeaveOneOut is imported into stats.outlier_influence. This class is also duplicated in nonparametric._kernel_baseSlightly different behavior with the same name.
• sandbox.stats.runs provides Runs, runstest_1samp, and runstest_2samp directly to stats.api.
  • The same three are imported in stats.tests.test_nonparametric, along with sandbox.stats.runs.mcnemar
• statsmodels.sandbox.panel.random_panel is imported for PanelSample in regression.tests.test_theil

De Facto Sandbox

Non-Sandbox code that belongs in sandbox or dustbin:

• nonparametric.smoothers_lowess_old (Is smoothers_lowess_old needed? #5142)
• tsa.descriptivestats
• tsa.mlemodel (MAINT: remove tsa/mlemodel.py #5139)
• tsa.arma_mle
• tsa.varma_process
• miscmodels.try_mlecov
• iolib.stata_summary_examples (iolib.stata_summary_examples.py useful? #5061)
• iolib.tests.test_summary_old (test_summary.py, test_summary_old.py have no tests #5064)

Chopping Block

Partially moved from #5137
Files that appear unlikely to become useful.

• README_l1.txt (Make readme_l1.txt a section in README.rst #4957)
• statsmodels/
  • info.py ([MAINT] Remove info.py #4981)
  • examples/t_est_rlm.py (MAINT: remove t_est_rlm.py; closes #5100 #5109, sm.examples.t_est_rlm --> supposed to be a test? #5100)
  • examples/tsa/lagpolynomial.py
  • graphics/plot_grids.py (SUMM: Possibly Unnecessary Files #5137 (comment))
  • statsmodels/satasets/anes96/src/anes96.csv appears to be a duplicate
• statsmodels/sandbox/
  • tests/datamlw.py (MAINT: remove exact duplicate file datamlw.py #4941)
  • tests/model_results.py (MAINT: remove file that just says to remove it #5070)
  • examples/ex_kaplan_meier.py (MAINT: remove example file corresponding to removed module #5071)
  • mle.py - starts with "What's the origin of this file? It is not ours"
  • nos4.mtx - only used by mle.py
  • mlogitpath.lyx - no idea why we would need a lyx file
  • formula.py - supplanted by patsy (remove sandbox.formula; long-since superseded by patsy #4940, MAINT: remove sandbox.formula, supplanted by patsy #5692)
  • examples/ex_formula.py - based on formula.py (MAINT: remove sandbox.formula, supplanted by patsy #5692)
  • examples/ex_formula_factor.py - based on formula.py (MAINT: remove sandbox.formula, supplanted by patsy #5692)
  • contrast_old.py - only used by formula (MAINT: remove sandbox.formula, supplanted by patsy #5692)
  • tools_old.py - only used by formula (MAINT: remove sandbox.formula, supplanted by patsy #5692)
  • dataset_notes.rst - likely redundant with README in datasets directory
  • bspline.py (imports from models import _hbspline which does not exist AFAIK)
  • mcevaluate/mcresuts_arma1.txt
  • regression/ols_anova_original.py - OK I haven't looked it this closely, but it looks like it was superceded and then never removed.
  • regression/notes_runml.txt
  • tests/test_bspline.py.txt - if no one noticed this was a .txt, cant be too important
  • tools/TODO.txt
  • tsa/notes_organize.txt - still refers to scikits.statsmodels
  • stats/notes_fdr.txt
  • regression/sympy_diff.py
  • panel/sandwich_covariance.py
• tools/
  • generate_formula_api.py (MAINT: remove file attempting to automatically generate an api.py file #5118)
  • code_maintenance.py
  • gh_api.py
  • github_statspy
  • matplotlibrc.qt4
  • matplotlibrc.qt5
  • README.rst
  • update_web.py
  • binstar/
  • check_dirs.py
  • backport_pr.py
• docs/
  • GLMNotes.lyx - ? not touched since 2012
  • GLMNotes.pdf - ? not touched since 2012
  • fix_longtable.py - ? not touched since 2012, appears unused
  • source/glm_techn1.rst.TXT - "Just a placeholder", last edit 2013
  • source/glm_techn2.rst.TXT - "Just a placeholder", last edit 2013
  • source/regression_techn1.rst.TXT- "Just a placeholder", last edit 2013

Duplication

Copy/Pasted code that should be imported instead:

• sandbox.tsa.garch copies all of LikelihoodModel (LikelihoodModel "copied from model.py" should be removed from garch #4422)
• LeaveOneOut class in both sandbox.tools.cross_val and nonparametric._kernel_base
• LikelihoodModel._fit_colinear comment "Copied from PR ENH: measures for multicollinearity #2380 remove when merge" (“Copied from PR #2380 remove when merged” #5023)
• gof.gof_binning_discrete comment "copy/paste, remove code duplication when it works" (gof.gof_binning_discrete "# copy/paste, remove code duplication when it works" #5125)
• base._parameter_inference._lm_robust comment "this is a copy from stats._diagnostic_other" (MAINT: wrap lm_robust instead of re-implementing #5772)

Disabled Code

Commenting out code is a way of temporarily disabling it that tends to become permanent and make things more difficult for subsequent readers.

• tsa.interp.denton.denton
• discrete.discrete_model.Weibull
• sm.open_help (is sm.open_help ever defined? #5134)

Namespace Issues

• tools.tools namespace is counter-intuitive (tools.tools namespace is counter-intuitive #4992)
• tools._testing and tools.testing: either choose just one or make a clear distinction between what belongs where (tools._testing and tools.testing #4876)
• What differentiates ...?:
  • examples/
  • statsmodels/examples/
  • statsmodels/sandbox/examples/
  • statsmodels/sandbox/distribution/examples/
  • docs/source/plots/

Unsorted

• miscmodels.tests.test_generic_mle contains tests for GenericLikelihoodModel, not anything in miscmodels (miscmodels.tests.test_generic_mle doesnt import miscmodels #5763). If, as I suspect, miscmodels serves primarily as tests/examples for GenericLikelihoodModel, that should be made explicit and implement systematically.
• Many files like e.g. sandbox.regression.example_kernridge that may belong in e.g. andbox/regression/examples/?
• Overlap between datasets/README.txt, tools/dataset_rst.py, sandbox/dataset_notes.rst, possibly others in docs/?

[jbrockmendel]

Moving from #5145

Mangled, Commented-Out, or Otherwise Disabled Tests

With pytest.mark.skip or pytest.mark.xfail we can easily find things that need attention. Other methods of "temporarily" skipping a test get lost easily. See also #5768.

• nonparametric.tests.test_kernels.T_estCosine
• nonparametric.tests.test_kernelsCheckKernelMixin. t_est_smoothconf_data
• test_data.TestDates
• test_data.T_estWaldAnovaOLSNoFormula
• test_discrete.pvalues ([TST] xfail/skip instead of commenting-out/mangling discrete tests #4983)
• test_discrete.test_cov_params ([TST] xfail/skip instead of commenting-out/mangling discrete tests #4983)
• test_discrete.test_normalized_cov_params ([TST] xfail/skip instead of commenting-out/mangling discrete tests #4983)
• test_discrete.test_jac ([TST] xfail/skip instead of commenting-out/mangling discrete tests #4983)
• test_discrete.test_predict
• test_discrete.test_predict_xb
• test_sarimax.test_bse_approx
• test_glm_weights.test_basic ([MAINT/TST] skip/xfail tests instead of mangling/commenting-out in genmod, regression #4980)
• test_glm_weights.test_compare_optimizers ([MAINT/TST] skip/xfail tests instead of mangling/commenting-out in genmod, regression #4980)
• test_robustcov.test_hac_simple ([MAINT/TST] skip/xfail tests instead of mangling/commenting-out in genmod, regression #4980)
• test_robustcov.test_fvalue ([MAINT/TST] skip/xfail tests instead of mangling/commenting-out in genmod, regression #4980)
• test_robustcov. t_est_fvalue ([MAINT/TST] skip/xfail tests instead of mangling/commenting-out in genmod, regression #4980)
• test_robustcov.test_f_value ([MAINT/TST] skip/xfail tests instead of mangling/commenting-out in genmod, regression #4980)
• test_robustcov.test_confint ([MAINT/TST] skip/xfail tests instead of mangling/commenting-out in genmod, regression #4980)
• test_generic_methods. T_estWaldAnovaOLSNoFormula ([MAINT/Test] skip test instead of mangling name in test_generic_methods #4978)
• test_dynamic_factor.test_mle (test_dynamic_factor.test_mle should be xfail, not skip #5115)
• examples.t_est_rlm (sm.examples.t_est_rlm --> supposed to be a test? #5100)
• test_diagnostic.notyet_atst (test_diagnostic.notyet_atst --> should be made a test? #5085)
• test_qsturng.test_all_to_tbl (test_qsturng.py "#remove from testsuite" #5084, test_qsturng.test_all_to_tbl skipped with "remove from testsuite" comment #5766) "#remove from testsuite"
• test_tools.test_arraylike2d (Commented-out tests in test_tools --> how to make useful? #4862, "comment out until we have type coercion" #4324)
• test_tools.test_arraylike1d (Commented-out tests in test_tools --> how to make useful? #4862, "comment out until we have type coercion" #4324)
• test_tools.test_arraylike2d_drop (Commented-out tests in test_tools --> how to make useful? #4862, "comment out until we have type coercion" #4324)
• test_tools.test_arraylike1d_drop (Commented-out tests in test_tools --> how to make useful? #4862, "comment out until we have type coercion" #4324)
• test_tools.test_array2d (Commented-out tests in test_tools --> how to make useful? #4862, "comment out until we have type coercion" #4324)
• test_tools.test_array1d (Commented-out tests in test_tools --> how to make useful? #4862, "comment out until we have type coercion" #4324)
• test_tools.test_array2d_drop (Commented-out tests in test_tools --> how to make useful? #4862, "comment out until we have type coercion" #4324)
• test_generic_methods.T_estWaldAnovaOLSNoFormula (should TestWaldAnoveOLSNoFormula subclass something? #4343)
• test_gmm est_other --> test_other
• test_gmm test_other commented out assertion
• test_gmm test_basic commented-out assertion
• test_gmm test_bse_other commented-out assertions
• test_gmm test_hypothesis commented-out assertions
• test_gmm test_iv2sls_r commented-out assertion
• test_tools tests named test_13, test_41, ... these presumably refer to GH issues, need meaningful names
• test_rootfinding.test_brentq_expanding commented-out assertions
• test_numdiff test_hess commented-out assertions.
• test_data test_patsy_577 should have meaningful name
• test_qsturng test_more_exotic_stuffcommented out, needs better name
• test_weightstats.TestZTest.test split into well-scoped tests, possibly parametrize
• test_weightstats CheckWeightstats1dMixin commented-out def with commented-out assertion
• test_weightstats.CheckWeightstats1dMixin.test_basic commented-out assertion
• test_weightstats test_weightstats_1-3 commented-out assertions/prints

Unmarked Smoke Tests

• test_penalized.test_smoke
• test_robustcov.test_smoke ([MAINT/TST] skip/xfail tests instead of mangling/commenting-out in genmod, regression #4980)

[jbrockmendel]

Unused/Unmaintained __main__ Sections

Moved from #5128

#3515 (comment)

Searching 1493 files for "__main__" (regex, case sensitive)

• setup.py
• statsmodels/base/tests/test_data.py
• statsmodels/base/tests/test_shrink_pickle.py
• statsmodels/base/wrapper.py
• statsmodels/compat/tests/test_itercompat.py
• statsmodels/discrete/tests/test_discrete.py
• statsmodels/discrete/tests/test_sandwich_cov.py
• statsmodels/distributions/edgeworth.py
• statsmodels/distributions/empirical_distribution.py
• statsmodels/distributions/mixture_rvs.py
• statsmodels/distributions/tests/test_discrete.py
• statsmodels/distributions/tests/test_edgeworth.py
• statsmodels/distributions/tests/test_mixture.py
• statsmodels/duration/tests/test_phreg.py
• statsmodels/examples/ex_kernel_regression2.py
• statsmodels/examples/ex_kernel_regression3.py
• statsmodels/examples/ex_kernel_regression_censored2.py
• statsmodels/examples/ex_kernel_regression_dgp.py
• statsmodels/examples/ex_kernel_regression_sigtest.py
• statsmodels/examples/ex_kernel_semilinear_dgp.py
• statsmodels/examples/ex_kernel_singleindex_dgp.py
• statsmodels/examples/ex_kernel_test_functional.py
• statsmodels/examples/ex_kernel_test_functional_li_wang.py
• statsmodels/examples/ex_multivar_kde.py
• statsmodels/examples/ex_outliers_influence.py
• statsmodels/examples/ex_pairwise.py
• statsmodels/examples/ex_rootfinding.py
• statsmodels/examples/l1_demo/demo.py
• statsmodels/examples/try_fit_constrained.py
• statsmodels/examples/try_power2.py
• statsmodels/genmod/_tweedie_compound_poisson.py
• statsmodels/genmod/families/tests/test_link.py
• statsmodels/genmod/generalized_linear_model.py
• statsmodels/genmod/tests/gee_categorical_simulation_check.py
• statsmodels/genmod/tests/gee_gaussian_simulation_check.py
• statsmodels/genmod/tests/gee_poisson_simulation_check.py
• statsmodels/genmod/tests/results/gee_generate_tests.py
• statsmodels/genmod/tests/test_glm.py
• statsmodels/graphics/tests/test_mosaicplot.py
• statsmodels/graphics/tests/test_regressionplots.py
• statsmodels/imputation/tests/test_mice.py
• statsmodels/iolib/foreign.py
• statsmodels/iolib/summary.py
• statsmodels/iolib/tests/test_foreign.py
• statsmodels/iolib/tests/test_summary.py
• statsmodels/miscmodels/nonlinls.py
• statsmodels/miscmodels/try_mlecov.py
• statsmodels/nonparametric/kde.py
• statsmodels/nonparametric/tests/test_bandwidths.py
• statsmodels/nonparametric/tests/test_kde.py
• statsmodels/nonparametric/tests/test_kernel_density.py
• statsmodels/nonparametric/tests/test_kernel_regression.py
• statsmodels/nonparametric/tests/test_kernels.py
• statsmodels/nonparametric/tests/test_lowess.py
• statsmodels/regression/linear_model.py
• statsmodels/regression/tests/test_glsar_gretl.py
• statsmodels/regression/tests/test_glsar_stata.py
• statsmodels/robust/robust_linear_model.py
• statsmodels/stats/_adnorm.py
• statsmodels/stats/anova.py
• statsmodels/stats/descriptivestats.py
• statsmodels/stats/libqsturng/tests/test_qsturng.py
• statsmodels/stats/tabledist.py
• statsmodels/stats/tests/test_anova.py
• statsmodels/stats/tests/test_diagnostic.py
• statsmodels/stats/tests/test_groups_sw.py
• statsmodels/stats/tests/test_moment_helpers.py
• statsmodels/stats/tests/test_power.py
• statsmodels/stats/tests/test_proportion.py
• statsmodels/stats/tests/test_sandwich.py
• statsmodels/stats/tests/test_statstools.py
• statsmodels/stats/tests/test_tost.py
• statsmodels/tools/catadd.py
• statsmodels/tools/dump2module.py
• statsmodels/tools/grouputils.py
• statsmodels/tools/numdiff.py
• statsmodels/tools/print_version.py
• statsmodels/tools/tests/test_catadd.py
• statsmodels/tools/tests/test_eval_measures.py
• statsmodels/tools/tests/test_numdiff.py
• statsmodels/tsa/ar_model.py
• statsmodels/tsa/arima_model.py
• statsmodels/tsa/base/tsa_model.py
• statsmodels/tsa/coint_tables.py
• statsmodels/tsa/filters/cf_filter.py
• statsmodels/tsa/filters/tests/test_filters.py
• statsmodels/tsa/interp/denton.py
• statsmodels/tsa/interp/tests/test_denton.py
• statsmodels/tsa/kalmanf/kalmanfilter.py
• statsmodels/tsa/seasonal.py
• statsmodels/tsa/tests/test_tsa_tools.py
• statsmodels/tsa/varma_process.py
• statsmodels/tsa/vector_ar/dynamic.py
• statsmodels/tsa/vector_ar/var_model.py

Some of these are in test files and are harmless-albeit-annoying.

[josef-pkt]

one comment to a group of __main__ issues as in statsmodels/examples/ex_kernel_regression_dgp.py

Those are/were required (at least on Windows) for multiprocessing.
I don't remember the details nor whether it would be really necessary in each case or whether the same pattern was used "just in case" if there is multiprocessing.