GLM raises ArrayIndexOutOfBoundsException with CV+lambda_search+alphas+categorical columns

[exalate-issue-sync]

The exception is pretty rare but reproducible with the attached test file.

{noformat}INFO water.default: Completing model GLM_model_python_1600368882855_5861_cv_5
WARN water.default: Model training job GLM completed with exception: java.lang.ArrayIndexOutOfBoundsException: 14
ERROR water.default:
java.lang.ArrayIndexOutOfBoundsException: 14
at water.util.ArrayUtils.subtract(ArrayUtils.java:1459)
at hex.glm.GLM$GLMDriver.fitIRLSM(GLM.java:1462)
at hex.glm.GLM$GLMDriver.fitModel(GLM.java:1726)
at hex.glm.GLM$GLMDriver.computeSubmodel(GLM.java:2095)
at hex.glm.GLM$GLMDriver.computeImpl(GLM.java:2202)
at hex.ModelBuilder$Driver.compute2(ModelBuilder.java:238)
at hex.glm.GLM$GLMDriver.compute2(GLM.java:890)
at water.H2O$H2OCountedCompleter.compute(H2O.java:1563)
at jsr166y.CountedCompleter.exec(CountedCompleter.java:468)
at jsr166y.ForkJoinTask.doExec(ForkJoinTask.java:263)
at jsr166y.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:974)
at jsr166y.ForkJoinPool.runWorker(ForkJoinPool.java:1477)
at jsr166y.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:104)
{noformat}

[exalate-issue-sync]

Wendy commented: First I would like to thank [~accountid:5b153fb1b0d76456f36daced]. His tests are what enabled me to locate the issues. I have added his tests to the PR.

[exalate-issue-sync]

Wendy commented: Here is my diagnostics of the problem:

{{During normal GLM model building, the coefficient length can shrink when coefficients/gram matrix has zero rows/columns. Since betaCnd is allocated at the beginning of iteration loop and the coefficient length change happened within the iteration loop, there can be a discrepancy in the coefficient lengths. Normally, this is not a problem because the action of betaCnd = ADMM_solve() or other solvers. But, in this case, that call is skipped. Hence, you will get betaCnd of one length and _state.beta() of another length. My fix is to make sure when there is a length difference, I will extract the correct coefficients from betaCnd such that it will be of the same length as _state.beta().}}

[h2o-ops]

JIRA Issue Migration Info

Jira Issue: PUBDEV-7782
Assignee: Wendy
Reporter: Sebastien Poirier
State: Resolved
Fix Version: 3.32.1.1
Attachments: Available (Count: 1)
Development PRs: Available

Linked PRs from JIRA

#5051

Attachments From Jira

Attachment Name: pyunit_tmp.py
Attached By: Sebastien Poirier
File Link:https://h2o-3-jira-github-migration.s3.amazonaws.com/PUBDEV-7782/pyunit_tmp.py