Check for duplicated columns before generating cpm matrix for MaxRSweep

[exalate-issue-sync]

No description provided.

[exalate-issue-sync]

Wendy Wong commented: With duplicated columns, I was able to regenerate the maxrsweep error:

Here is the code:

{noformat} train2 = h2o.import_file(pyunit_utils.locate("bigdata/laptop/model_selection/maxrglm200Cols50KRows.csv"))
train2.drop(["response"], axis=1)
train = h2o.import_file(pyunit_utils.locate("bigdata/laptop/model_selection/maxrglm200Cols50KRows.csv"))
train = train.cbind(train2)

train = train.cbind(train2)

response="response"
predictors = train.names
predictors.remove(response)
maxrsweep3_model = H2OModelSelectionEstimator(mode="maxrsweep", max_predictor_number=300, intercept=True, build_glm_model=False)
maxrsweep3_model.train(x=predictors, y=response, training_frame=train){noformat}

Here is the error message:

02-02 14:38:03.930 127.0.0.1:54321    29517   FJ-1-7 ERROR water.default:
java.lang.NullPointerException: null
at hex.modelselection.ModelSelectionUtils.oneSweepWSweepVector(ModelSelectionUtils.java:692) ~[h2o.jar:?]
at hex.modelselection.ModelSelectionUtils.applySweepVectors2NewPred(ModelSelectionUtils.java:665) ~[h2o.jar:?]
at hex.modelselection.ModelSelectionUtils$1.compute(ModelSelectionUtils.java:302) ~[h2o.jar:?]
at jsr166y.RecursiveAction.exec(RecursiveAction.java:160) ~[h2o.jar:?]
at jsr166y.ForkJoinTask.doExec(ForkJoinTask.java:263) ~[h2o.jar:?]
at jsr166y.ForkJoinTask.doInvoke(ForkJoinTask.java:360) ~[h2o.jar:?]
at jsr166y.ForkJoinTask.invokeAll(ForkJoinTask.java:741) [h2o.jar:?]
at hex.modelselection.ModelSelectionUtils.generateAllErrorVariances(ModelSelectionUtils.java:280) ~[h2o.jar:?]
at hex.modelselection.ModelSelection.forwardStepR(ModelSelection.java:604) ~[h2o.jar:?]
at hex.modelselection.ModelSelection.replacement(ModelSelection.java:675) ~[h2o.jar:?]
at hex.modelselection.ModelSelection$ModelSelectionDriver.buildMaxRSweepModels(ModelSelection.java:297) ~[h2o.jar:?]
at hex.modelselection.ModelSelection$ModelSelectionDriver.buildModel(ModelSelection.java:242) ~[h2o.jar:?]
at hex.modelselection.ModelSelection$ModelSelectionDriver.computeImpl(ModelSelection.java:461) ~[h2o.jar:?]
at hex.ModelBuilder$Driver.compute2(ModelBuilder.java:252) ~[h2o.jar:?]
at hex.modelselection.ModelSelection$ModelSelectionDriver.compute2(ModelSelection.java:203) ~[h2o.jar:?]
at water.H2O$H2OCountedCompleter.compute(H2O.java:1677) ~[h2o.jar:?]
at jsr166y.CountedCompleter.exec(CountedCompleter.java:468) ~[h2o.jar:?]
at jsr166y.ForkJoinTask.doExec(ForkJoinTask.java:263) [h2o.jar:?]
at jsr166y.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:976) [h2o.jar:?]
at jsr166y.ForkJoinPool.runWorker(ForkJoinPool.java:1479) [h2o.jar:?]
at jsr166y.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:104) [h2o.jar:?]
02-02 14:38:04.268 127.0.0.1:54321    29517 0232440-81 INFO water.default: DELETE /4/sessions/_sid_b2c3, parms: {}

[wendycwong]

Fixed.

[h2o-ops]

JIRA Issue Details

Jira Issue: PUBDEV-8952
Assignee: Wendy Wong
Reporter: Wendy Wong
State: Resolved
Fix Version: 3.40.0.2
Attachments: N/A
Development PRs: Available

[h2o-ops]

Linked PRs from JIRA

#6487