Predict errors when using Mojo loaded models into H2O

[exalate-issue-sync]

We are running into an issue that we are hoping to get your help with. On AWS we use a single instance to train GLM, GBM and Random Forest models. Right after training we run prediction on the actively trained model without issues. After the training we save off the model using save_model() and save_mojo(). We use the saved models to predict on hold out datasets which is done by loading the models.

The issue we are having is when a model trained with an offset is loaded for prediction via import_mojo() followed by a predict we get the error below. The same trained model loaded with h2o.load_model predicts successfully without issues. So two questions to start:

1.  Thoughts on why import_mojo() doesn’t work but load_model does?
   

2.  The model loaded with import_mojo creates a H2OGenericEstimator while the h2o.load_model creates a H2OGradientBoostingEstimator or a H2ORandomForestEstimator estimator. Is there a way to typecast the GenericEstimator to the specific predictor type?
   

Thanks!
Kelsey

predictions = model.predict(data)

File "/opt/.venv/lib/python3.9/site-packages/h2o/model/model_base.py", line 237, in predict
j.poll()
File "/opt/.venv/lib/python3.9/site-packages/h2o/job.py", line 79, in poll
raise EnvironmentError("Job with key {} failed with an exception: {}\nstacktrace: "
OSError: Job with key $03017f00000132d4ffffffff$_8c93744a88d3852db449ca6cacf94658 failed with an exception: DistributedException from /127.0.0.1:54321: 'Override this method for non-trivial offset!', caused by java.lang.AssertionError: Override this method for non-trivial offset!
stacktrace:
DistributedException from /127.0.0.1:54321: 'Override this method for non-trivial offset!', caused by java.lang.AssertionError: Override this method for non-trivial offset!
#011at water.MRTask.getResult(MRTask.java:654)
Closing connection _sid_a398 at exit
#011at water.MRTask.getResult(MRTask.java:664)
#011at water.MRTask.doAll(MRTask.java:524)
#011at water.MRTask.doAll(MRTask.java:543)
#011at hex.Model.predictScoreImpl(Model.java:1877)
#011at hex.Model.score(Model.java:1709)
#011at water.api.ModelMetricsHandler$1.compute2(ModelMetricsHandler.java:422)
#011at water.H2O$H2OCountedCompleter.compute(H2O.java:1637)
#011at jsr166y.CountedCompleter.exec(CountedCompleter.java:468)
#011at jsr166y.ForkJoinTask.doExec(ForkJoinTask.java:263)
#011at jsr166y.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:974)
#011at jsr166y.ForkJoinPool.runWorker(ForkJoinPool.java:1477)
#011at jsr166y.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:104)
Caused by: java.lang.AssertionError: Override this method for non-trivial offset!
#011at hex.Model.score0(Model.java:2116)
#011at hex.Model.score0(Model.java:2084)
#011at hex.Model$BigScore.score0(Model.java:2028)
#011at hex.Model$BigScore.map(Model.java:2006)
#011at water.MRTask.compute2(MRTask.java:837)
#011at water.H2O$H2OCountedCompleter.compute1(H2O.java:1640)
#011at hex.Model$BigScore$Icer.compute1(Model$BigScore$Icer.java)
#011at water.H2O$H2OCountedCompleter.compute(H2O.java:1636)
#11... 5 more

H2O session _sid_a398 closed.

[h2o-ops]

JIRA Issue Details

Jira Issue: PUBDEV-8234
Assignee: Michal Kurka
Reporter: Arun Aryasomayajula
State: Resolved
Fix Version: 3.32.1.7
Attachments: N/A
Development PRs: Available

[h2o-ops]

Linked PRs from JIRA

#5652