Distributed Random Forest (DRF) fails to create SHAP summary plot

[exalate-issue-sync]

To repro:

Create DRF model:

{code:python}from h2o.estimators import H2ORandomForestEstimator

Import the cars dataset into H2O:

cars = h2o.import_file("https://s3.amazonaws.com/h2o-public-test-data/smalldata/junit/cars_20mpg.csv")

Set the predictors and response;

set the response as a factor:

cars["economy_20mpg"] = cars["economy_20mpg"].asfactor()
predictors = ["displacement","power","weight","acceleration","year"]
response = "economy_20mpg"

Split the dataset into a train and valid set:

train, valid = cars.split_frame(ratios=[.8], seed=1234)

Build and train the model:

cars_drf = H2ORandomForestEstimator(ntrees=10,
max_depth=5,
min_rows=10,
calibrate_model=True,
calibration_frame=valid,
binomial_double_trees=True)
cars_drf.train(x=predictors,
y=response,
training_frame=train,
validation_frame=valid){code}

.explain fails:

{code:python}cars_drf.explain(valid){code}

{noformat}OSError: Job with key $03017f00000132d4ffffffff$_a55bbe76e23a0251c456b85d5e574a9c failed with an exception: java.lang.AssertionError
stacktrace:
java.lang.AssertionError
at hex.tree.SharedTreeModelWithContributions$ScoreContributionsTask.setupLocal(SharedTreeModelWithContributions.java:88)
at water.MRTask.setupLocal0(MRTask.java:728)
at water.MRTask.dfork(MRTask.java:622)
at water.MRTask.doAll(MRTask.java:523)
at water.MRTask.doAll(MRTask.java:543)
at hex.tree.SharedTreeModelWithContributions.scoreContributions(SharedTreeModelWithContributions.java:52)
at hex.tree.SharedTreeModelWithContributions.scoreContributions(SharedTreeModelWithContributions.java:30)
at hex.Model$Contributions.scoreContributions(Model.java:134)
at water.api.ModelMetricsHandler$1.compute2(ModelMetricsHandler.java:420)
at water.H2O$H2OCountedCompleter.compute(H2O.java:1637)
at jsr166y.CountedCompleter.exec(CountedCompleter.java:468)
at jsr166y.ForkJoinTask.doExec(ForkJoinTask.java:263)
at jsr166y.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:974)
at jsr166y.ForkJoinPool.runWorker(ForkJoinPool.java:1477)
at jsr166y.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:104){noformat}

.shap_summary_plot() fails:

{code:python}cars_drf.shap_summary_plot(valid){code}

{noformat}---------------------------------------------------------------------------
OSError Traceback (most recent call last)
in
----> 1 cars_drf.shap_summary_plot(valid)

~/anaconda3/envs/py_36/lib/python3.6/site-packages/h2o/explanation/_explain.py in shap_summary_plot(model, frame, columns, top_n_features, samples, colorize_factors, alpha, colormap, figsize, jitter)
596
597 with no_progress():
--> 598 contributions = NumpyFrame(model.predict_contributions(frame))
599 frame = NumpyFrame(frame)
600 contribution_names = contributions.columns

~/anaconda3/envs/py_36/lib/python3.6/site-packages/h2o/model/model_base.py in predict_contributions(self, test_data, output_format)
198 data={"predict_contributions": True, "predict_contributions_output_format": output_format}),
199 "contributions")
--> 200 j.poll()
201 return h2o.get_frame(j.dest_key)
202

~/anaconda3/envs/py_36/lib/python3.6/site-packages/h2o/job.py in poll(self, poll_updates)
78 if (isinstance(self.job, dict)) and ("stacktrace" in list(self.job)):
79 raise EnvironmentError("Job with key {} failed with an exception: {}\nstacktrace: "
---> 80 "\n{}".format(self.job_key, self.exception, self.job["stacktrace"]))
81 else:
82 raise EnvironmentError("Job with key %s failed with an exception: %s" % (self.job_key, self.exception))

OSError: Job with key $03017f00000132d4ffffffff$_a15125cf3a4bf86e1e4c1c6c3fc94a94 failed with an exception: java.lang.AssertionError
stacktrace:
java.lang.AssertionError
at hex.tree.SharedTreeModelWithContributions$ScoreContributionsTask.setupLocal(SharedTreeModelWithContributions.java:88)
at water.MRTask.setupLocal0(MRTask.java:728)
at water.MRTask.dfork(MRTask.java:622)
at water.MRTask.doAll(MRTask.java:523)
at water.MRTask.doAll(MRTask.java:543)
at hex.tree.SharedTreeModelWithContributions.scoreContributions(SharedTreeModelWithContributions.java:52)
at hex.tree.SharedTreeModelWithContributions.scoreContributions(SharedTreeModelWithContributions.java:30)
at hex.Model$Contributions.scoreContributions(Model.java:134)
at water.api.ModelMetricsHandler$1.compute2(ModelMetricsHandler.java:420)
at water.H2O$H2OCountedCompleter.compute(H2O.java:1637)
at jsr166y.CountedCompleter.exec(CountedCompleter.java:468)
at jsr166y.ForkJoinTask.doExec(ForkJoinTask.java:263)
at jsr166y.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:974)
at jsr166y.ForkJoinPool.runWorker(ForkJoinPool.java:1477)
at jsr166y.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:104){noformat}

[exalate-issue-sync]

Arun Aryasomayajula commented: [~accountid:5f8e6929461cc40075215ee0] to take a look and assign

[exalate-issue-sync]

Adam Valenta commented: Hello [~accountid:5dc4f5bbb6e6b50c58af0624] IMHO drf with binomial_double_trees behave like multinomial models. Multinomial models are not supported.

Does it come from customer support ticket?

[exalate-issue-sync]

Neema Mashayekhi commented: Yes, [~accountid:5f8e6929461cc40075215ee0] , I added the customer support ticket [https://support.h2o.ai/a/tickets/99835|https://support.h2o.ai/a/tickets/99835]

[exalate-issue-sync]

Adam Valenta commented: Thanks, and what solutions do you require? Better error message, because it is actually not supported, or implement this feature? [~accountid:5dc4f5bbb6e6b50c58af0624]

[exalate-issue-sync]

Neema Mashayekhi commented: It would be better if we can implement this features so it’s consistent with the other models. It was also requested by the user to have this feature

[h2o-ops]

JIRA Issue Details

Jira Issue: PUBDEV-8220
Assignee: Adam Valenta
Reporter: Neema Mashayekhi
State: Resolved
Fix Version: 3.38.0.1
Attachments: N/A
Development PRs: Available

[h2o-ops]

Linked PRs from JIRA

#5561
#6327
https://github.com/h2oai/h2oai-serving/pull/1079