Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Any Advice on Avoiding 'NaN' errors #189

Open
dfossl opened this issue Jul 22, 2022 · 9 comments
Open

Any Advice on Avoiding 'NaN' errors #189

dfossl opened this issue Jul 22, 2022 · 9 comments
Assignees
@mandjevant
Labels
bug

Comments

@dfossl
Copy link

dfossl commented Jul 22, 2022

Sometimes when I run this:

test_model = HyperoptEstimator(regressor=gradient_boosting_regressor('test_regres'),
                            preprocessing=[],
                            algo=tpe.suggest,
                            max_evals=50,
                            trial_timeout=30)
test_model.fit(X_train.to_numpy(), y_train.to_numpy().ravel())

I get an error ValueError: Input contains NaN. during training. It doesn't happen every time and I know that the data has no nan's, infinites, or duplicates. This leads me to believe one of the operations is creating a NaN. Is there anyway to skip these operations or deduce what operation is causing this?
@mandjevant
Copy link
Contributor

Hey there, could you provide me with a sample of your training data?
I am unable to reproduce this issue.

@mandjevant mandjevant self-assigned this Jul 27, 2022
@mandjevant mandjevant added the bug label Jul 27, 2022
@RaistlinTAO
Copy link

RaistlinTAO commented Dec 19, 2022 •
edited

I just got same error during HyperoptEstimator search, hope the following log helps

2|hpsklearnBlack  | job exception: Input contains NaN.
 98%|█████████▊| 48/49 [00:02<?, ?trial/s, best loss=?]
2|hpsklearnBlack  | Traceback (most recent call last):
2|hpsklearnBlack  |   File "/home/ubuntu/python/painter/Test/playground_gbm.py", line 48, in <module>
2|hpsklearnBlack  |     find_best_model(X_train, y_train, X_test, y_test)
2|hpsklearnBlack  |   File "/home/ubuntu/python/painter/Test/playground_gbm.py", line 29, in find_best_model
2|hpsklearnBlack  |     estimator.fit(x, y)
2|hpsklearnBlack  |   File "/home/ubuntu/python/painter/venv/lib/python3.10/site-packages/hpsklearn/estimator/estimator.py", line 464, in fit
2|hpsklearnBlack  |     fit_iter.send(increment)
2|hpsklearnBlack  |   File "/home/ubuntu/python/painter/venv/lib/python3.10/site-packages/hpsklearn/estimator/estimator.py", line 339, in fit_iter
2|hpsklearnBlack  |     hyperopt.fmin(_fn_with_timeout,
2|hpsklearnBlack  |   File "/home/ubuntu/python/painter/venv/lib/python3.10/site-packages/hyperopt/fmin.py", line 540, in fmin
2|hpsklearnBlack  |     return trials.fmin(
2|hpsklearnBlack  |   File "/home/ubuntu/python/painter/venv/lib/python3.10/site-packages/hyperopt/base.py", line 671, in fmin
2|hpsklearnBlack  |     return fmin(
2|hpsklearnBlack  |   File "/home/ubuntu/python/painter/venv/lib/python3.10/site-packages/hyperopt/fmin.py", line 586, in fmin
2|hpsklearnBlack  |     rval.exhaust()
2|hpsklearnBlack  |   File "/home/ubuntu/python/painter/venv/lib/python3.10/site-packages/hyperopt/fmin.py", line 364, in exhaust
2|hpsklearnBlack  |     self.run(self.max_evals - n_done, block_until_done=self.asynchronous)
2|hpsklearnBlack  |   File "/home/ubuntu/python/painter/venv/lib/python3.10/site-packages/hyperopt/fmin.py", line 300, in run
2|hpsklearnBlack  |     self.serial_evaluate()
2|hpsklearnBlack  |   File "/home/ubuntu/python/painter/venv/lib/python3.10/site-packages/hyperopt/fmin.py", line 178, in serial_evaluate
2|hpsklearnBlack  |     result = self.domain.evaluate(spec, ctrl)
2|hpsklearnBlack  |   File "/home/ubuntu/python/painter/venv/lib/python3.10/site-packages/hyperopt/base.py", line 892, in evaluate
2|hpsklearnBlack  |     rval = self.fn(pyll_rval)
2|hpsklearnBlack  |   File "/home/ubuntu/python/painter/venv/lib/python3.10/site-packages/hpsklearn/estimator/estimator.py", line 311, in _fn_with_timeout
2|hpsklearnBlack  |     raise fn_rval[1]
2|hpsklearnBlack  | ValueError: Input contains NaN.

According the result from df.isnull().sum() there is no NaN in data, hence, the NaH error occurs during parameter injection.

@mandjevant
Copy link
Contributor

@RaistlinTAO Could I get a snippet of your code please?
I understand that your data does not contain nan values. But I need to see what code exactly is causing the issue.

@RaistlinTAO
Copy link

@RaistlinTAO Could I get a snippet of your code please? I understand that your data does not contain nan values. But I need to see what code exactly is causing the issue.

Of course, happy to help

def find_best_model(x, y, test_x, test_y):
    estimator = HyperoptEstimator(
        regressor=gradient_boosting_regressor("T"),
        algo=tpe.suggest,
        max_evals=800,
        trial_timeout=300)
    estimator.fit(x, y)
    print('HyperoptEstimator Score: ')
    print(estimator.score(test_x, test_y))
    print('Best Model: ')
    print(estimator.best_model())

find_best_model(X_train, y_train, X_test, y_test)

@RaistlinTAO
Copy link

RaistlinTAO commented Dec 20, 2022 •
edited

Update:

With same code and same settings, it will bypass the error after give it another 3-5 tries.

I think it related to hyperparameter combination. sometimes it just skip the wrong combination or preprocessing

@mandjevant
Copy link
Contributor

Thanks a lot @RaistlinTAO that is what I was thinking as well. I've noticed this behaviour before.
Unfortunately, I cannot reproduce the error in our testing environment so it is a difficult issue to fix.
Your code snippet will help me to point down the problem and focus my testing.

I will work on a fix for this. But for the time being, retrying a few times should bypass the error.

@RaistlinTAO
Copy link

RaistlinTAO commented Dec 20, 2022 •
edited

After several attempts on debug in local env I have located the root cause.

In sklearn\ensemble_gb_losses.py
update_terminal_regions() statement:

    def _update_terminal_region(...):
        ...
        diff_minus_median = diff - median
        ...

# and 

def update_terminal_regions(...):
        raw_predictions[:, k] += learning_rate * tree.predict(X).ravel()

SAMPLE OUTPUT

E:\Projects\hyperopt-sklearn\venv\lib\site-packages\sklearn\ensemble\_gb_losses.py:231: RuntimeWarning: overflow encountered in square
  * np.sum(sample_weight * ((y - raw_predictions.ravel()) ** 2))
E:\Projects\hyperopt-sklearn\venv\lib\site-packages\sklearn\ensemble\_gb_losses.py:231: RuntimeWarning: overflow encountered in square
  * np.sum(sample_weight * ((y - raw_predictions.ravel()) ** 2))
E:\Projects\hyperopt-sklearn\venv\lib\site-packages\sklearn\ensemble\_gb_losses.py:288: RuntimeWarning: overflow encountered in multiply
  raw_predictions[:, k] += learning_rate * tree.predict(X).ravel()
E:\Projects\hyperopt-sklearn\venv\lib\site-packages\sklearn\ensemble\_gb_losses.py:288: RuntimeWarning: invalid value encountered in add
  raw_predictions[:, k] += learning_rate * tree.predict(X).ravel()
 97%|█████████▋| 32/33 [00:06<?, ?trial/s, best loss=?]
job exception: Input contains NaN.

There are two different outputs indicate that diff/median/learning_rate is NaN under certain circumstances.
I ll just leave this here since my hands are tied atm.

@crieux
Copy link

crieux commented Jul 20, 2023 •
edited

Hello,
I get the same issue, is there a fix for this one ?

 self.estim = HyperoptEstimator(
            regressor = any_regressor("my_clf"),
            preprocessing = [],
            algo = tpe.suggest,
            max_evals = 65,
            trial_timeout = 120
            )

Stack trace :

100%|██████████| 1/1 [00:00<00:00,  5.23trial/s, best loss: 0.05266970464190446]
100%|██████████| 2/2 [00:00<00:00,  2.18trial/s, best loss: 0.033195277678000346]
100%|██████████| 3/3 [00:00<00:00,  3.74trial/s, best loss: 0.033195277678000346]
100%|██████████| 4/4 [00:02<00:00,  2.30s/trial, best loss: 0.03245292544638456]
100%|██████████| 5/5 [00:00<00:00, 16.75trial/s, best loss: 0.03245292544638456]
100%|██████████| 6/6 [00:00<00:00, 13.88trial/s, best loss: 0.03245292544638456]
100%|██████████| 7/7 [00:00<00:00,  1.38trial/s, best loss: 0.03245292544638456]
100%|██████████| 8/8 [00:01<00:00,  1.57s/trial, best loss: 0.03245292544638456]
100%|██████████| 9/9 [00:00<00:00, 13.97trial/s, best loss: 0.03245292544638456]
100%|██████████| 10/10 [00:00<00:00, 16.76trial/s, best loss: 0.03245292544638456]
100%|██████████| 11/11 [00:00<00:00,  7.96trial/s, best loss: 0.03245292544638456]
100%|██████████| 12/12 [00:00<00:00,  1.48trial/s, best loss: 0.03245292544638456]
100%|██████████| 13/13 [00:00<00:00,  2.03trial/s, best loss: 0.030216599846460745]
100%|██████████| 14/14 [00:00<00:00, 16.23trial/s, best loss: 0.030216599846460745]
100%|██████████| 15/15 [00:00<00:00,  2.10trial/s, best loss: 0.030216599846460745]
100%|██████████| 16/16 [00:00<00:00, 17.83trial/s, best loss: 0.030216599846460745]
100%|██████████| 17/17 [00:00<00:00,  1.30trial/s, best loss: 0.030216599846460745]
100%|██████████| 18/18 [00:00<00:00, 15.57trial/s, best loss: 0.030216599846460745]
100%|██████████| 19/19 [00:00<00:00,  1.66trial/s, best loss: 0.030216599846460745]
100%|██████████| 20/20 [00:00<00:00, 17.59trial/s, best loss: 0.030216599846460745]
100%|██████████| 21/21 [00:00<00:00, 10.58trial/s, best loss: 0.030216599846460745]
100%|██████████| 22/22 [00:00<00:00,  7.55trial/s, best loss: 0.030216599846460745]
100%|██████████| 23/23 [00:02<00:00,  2.01s/trial, best loss: 0.030216599846460745]
100%|██████████| 24/24 [02:00<00:00, 120.18s/trial, best loss: 0.030216599846460745]
100%|██████████| 25/25 [00:02<00:00,  2.64s/trial, best loss: 0.030216599846460745]
100%|██████████| 26/26 [00:00<00:00,  7.15trial/s, best loss: 0.030216599846460745]
100%|██████████| 27/27 [00:01<00:00,  1.15s/trial, best loss: 0.030216599846460745]
100%|██████████| 28/28 [00:00<00:00,  1.77trial/s, best loss: 0.030216599846460745]
100%|██████████| 29/29 [00:00<00:00, 10.25trial/s, best loss: 0.030216599846460745]
100%|██████████| 30/30 [00:09<00:00,  9.18s/trial, best loss: 0.030216599846460745]
100%|██████████| 31/31 [00:00<00:00, 10.90trial/s, best loss: 0.030216599846460745]
100%|██████████| 32/32 [00:01<00:00,  1.49s/trial, best loss: 0.027541941347137833]
 97%|█████████▋| 32/33 [00:00<?, ?trial/s, best loss=?]
ValueError: Input contains NaN, infinity or a value too large for dtype('float64').

@RaistlinTAO
Copy link

Hello, I get the same issue, is there a fix for this one ?

 self.estim = HyperoptEstimator(
            regressor = any_regressor("my_clf"),
            preprocessing = [],
            algo = tpe.suggest,
            max_evals = 65,
            trial_timeout = 120
            )

Stack trace :

100%|██████████| 1/1 [00:00<00:00,  5.23trial/s, best loss: 0.05266970464190446]
100%|██████████| 2/2 [00:00<00:00,  2.18trial/s, best loss: 0.033195277678000346]
100%|██████████| 3/3 [00:00<00:00,  3.74trial/s, best loss: 0.033195277678000346]
100%|██████████| 4/4 [00:02<00:00,  2.30s/trial, best loss: 0.03245292544638456]
100%|██████████| 5/5 [00:00<00:00, 16.75trial/s, best loss: 0.03245292544638456]
100%|██████████| 6/6 [00:00<00:00, 13.88trial/s, best loss: 0.03245292544638456]
100%|██████████| 7/7 [00:00<00:00,  1.38trial/s, best loss: 0.03245292544638456]
100%|██████████| 8/8 [00:01<00:00,  1.57s/trial, best loss: 0.03245292544638456]
100%|██████████| 9/9 [00:00<00:00, 13.97trial/s, best loss: 0.03245292544638456]
100%|██████████| 10/10 [00:00<00:00, 16.76trial/s, best loss: 0.03245292544638456]
100%|██████████| 11/11 [00:00<00:00,  7.96trial/s, best loss: 0.03245292544638456]
100%|██████████| 12/12 [00:00<00:00,  1.48trial/s, best loss: 0.03245292544638456]
100%|██████████| 13/13 [00:00<00:00,  2.03trial/s, best loss: 0.030216599846460745]
100%|██████████| 14/14 [00:00<00:00, 16.23trial/s, best loss: 0.030216599846460745]
100%|██████████| 15/15 [00:00<00:00,  2.10trial/s, best loss: 0.030216599846460745]
100%|██████████| 16/16 [00:00<00:00, 17.83trial/s, best loss: 0.030216599846460745]
100%|██████████| 17/17 [00:00<00:00,  1.30trial/s, best loss: 0.030216599846460745]
100%|██████████| 18/18 [00:00<00:00, 15.57trial/s, best loss: 0.030216599846460745]
100%|██████████| 19/19 [00:00<00:00,  1.66trial/s, best loss: 0.030216599846460745]
100%|██████████| 20/20 [00:00<00:00, 17.59trial/s, best loss: 0.030216599846460745]
100%|██████████| 21/21 [00:00<00:00, 10.58trial/s, best loss: 0.030216599846460745]
100%|██████████| 22/22 [00:00<00:00,  7.55trial/s, best loss: 0.030216599846460745]
100%|██████████| 23/23 [00:02<00:00,  2.01s/trial, best loss: 0.030216599846460745]
100%|██████████| 24/24 [02:00<00:00, 120.18s/trial, best loss: 0.030216599846460745]
100%|██████████| 25/25 [00:02<00:00,  2.64s/trial, best loss: 0.030216599846460745]
100%|██████████| 26/26 [00:00<00:00,  7.15trial/s, best loss: 0.030216599846460745]
100%|██████████| 27/27 [00:01<00:00,  1.15s/trial, best loss: 0.030216599846460745]
100%|██████████| 28/28 [00:00<00:00,  1.77trial/s, best loss: 0.030216599846460745]
100%|██████████| 29/29 [00:00<00:00, 10.25trial/s, best loss: 0.030216599846460745]
100%|██████████| 30/30 [00:09<00:00,  9.18s/trial, best loss: 0.030216599846460745]
100%|██████████| 31/31 [00:00<00:00, 10.90trial/s, best loss: 0.030216599846460745]
100%|██████████| 32/32 [00:01<00:00,  1.49s/trial, best loss: 0.027541941347137833]
 97%|█████████▋| 32/33 [00:00<?, ?trial/s, best loss=?]
ValueError: Input contains NaN, infinity or a value too large for dtype('float64').

Well my suggestions are:

  1. Use Optuna (https://github.com/optuna/optuna),
    however they have different issues, but it's rare.
  2. Use GridSearchCV, but considering the time/resource consumption.

Pick any solution that fit your needs and have a good one

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@mandjevant mandjevant

Labels
bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@crieux @RaistlinTAO @mandjevant @dfossl