Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Long evaluation in _pre_test hangs search #1023

Open
josepablocam opened this issue Feb 21, 2020 · 1 comment
Open

Long evaluation in _pre_test hangs search #1023

josepablocam opened this issue Feb 21, 2020 · 1 comment

Comments

@josepablocam
Copy link

Thanks for the great tool. When using it, I ran into a bit of
an edge case. For some operators, the
reasonable _pre_test decorator (tpot/decorators.py)
can actually take an arbitrarily long amount of time, effectively
hanging the search (since it never moves past generating candidates).
The small sampled down pretest_X might be reasonable, but
for some operators complexity scales with columns (and it may be
hard to determine the correct number to sample, if any).

You can reproduce the issue by using the following example:

> cat example.py

import tpot
import numpy as np
import pmlb

X, y = pmlb.fetch_data("Hill_Valley_without_noise", return_X_y=True)

clf = tpot.TPOTClassifier(
  max_time_mins=1,
  max_eval_time_mins=1,
  cv=2,
  config_dict={
      'sklearn.decomposition.SparsePCA':{},
      'sklearn.linear_model.LogisticRegression':{},
  },
  verbosity=3,
)

clf.fit(X, y)

This will timeout, rather than finish appropriately

timeout 5m python example.py # will hang (and get killed 5 minutes after)
echo $? # 124 -> timeout

Possible fix

I have a simple patch that adds a timeout to the
.fit call taking place at line 88 in tpot/decorators.py. I can PR it if this seems useful.

@weixuanfu
Copy link
Contributor

Hmm, nice suggestion! Please submit a PR for this issue. Thank you!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants