Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

TypeError when sorting candidates during Hyperband search #33

Open
floscha opened this issue Jul 10, 2019 · 6 comments

Comments

@floscha
Copy link
Contributor

commented Jul 10, 2019

The following simple toy example fails with the error message shown below. When using the RandomSearch tuner instead, everything works as expected.

def build_model(hp):
    inputs = layers.Input(shape=(5, ))
    x = layers.Dense(units=hp.Range('units', min_value=32, max_value=512, step=32),
                         activation='relu')(inputs)
    predictions = layers.Dense(1)(x)
    model = keras.models.Model(inputs=inputs, outputs=predictions)
    model.compile(optimizer='adam', loss='mean_squared_error')
    return model

tuner = kerastuner.tuners.Hyperband(
    build_model,
    objective='val_loss',
    max_trials=2,
    executions_per_trial=1
)

tuner.search(np.eye(5), np.ones((5, 1)),
             validation_data=(np.eye(5), np.ones((5, 1))),
             epochs=2)
1/2 trials left
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-3-c7537b51ff32> in <module>
     19 tuner.search(np.eye(5), np.ones((5, 1)),
     20              validation_data=(np.eye(5), np.ones((5, 1))),
---> 21              epochs=2)

/opt/conda/lib/python3.6/site-packages/kerastuner/engine/tuner.py in search(self, *fit_args, **fit_kwargs)
    207             # Obtain unique trial ID to communicate with the oracle.
    208             trial_id = tuner_utils.generate_trial_id()
--> 209             hp = self._call_oracle(trial_id)
    210             if hp is None:
    211                 # Oracle triggered exit

/opt/conda/lib/python3.6/site-packages/kerastuner/engine/tuner.py in _call_oracle(self, trial_id)
    525         # Obtain hp value suggestions from the oracle.
    526         while 1:
--> 527             oracle_answer = self.oracle.populate_space(trial_id, hp.space)
    528             if oracle_answer['status'] == 'RUN':
    529                 hp.values = oracle_answer['values']

/opt/conda/lib/python3.6/site-packages/kerastuner/tuners/hyperband.py in populate_space(self, trial_id, space)
     86         if self._bracket_index + 1 < self._num_brackets:
     87             self._bracket_index += 1
---> 88             self._select_candidates()
     89         # If the current band ends
     90         else:

/opt/conda/lib/python3.6/site-packages/kerastuner/tuners/hyperband.py in _select_candidates(self)
    135     def _select_candidates(self):
    136         sorted_candidates = sorted(list(range(len(self._candidates))),
--> 137                                    key=lambda i: self._candidate_score[i])
    138         num_selected_candidates = self._model_sequence[self._bracket_index]
    139         for index in sorted_candidates[:num_selected_candidates]:

TypeError: '<' not supported between instances of 'NoneType' and 'float'

The code was run with Python 3.6.6 and the following relevant libraries:

tensorflow                         2.0.0b1      
Keras-Tuner                        0.9.0.1562790722 
numpy                              1.16.4 
@SivamPillai

This comment has been minimized.

Copy link

commented Jul 25, 2019

Also facing the same issue using a different dataset. @floscha Were you able to find a solution to this problem as yet? Random Search works fine also for me.

@floscha

This comment has been minimized.

Copy link
Contributor Author

commented Jul 25, 2019

Hey @SivamPillai. Unfortunately, I couldn't find the time yet to look deep enough into the code to see where the error lies. But since I'm apparently not the only one with this problem, I'll see what I can do within the next couple of days, unless of course, @jhfjhfj1 does so as he's certainly more familiar with the codebase.

@jaredcolerosenberg

This comment has been minimized.

Copy link

commented Jul 25, 2019

I don't know the exact reason why None is appended to self._candidate_score (I tried to trace through the code but I am still learning exactly how Hyperband works), but that is the root of the problem. If you instead append a large number (largest possible score b/c they are sorted later on), then it seems to work.


  def _generate_candidates(self):
        self._candidates = []
        self._candidate_score = []
        num_models = self._model_sequence[0]

        for index in range(num_models):
            instance = self._new_trial()
            if instance is not None:
                self._candidates.append(instance)
                self._candidate_score.append(None)

        for index, instance in enumerate(self._candidates):
            self._queue.put(index)
@mikulskibartosz

This comment has been minimized.

Copy link

commented Jul 29, 2019

How I found a possible fix

I got it working, by using some monkey-patching in the runtime. I had to redefine the _select_candidates function inside a HyperbandOracle instance used by the tuner.

from functools import total_ordering
import types

@total_ordering
class MaxType(object):
    def __le__(self, other):
        return False

    def __eq__(self, other):
        return (self is other)

Max = MaxType()

def _select_candidates(self):
        sorted_candidates = sorted(list(range(len(self._candidates))),
                                   key=lambda i: Max if self._candidate_score[i] is None else self._candidate_score[i])
        num_selected_candidates = self._model_sequence[self._bracket_index]
        for index in sorted_candidates[:num_selected_candidates]:
            self._queue.put(index)

tuner.oracle._select_candidates = types.MethodType(_select_candidates, tuner.oracle)

The suggested change

Right now, I think we can get it fixed by:

  1. Defining the MaxType used for comparison in case of a None value
  2. Replacing the key lambda in _select_candidates function with: lambda i: Max if self._candidate_score[i] is None else self._candidate_score[i]

What do you think?

@samuelBedard

This comment has been minimized.

Copy link

commented Aug 2, 2019

facing the same issue

@SivamPillai

This comment has been minimized.

Copy link

commented Aug 3, 2019

How I found a possible fix

I got it working, by using some monkey-patching in the runtime. I had to redefine the _select_candidates function inside a HyperbandOracle instance used by the tuner.

from functools import total_ordering
import types

@total_ordering
class MaxType(object):
    def __le__(self, other):
        return False

    def __eq__(self, other):
        return (self is other)

Max = MaxType()

def _select_candidates(self):
        sorted_candidates = sorted(list(range(len(self._candidates))),
                                   key=lambda i: Max if self._candidate_score[i] is None else self._candidate_score[i])
        num_selected_candidates = self._model_sequence[self._bracket_index]
        for index in sorted_candidates[:num_selected_candidates]:
            self._queue.put(index)

tuner.oracle._select_candidates = types.MethodType(_select_candidates, tuner.oracle)

The suggested change

Right now, I think we can get it fixed by:

  1. Defining the MaxType used for comparison in case of a None value
  2. Replacing the key lambda in _select_candidates function with: lambda i: Max if self._candidate_score[i] is None else self._candidate_score[i]

What do you think?

This fixed it for me. Thanks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
5 participants
You can’t perform that action at this time.