Number of layers

[BenLim88]

We can use the following for number of layers (assuming 2 dense layers have been already defined), according to the examples:

    if conditional({{choice(['two', 'three'])}}) == 'three':
        model.add(Dense({{choice([np.power(2,5),np.power(2,9),np.power(2,11)])}}))
        model.add(Activation({{choice(['tanh','relu', 'sigmoid'])}}))
        model.add(Dropout({{uniform(0, 1)}}))

Is there a way to add the number of layers by means of a number? e.g. 10 = 10 dense layers.

I tried this, but in the best_run, it gives me 15 dense layers, which does not reflect the actual results

 if conditional({{choice(['two', 'three','four','five','ten'])}}) == 'three':
        model.add(Dense({{choice([np.power(2,5),np.power(2,9),np.power(2,11)])}}))
        model.add(Activation({{choice(['tanh','relu', 'sigmoid'])}}))
        model.add(Dropout({{uniform(0, 1)}}))
    elif conditional({{choice(['two', 'three','four','five','ten'])}}) == 'four':
        model.add(Dense({{choice([np.power(2,5),np.power(2,9),np.power(2,11)])}}))
        model.add(Activation({{choice(['tanh','relu', 'sigmoid'])}}))
        model.add(Dropout({{uniform(0, 1)}}))
        model.add(Dense({{choice([np.power(2,5),np.power(2,9),np.power(2,11)])}}))
        model.add(Activation({{choice(['tanh','relu', 'sigmoid'])}}))
        model.add(Dropout({{uniform(0, 1)}}))
    elif conditional({{choice(['two', 'three','four','five','ten'])}}) == 'five':
        model.add(Dense({{choice([np.power(2,5),np.power(2,9),np.power(2,11)])}}))
        model.add(Activation({{choice(['tanh','relu', 'sigmoid'])}}))
        model.add(Dropout({{uniform(0, 1)}}))
        model.add(Dense({{choice([np.power(2,5),np.power(2,9),np.power(2,11)])}}))
        model.add(Activation({{choice(['tanh','relu', 'sigmoid'])}}))
        model.add(Dropout({{uniform(0, 1)}}))
        model.add(Dense({{choice([np.power(2,5),np.power(2,9),np.power(2,11)])}}))
        model.add(Activation({{choice(['tanh','relu', 'sigmoid'])}}))
        model.add(Dropout({{uniform(0, 1)}}))
    elif conditional({{choice(['two', 'three','four','five','ten'])}}) == 'ten':
        model.add(Dense({{choice([np.power(2,5),np.power(2,9),np.power(2,11)])}}))
        model.add(Activation({{choice(['tanh','relu', 'sigmoid'])}}))
        model.add(Dropout({{uniform(0, 1)}}))
        model.add(Dense({{choice([np.power(2,5),np.power(2,9),np.power(2,11)])}}))
        model.add(Activation({{choice(['tanh','relu', 'sigmoid'])}}))
        model.add(Dropout({{uniform(0, 1)}}))
        model.add(Dense({{choice([np.power(2,5),np.power(2,9),np.power(2,11)])}}))
        model.add(Activation({{choice(['tanh','relu', 'sigmoid'])}}))
        model.add(Dropout({{uniform(0, 1)}}))
        model.add(Dense({{choice([np.power(2,5),np.power(2,9),np.power(2,11)])}}))
        model.add(Activation({{choice(['tanh','relu', 'sigmoid'])}}))
        model.add(Dropout({{uniform(0, 1)}}))
        model.add(Dense({{choice([np.power(2,5),np.power(2,9),np.power(2,11)])}}))
        model.add(Activation({{choice(['tanh','relu', 'sigmoid'])}}))
        model.add(Dropout({{uniform(0, 1)}}))
        model.add(Dense({{choice([np.power(2,5),np.power(2,9),np.power(2,11)])}}))
        model.add(Activation({{choice(['tanh','relu', 'sigmoid'])}}))
        model.add(Dropout({{uniform(0, 1)}}))
        model.add(Dense({{choice([np.power(2,5),np.power(2,9),np.power(2,11)])}}))
        model.add(Activation({{choice(['tanh','relu', 'sigmoid'])}}))
        model.add(Dropout({{uniform(0, 1)}}))
        model.add(Dense({{choice([np.power(2,5),np.power(2,9),np.power(2,11)])}}))
        model.add(Activation({{choice(['tanh','relu', 'sigmoid'])}}))
        model.add(Dropout({{uniform(0, 1)}}))

[jeremywang123]

Hi BenLim88,

First of all, i usually use "if {{choice(['three', 'four'])}} as condition and you might want to double check it.
Except this, everything looks fine for me, could you post ur search space & result here?

[BenLim88]

Dear Jeremy

Thanks for the reply. How would you write the code if you are trying to iterate through, say, 2, 3 and 4 layers?

I assume you will start by adding 2 dense layers:
model.add(Dense...) model.add(Dense...)

Then run the condition, for the 3rd layer: if {{choice(['two', 'three'])}} == 'three'

For the 4th layer, i presume it has to be nested within the 'if' statement for the 3rd layer, but i'm unsure if it is correct.

[maxpumperla]

@BenLim88 you should only evaluate choice once and then check the results against actual values. you can also use a randint between bounds to determine the number of layers and loop to add layers:

model = ...
num_layers = <result of randint>
for _ in range(num_layers):
        model.add(Dense({{choice([np.power(2,5),np.power(2,9),np.power(2,11)])}}))
        model.add(Activation({{choice(['tanh','relu', 'sigmoid'])}}))
        model.add(Dropout({{uniform(0, 1)}}))

[BenLim88]

could i define the search range for number of layers as well? e.g. choice([10,20,30]) for 10, 20 and 30 layers?

[kenfus]

Does anyone know how Hyperas deals with choices in a "if" statement? I added a choice for the dropout for each layer. Hyperas can chose between three to seven layers, with different dropout and nodes for each layer. Does Hyperas try to tune for values behind an if-statement?

[loublock]

@BenLim88 you should only evaluate choice once and then check the results against actual values. you can also use a randint between bounds to determine the number of layers and loop to add layers:

model = ...
num_layers = <result of randint>
for _ in range(num_layers):
        model.add(Dense({{choice([np.power(2,5),np.power(2,9),np.power(2,11)])}}))
        model.add(Activation({{choice(['tanh','relu', 'sigmoid'])}}))
        model.add(Dropout({{uniform(0, 1)}}))

This solution works great, but when it comes to the space object, there are never stored more than two layers (input & first from for-loop) with their parameters. Any suggestions?