No getting repeatable results

[ivan-marroquin]

For some reason, when i combine multiple pairs of layers (see the attached script) the result i get is not the same when i run the same script with a single pair of layers. for convenience, in addition to the script. i also attached the input files.

many thanks for your help!
ivan

Script_Input_Files.zip

[ivan-marroquin]

i forgot to mention that i am using the latest version of neupy using anaconda in windows 7 64bit

also,

[itdxer]

if you use different layers for the network, results will be different, because you are trying to train to different networks

[ivan-marroquin]

thanks for the answer. however, what i reported is different.

if you look at the html files (first_run and second_run), and focus when i used softmax for the hidden layer, you will notice the following: the mse error and the output of network.architecture() are different from one run to another.

the difference between first_run with respect to second_run is simple. in the first_run, there is another network (with different activation function for the hidden layer) called before i used softmax. it seems to me that the issue is caused by a "somewhat" interference between a previous result and the current being computed.

the only way to get repeatable results is to evaluate a pair of activation functions one at the time.

do you have any suggestions?

thanks

[itdxer]

ok, now I see. The problem is that you are initializing layers and then re-using them. For instance,

from neupy import layers

hidden_layer = layers.Relu(5)
network1 = layers.join(
     layers.Input(1),
     hidden_layer,
     layers.Softmax(10),
)
network2 = layers.join(
     layers.Input(1),
     hidden_layer,
     layers.Softmax(10),
)

network1 and network2 re-use the same hidden layer, which means that if you train the first network (network1) the second one (network2) will have already pre-trained parameter in the hidden layer. Something similar happens in your case. If you want different layers on different runs than you have to initialize new layer every iteration.

But it's shows architecture incorrectly, so there is definitely some bug in the library. I'll try to fix it till the next release. thanks

[ivan-marroquin]

thanks for the clarification. i have this question: this issue could impact the results of hyper-parameter optimization? in your script example, the same neural network architecture is used. so, it's likely that a pre-trained parameter from a previous run will be used on the current run. am i correct? if so, is there a way to "clean" the neural network before proceeding to the next run? a such solution could force the network to be re-initialized at every single run.

thanks for all!

[itdxer]

In the example I construct networks dynamically and on every iteration I initialize a new layer, so they are going to produce different results. So I don't re-use it on every iteration.

Right now there is no simple way to re-initialize the network. You can specify a set of parameters in your list and to initialize layer during the iteration. Instead of passing layer and it's name you can pass layer class, set of parameters and name. After that you can initialize new layer on each loop.

I hope it will help

[itdxer]

@ivan-marroquin

I fixed problem with incorrectly displayed architecture. Thank you for reporting it. That was a tricky bug. If you need you can try to install it from the upcoming release:

$ pip install --upgrade git+https://github.com/itdxer/neupy.git@release/v0.5.2

[ivan-marroquin]

I tried to install the upcoming release using the command line, and got the following error:

(C:\Temp\Anaconda3) C:\Users\IMarroquin>pip install --upgrade git+https://github
.com/itdxer/neupy.git@release/v0.5.2
Collecting git+https://github.com/itdxer/neupy.git@release/v0.5.2
  Cloning https://github.com/itdxer/neupy.git (to release/v0.5.2) to c:\users\im
arro~1\appdata\local\temp\pip-g7s18exp-build
  Error [WinError 2] The system cannot find the file specified while executing c
ommand git clone -q https://github.com/itdxer/neupy.git C:\Users\IMARRO~1\AppDat
a\Local\Temp\pip-g7s18exp-build
Cannot find command 'git'

I tried from both: conda prompt window and dos window.

Am I missing something?

[itdxer]

Message says that you don't have git (or it's just not visible). You can try to add git like it has been suggested here: http://stackoverflow.com/a/29957544/2759088
Or you can do it manually

1. Download upcoming release (v0.5.2): https://github.com/itdxer/neupy/archive/release/v0.5.2.zip
2. Unzip it
3. Go to the folder from command line using cd command
4. Install library with command: python setup.py install

You can check neupy's version in order to make sure that everything is fine. import neupy; neupy.__version__. Currently it's 0.5.2.b1

[ivan-marroquin]

thanks for the clarification. i got the package installed as per your instructions. i will a round of tests and let you know how it flies.

[itdxer]

I just want to point out that parameters are still will be shared between network in case if you re-use the same. That's the right behavior which is useful for some of network architectures. Before there was a problem in a way network was constructed and now it suppose to be fixed. Let me know in case if there are any problems

[itdxer]

Here something that you can do in order to test the same layer multiple times

In [29]: from functools import partial
    ...: from neupy import layers, init
    ...:
    ...: layers = [
    ...:     partial(layers.Relu, 10, weight=init.Normal()),
    ...:     partial(layers.Softplus, 10, weight=init.Normal()),
    ...: ]
    ...:
    ...: for i in range(3):
    ...:     for layer in layers:
    ...:         new_layer = layer()
    ...:         print(new_layer, id(new_layer))
    ...:     print('===' * 10)
    ...:
Relu(10) 4536394752
Softplus(10) 4536395424
==============================
Relu(10) 4536396712
Softplus(10) 4536394528
==============================
Relu(10) 4536397720
Softplus(10) 4536397104
==============================

As you can these are different objects on every loop

[itdxer]

FYI @ivan-marroquin
I had one problem with my fix for this bug on Python 2.7 version and you might be interested in the latest update on the release/v0.5.2 in case if you use the same python version.

[itdxer]

This fix is also available now in the latest neupy version 0.5.2. You can install it with pip

[ivan-marroquin]

I have been testing the version that you gave me access to (v0.5.2), and it is working. I am getting reproducible results with other scripts (see attached zip file). Also, I thank you for your suggestion on how to test the same layer multiple times - I will give it a try.

On the other hand, I wrote an script following your TPE example (see attached zip file). Would you like to review it? I want to make sure that I didn't miss anything.

I will install the latest version of Neupy using pip. Could you explain how did you manage to solve the issue on reproducible results?

Again, many thanks for all!

Ivan
TPE_script.zip

[itdxer]

On the other hand, I wrote an script following your TPE example (see attached zip file). Would you like to review it? I want to make sure that I didn't miss anything.

Sorry, there are to much code to review it. I won't have time for it. I did just a quick look and you probably need to use functions like hp.uniform to define you parameter space instead of using np.random. In your case every distribution is discrete which means that it's very limited and less efficient.

Could you explain how did you manage to solve the issue on reproducible results?

It's difficult to explain. The problem that you encountered was one of the "symptoms" of the problem that I encountered a while ago. Basically, it's difficult do differentiate the following to cases in python

# Case 1
x = a > b > c

# Case 2
x1 = a > b
x2 = b > c

In terms of operation that python trigger there is no difference, but in neupy there should be way to differentiate between these two cases. The way I tried to handle this problem before apparently wasn't very good since you managed to encounter this problem. I managed to solve issue in different for the most common case, but there is still small possibility to break it. This possibility is just unlikely to happen. Any solution to this problem has to be tricky, because it goes against Python API.