Running same code multiple times produces different outcomes, and adds weird priors to model.

[OssiGalkin]

Summary:

Running same code multiple times produces different outcomes, and adds weird priors to model.

Description:

Running same code multiple times, e.g. with IPython, produces different outcomes. It seems to me that defining new priors and simulators does not handle memory correctly and they append, not overwrite, old priors and simulator.

Reproducible Steps:

How to reproduce: Open Spyder, copy following code and save it:

import elfi
import numpy as np
from elfi.examples import ma2

seed = 20170530
np.random.seed(seed)

# Define everything similarly as done in introduction. However here all code is taken from examples module.
t1_true = 0.6
t2_true = 0.2

y_obs = ma2.MA2(t1_true, t2_true)

t1 = elfi.Prior(ma2.CustomPrior1, 2)
t2 = elfi.Prior(ma2.CustomPrior2, t1, 1)

Y = elfi.Simulator(ma2.MA2, t1, t2, observed=y_obs)

S1 = elfi.Summary(ma2.autocov, Y)
S2 = elfi.Summary(ma2.autocov, Y, 2)
d = elfi.Distance('euclidean', S1, S2) #

rej = elfi.Rejection(d, batch_size=5000, seed=seed)
result = rej.sample(1000, threshold=0.5) 
print(result.summary)

Run (F5) code. Everything is fine and output should be print in couple of seconds. Then run the code again. Output is different and weird prior nodes are appeared to the model. If code is run again even more priors keep appearing.

Same behavior exists if code is run multiple times from console with IPython.

I think that problem arise from fact that running lines

t2 = elfi.Prior(ma2.CustomPrior2, t1, 1)

Y = elfi.Simulator(ma2.MA2, t1, t2, observed=y_obs)```

again, do not overwrite old prior nodes and/or simulator, but instead they append nodes to old simulator.



#### Current Output for both runs:
runfile('XXX.py', wdir='XXX')
<bound method Sample.summary of Method: Rejection
Number of samples: 1000
Number of simulations: 10000
Threshold: 0.387
Sample means: t1: 0.544, t2: 0.227
>

runfile('XXX.py', wdir='XXX')
<bound method Sample.summary of Method: Rejection
Number of samples: 1000
Number of simulations: 10000
Threshold: 0.386
Sample means: _prior_0466: 0.252, _prior_7dfe: 0.537, t1: -0.0194, t2: 0.356
>

#### Expected Output:

Output should be the same for both runs.


#### ELFI Version:
0.7.1

#### Python Version:
3.6.5

#### Operating System:

Windows 10, version 1709, 64 bit
CPU: Intel i7-5600U

[vuolleko]

You are correct that rerunning code that builds the ELFI graph does not replace but appends new nodes. This behaviour is intended and allows flexibility, though it can cause confusion as well.

Add a call to elfi.new_model() in your script before building the graph.

[OssiGalkin]

That elfi.new_model() should definitely explained more clearly in documentation. I originally encountered this while defining both the simulator and the sampler in the same code shell and run it multiple times while writing the code. I think this is pretty common way to use elfi.

[vuolleko]

Yes, this can cause confusion, and I have added more explanation in the notebooks. It will take a while to have it updated in readthedocs.

elfi-dev/notebooks#11