MESA example #102

TabernaA · 2021-04-14T15:52:49Z

Dear all,
I am trying to connect the EMA workbench framework to my MESA model. I have followed the tutorial for python. Yet, I encounter some errors and I cannot make it run ('Simple model' object is not subscriptable- when running perform_experiment). Is there any chance you could upload an example with a known MESA model (like the MESA version of Wolf Sheep https://github.com/projectmesa/mesa/tree/main/examples/wolf_sheep ). Since I am still a beginner in Python I am pretty sure it is a very stupid mistake that I am doing to not run the model.
Thanks in advance
A

quaquel · 2021-04-14T16:16:27Z

are you trying to extend an AbstractModel class, or are you simply wrapping the mesa model run within a python function?

if you have some code to share, I am happy to take a look.

In fact, I have been meaning to write some documentation on how to write your own connectors by using MESA as an example.

TabernaA · 2021-04-15T08:46:09Z

Hi, thanks for the answer.
Actually I only tried to apply the simplest example to my model mesa object:
but of course it does not work so I don' really know what to do.
An example to see how to build your connector with MESA would be great.
Thanks

steipatr · 2021-04-20T16:14:44Z

Hi Alessandro,
Our files from SocSimFest are available here: https://github.com/BROSE-Uninc/SSF2021 and include an example with the EMA Workbench and a Mesa model. Or are you trying to do something else?

@quaquel I can add this as an MVP example for EMA and mesa if you would like, connected just through Python.

TabernaA · 2021-05-12T18:24:42Z

Hi, thanks, that is great!
Is there any chance i could find the recording somewhere? it would be very helpful

steipatr · 2021-05-18T14:16:24Z

Unfortunately, there were some technical issues on ESSA's side, so the recording is not available.

TabernaA · 2021-05-23T14:01:33Z

Okay thanks. I managed to connect my model to EMA. Yet it is very slow, I remembered there was an option for multiprocessing, is there any example of how to implement that?

Also, which SA (that in in EMA workbench), would you recommend for a non-ergodic model? (2 final possible equilibria)

Thanks in advance!

quaquel · 2021-05-23T16:13:56Z

depends a bit on whether your model needs access to the file system and how you made it work with MESA.

If done correctly, all you need to do is

from ema_workbench import MultiprocessingEvaluator

with MultiprocessingEvaluator(model) as evaluator:
    experiments, outcomes = evaluator.perform_experiments(100)

TabernaA · 2021-05-23T18:10:39Z

Thanks. I actually did adapt my model to be the same as the tutorial:

This is my code:

   def model_EEG_ABM(F1=5,
                 F2 =25,
                 H =350 ,
                 B =1 ,
                 T = 2,
                 S = 0,
                 seed = 4,
                 steps=200):

        import model

        EEG_ABM = model.KSModel(F1=50,
                        F2 =250,
                        H =3500 ,
                        B =1 ,
                        T = 0.02 ,
                        S =0,
                        seed = 4)
         
        EEG_ABM.run_model(steps)

        outcomes = EEG_ABM.datacollector.get_model_vars_dataframe()

        return {'TIME' : list(range(steps + 1)),
               "Real GDP coastal" : outcomes["Real GDP coastal"].tolist(),
               "Real GDP internal" : outcomes["Real GDP internal"].tolist(),
               "Unemployment rate coastal" : outcomes["Unemployment rate coastal"].tolist(),
               "Unemployment rate internal" : outcomes["Unemployment rate internal"].tolist()}

Importantly, if i test it like this it works fine:

    F1=50
    F2 =250
    H =3500
    B = 1 
    T = 2 
    S =0
    seed = 4
    steps = 10
    model_EEG_ABM(F1= F1,
                           F2 =F2,
                           H =H ,
                           B = B ,
                           T = T ,
                           S =S ,
                           seed = seed,
                          steps= steps)

Outputs:
Out[18]:
{'TIME': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10],
'Real GDP coastal': [125, 0, 500, 1000, 1770, 2898, 3008, 3010, 3000, 3000, 2750],
'Real GDP internal': [125, 0, 500, 1000, 1772, 2846, 2750, 2750, 2500, 2500, 2127],
'Unemployment rate coastal': [0.0, 1.0, 0.57, 0.17, 0.0, 0.11, 0.14, 0.14, 0.14, 0.14, 0.21],
'Unemployment rate internal': [0.0, 1.0, 0.57, 0.19, 0.0, 0.11, 0.21, 0.21, 0.29, 0.29, 0.0]}

However I get some issues when I run the experiments:

  model = ReplicatorModel('KSModel', function=model_EEG_ABM)

  model.uncertainties = [RealParameter("T", 0.01, 0.1),
                   RealParameter("S", 0.02, 0.15)]

  model.constants = [Constant('F1', F1),
               Constant('F2', F2),
               Constant('H', H),
               Constant('B', B),
               Constant('seed', seed),
               Constant('steps', steps)]

   model.outcomes = [TimeSeriesOutcome('TIME'),
              TimeSeriesOutcome('Real GDP coastal'),
              TimeSeriesOutcome('Unemployment rate coastal')]

   model.replications = 1

results = perform_experiments(model, 10, uncertainty_sampling='sobol')

I get this error:

File "/usr/local/anaconda3/lib/python3.8/site-packages/ema_workbench/em_framework/callbacks.py", line 230, in _store_outcomes
self.results[outcome][case_id, ] = outcome_res

KeyError: 'TIME'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

File "", line 25, in
results = perform_experiments(model, 10, uncertainty_sampling='sobol')

File "/usr/local/anaconda3/lib/python3.8/site-packages/ema_workbench/em_framework/evaluators.py", line 468, in perform_experiments
evaluator.evaluate_experiments(scenarios, policies, callback)

File "/usr/local/anaconda3/lib/python3.8/site-packages/ema_workbench/em_framework/evaluators.py", line 230, in evaluate_experiments
callback(experiment, outcomes)

File "/usr/local/anaconda3/lib/python3.8/site-packages/ema_workbench/em_framework/callbacks.py", line 265, in call
self._store_outcomes(experiment.experiment_id, outcomes)

File "/usr/local/anaconda3/lib/python3.8/site-packages/ema_workbench/em_framework/callbacks.py", line 244, in _store_outcomes
self.results[outcome][:] = np.nan

ValueError: cannot convert float NaN to integer

Any clue?
Thanks in advance for the support I am almost there.

quaquel · 2021-05-23T19:00:08Z

which version of the workbench are you using? I recently addressed a bug related to this.

TabernaA · 2021-05-23T19:10:28Z

I am using 2.0.9

quaquel · 2021-05-23T19:40:37Z

the problem is presently TIME is a range of integers. If this were a range of floats, the problem would be gone.

more fundamentally, this is a bug that needs fixing on the workbench side.

TabernaA · 2021-05-23T20:07:01Z

I have changed time in:
'TIME' : [float(i) for i in range(steps +1)]
But still I get the same error.
I have also tried to remove 'TIME' so using only the other variables but still I get the same error.

Then I changed also the other variables into floats and it works. So the bottom line is: only outcomes with float.
thanks!

quaquel · 2021-05-24T07:30:01Z

and the fact that outcomes need to be floats is indeed the bug that needs fixing

TabernaA · 2021-05-24T13:15:20Z

I see, perfect!
Thanks.
I still got problems with parallel runs but I guess is something on my end that I got to fix.

quaquel · 2021-05-24T13:18:28Z

regarding the parallel running a few questions because the code seems fine.

what OS are you using?
Are you in a .py file or a notebook?
What is the nature of the error?

A fix is being tested for the other bug, so hopefully it will be more permanently fixed in the future.

TabernaA · 2021-05-24T13:49:07Z

At the bottom of the code I have added:

   from ema_workbench import MultiprocessingEvaluator

   with MultiprocessingEvaluator(model) as evaluator:
   experiments, outcomes = evaluator.perform_experiments(10)

I am using macOS Big Sur 11.3.1
I am in a .py file
The error is that I keep receiving this error in a loop:

[MainProcess/INFO] pool started
[MainProcess/INFO] performing 10 scenarios * 1 policies * 1 model(s) = 10 experiments
/usr/local/lib/python3.9/site-packages/ema_workbench/em_framework/optimization.py:48: ImportWarning: platypus based optimization not available

Traceback (most recent call last):
File "", line 1, in
File "/usr/local/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/spawn.py", line 116, in spawn_main
exitcode = _main(fd, parent_sentinel)
File "/usr/local/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/spawn.py", line 125, in _main
prepare(preparation_data)
File "/usr/local/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/spawn.py", line 236, in prepare
_fixup_main_from_path(data['init_main_from_path'])
File "/usr/local/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/spawn.py", line 287, in _fixup_main_from_path
main_content = runpy.run_path(main_path,
File "/usr/local/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/runpy.py", line 268, in run_path
return _run_module_code(code, init_globals, run_name,
File "/usr/local/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/runpy.py", line 97, in _run_module_code
_run_code(code, mod_globals, init_globals,
File "/usr/local/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/Users/ataberna/Documents/Theoretical ABM/code/Compiler ready to run version/untitled0.py", line 125, in
with MultiprocessingEvaluator(model) as evaluator:
File "/usr/local/lib/python3.9/site-packages/ema_workbench/em_framework/evaluators.py", line 95, in enter
self.initialize()
File "/usr/local/lib/python3.9/site-packages/ema_workbench/em_framework/evaluators.py", line 279, in initialize
self._pool = multiprocessing.Pool(self.n_processes, initializer,
File "/usr/local/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/context.py", line 119, in Pool
return Pool(processes, initializer, initargs, maxtasksperchild,
File "/usr/local/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/pool.py", line 212, in init
self._repopulate_pool()
File "/usr/local/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/pool.py", line 303, in _repopulate_pool
return self._repopulate_pool_static(self._ctx, self.Process,
File "/usr/local/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/pool.py", line 326, in _repopulate_pool_static
w.start()
File "/usr/local/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/process.py", line 121, in start
self._popen = self._Popen(self)
File "/usr/local/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/context.py", line 284, in _Popen
return Popen(process_obj)
File "/usr/local/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/popen_spawn_posix.py", line 32, in init
super().init(process_obj)
File "/usr/local/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/popen_fork.py", line 19, in init
self._launch(process_obj)
File "/usr/local/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/popen_spawn_posix.py", line 42, in _launch
prep_data = spawn.get_preparation_data(process_obj._name)
File "/usr/local/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/spawn.py", line 154, in get_preparation_data
_check_not_importing_main()
File "/usr/local/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/spawn.py", line 134, in _check_not_importing_main
raise RuntimeError('''
RuntimeError:
An attempt has been made to start a new process before the
current process has finished its bootstrapping phase.

    This probably means that you are not using fork to start your
    child processes and you have forgotten to use the proper idiom
    in the main module:

        if __name__ == '__main__':
            freeze_support()
            ...

    The "freeze_support()" line can be omitted if the program
    is not going to be frozen to produce an executable.

And then it starts again...
Traceback (most recent call last):
File "", line 1, in
File "/usr/local/Cellar/python@3.9/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/spawn.py", line 116, in spawn_main
exitcode = _main(fd, parent_sentinel)

....and so on

quaquel · 2021-05-24T13:56:34Z

hmmmm.... You are using brew for your python distribution if I am not mistaken. And python 3.9 as well.

I am on a Mac as well, Big Sur also, but anaconda distribution and python 3.8. I have not seen your error before.

TabernaA · 2021-05-24T14:12:20Z

Interesting. I think it might be something wrong with brew. I have to test
Thanks for the help!

TabernaA · 2021-06-22T20:10:39Z

Hi guys, just a follow-up. I made the general ema workbench working (I had a problem with brew and conda directory mixed up). However I can't make it work with the multiprocessor. It does not give any error yet it nevers finish the run. (it basically says "n experiments running sequentially". Anyone noticed that before (the code is the same as before)?
Cheers

quaquel · 2021-06-23T05:46:31Z

if it says running sequentially, you are not using multiprocessing. Can you try one of the pure python examples that come with the workbench (e.g., the lake problem)?

TabernaA · 2021-06-23T08:33:13Z

I am sorry, it says 'pool started' not running sequentially. I tried to copy and paste the sobol lake problem, it stayed like this the for 3 hours: (I am running in spyder, in a server that has 40 CPUs so it is usually very fast)

C:\Users\TabernaA\Anaconda3\lib\site-packages\ema_workbench\em_framework\optimization.py:48: ImportWarning: platypus based optimization not available
warnings.warn("platypus based optimization not available", ImportWarning)
[MainProcess/INFO] pool started
[MainProcess/INFO] performing 12000 scenarios * 1 policies * 1 model(s) = 12000 experiments

TabernaA · 2021-06-23T08:33:35Z

I closed by mistake sorry

quaquel · 2021-06-23T08:45:32Z

you really need to provide much more context. For example, are you using notebooks, a normal script, can you show the exact code? Also why not try to make it work on your Mac first.

TabernaA · 2021-06-23T09:04:49Z

Hey thanks for the quick reply.
As I said I am using Spyder, the code is the same as the one I posted before so:

def model_EEG_ABM(F1=5,
F2 =25,
H =350 ,
B =1 ,
T = 2,
S = 0,
seed = 4,
steps=200):

    import model

    EEG_ABM = model.KSModel(F1=50,
                    F2 =250,
                    H =3500 ,
                    B =1 ,
                    T = 0.02 ,
                    S =0,
                    seed = 4)
     
    EEG_ABM.run_model(steps)

    outcomes = EEG_ABM.datacollector.get_model_vars_dataframe()

    return {'TIME' : list(range(steps + 1)),
           "Real GDP coastal" : outcomes["Real GDP coastal"].tolist(),
           "Real GDP internal" : outcomes["Real GDP internal"].tolist(),
           "Unemployment rate coastal" : outcomes["Unemployment rate coastal"].tolist(),
           "Unemployment rate internal" : outcomes["Unemployment rate internal"].tolist()}


F1=50
F2 =250
H =3500
B = 1 
T = 2 
S =0
seed = 4
steps = 10
model_EEG_ABM(F1= F1,
                       F2 =F2,
                       H =H ,
                       B = B ,
                       T = T ,
                       S =S ,
                       seed = seed,
                      steps= steps)

Outputs:
Out[18]:
{'TIME': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10],
'Real GDP coastal': [125, 0, 500, 1000, 1770, 2898, 3008, 3010, 3000, 3000, 2750],
'Real GDP internal': [125, 0, 500, 1000, 1772, 2846, 2750, 2750, 2500, 2500, 2127],
'Unemployment rate coastal': [0.0, 1.0, 0.57, 0.17, 0.0, 0.11, 0.14, 0.14, 0.14, 0.14, 0.21],
'Unemployment rate internal': [0.0, 1.0, 0.57, 0.19, 0.0, 0.11, 0.21, 0.21, 0.29, 0.29, 0.0]}

  model = ReplicatorModel('KSModel', function=model_EEG_ABM)
 model.uncertainties = [RealParameter("T", 0.01, 0.1),
               RealParameter("S", 0.02, 0.15)]
 model.constants = [Constant('F1', F1),
           Constant('F2', F2),
           Constant('H', H),
           Constant('B', B),
           Constant('seed', seed),
           Constant('steps', steps)]
   model.outcomes = [TimeSeriesOutcome('TIME'),
          TimeSeriesOutcome('Real GDP coastal'),
          TimeSeriesOutcome('Unemployment rate coastal')]
    model.replications = 1

If I do individual run it works:

 results = perform_experiments(model, 10, uncertainty_sampling='sobol')

While if I do:

 from ema_workbench import MultiprocessingEvaluator  
 with MultiprocessingEvaluator(model) as evaluator:
        evaluator.perform_experiments(100, uncertainty_sampling='sobol'))

I get:

In[4]
with MultiprocessingEvaluator(model) as evaluator:
experiments, outcomes = evaluator.perform_experiments(100, uncertainty_sampling='sobol')
[MainProcess/INFO] pool started
[MainProcess/INFO] performing 600 scenarios * 1 policies * 1 model(s) = 600 experiments

But it stays like this forever, without never finishing...I would like to run in the server because with MESA batch run I can 40 in parallel while in my mac only 12 and the server can run overnight without problems.

Also if I try with the lake model (the sobol one), just copy and paste in a .py file in spyder, I get the same problem.

quaquel · 2021-06-23T09:23:18Z

please show the full code, not snippets. My hunch is that you miss

if __name__ == '__main__':
    # model definition and run code needs to go here

EwoutH · 2023-12-14T10:56:57Z

@quaquel I think we should try not to rescope this one again, but get it in 2.5.

If not a tutorial on connectors, just a simple example.

EwoutH · 2023-12-15T20:33:41Z

Don't have a computer at hand so just writing this thought down: Can we make the EMAworkbench leverage or interface with the Mesa batchrunner?

quaquel · 2023-12-15T20:51:41Z

I'm unsure what that would mean or why you would want to do that. The batch runner, in my understanding, specifies a full factorial over the provided parameter values. The workbench offers you much more flexibility.

TabernaA closed this as completed Jun 23, 2021

TabernaA reopened this Jun 23, 2021

quaquel added the enhancement label Jul 5, 2022

EwoutH assigned quaquel Aug 31, 2022

EwoutH added this to the 2.3.0 milestone Aug 31, 2022

EwoutH added the docs label Aug 31, 2022

EwoutH mentioned this issue Oct 28, 2022

release of version 2.3 #210

Merged

quaquel modified the milestones: 2.3.0, 2.4.0 Nov 7, 2022

quaquel modified the milestones: 2.4.0, 2.5.0 Apr 19, 2023

steipatr mentioned this issue Dec 14, 2023

Add Mesa example #335

Merged

quaquel closed this as completed in #335 Dec 16, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

MESA example #102

MESA example #102

TabernaA commented Apr 14, 2021

quaquel commented Apr 14, 2021

TabernaA commented Apr 15, 2021

steipatr commented Apr 20, 2021

TabernaA commented May 12, 2021 •

edited

steipatr commented May 18, 2021

TabernaA commented May 23, 2021 •

edited

quaquel commented May 23, 2021 •

edited

TabernaA commented May 23, 2021 •

edited

quaquel commented May 23, 2021

TabernaA commented May 23, 2021

quaquel commented May 23, 2021

TabernaA commented May 23, 2021 •

edited

quaquel commented May 24, 2021

TabernaA commented May 24, 2021

quaquel commented May 24, 2021

TabernaA commented May 24, 2021 •

edited

quaquel commented May 24, 2021

TabernaA commented May 24, 2021

TabernaA commented Jun 22, 2021

quaquel commented Jun 23, 2021

TabernaA commented Jun 23, 2021

TabernaA commented Jun 23, 2021

quaquel commented Jun 23, 2021

TabernaA commented Jun 23, 2021 •

edited

quaquel commented Jun 23, 2021

EwoutH commented Dec 14, 2023

EwoutH commented Dec 15, 2023

quaquel commented Dec 15, 2023

MESA example #102

MESA example #102

Comments

TabernaA commented Apr 14, 2021

quaquel commented Apr 14, 2021

TabernaA commented Apr 15, 2021

steipatr commented Apr 20, 2021

TabernaA commented May 12, 2021 • edited

steipatr commented May 18, 2021

TabernaA commented May 23, 2021 • edited

quaquel commented May 23, 2021 • edited

TabernaA commented May 23, 2021 • edited

quaquel commented May 23, 2021

TabernaA commented May 23, 2021

quaquel commented May 23, 2021

TabernaA commented May 23, 2021 • edited

quaquel commented May 24, 2021

TabernaA commented May 24, 2021

quaquel commented May 24, 2021

TabernaA commented May 24, 2021 • edited

quaquel commented May 24, 2021

TabernaA commented May 24, 2021

TabernaA commented Jun 22, 2021

quaquel commented Jun 23, 2021

TabernaA commented Jun 23, 2021

TabernaA commented Jun 23, 2021

quaquel commented Jun 23, 2021

TabernaA commented Jun 23, 2021 • edited

quaquel commented Jun 23, 2021

EwoutH commented Dec 14, 2023

EwoutH commented Dec 15, 2023

quaquel commented Dec 15, 2023

TabernaA commented May 12, 2021 •

edited

TabernaA commented May 23, 2021 •

edited

quaquel commented May 23, 2021 •

edited

TabernaA commented May 23, 2021 •

edited

TabernaA commented May 23, 2021 •

edited

TabernaA commented May 24, 2021 •

edited

TabernaA commented Jun 23, 2021 •

edited