Merge Grid, SingleGrid, and MultiGrid into one class

[kumom]

This is some follow-up work for reasons explained in #623. I basically merge three classes (Grid, SingleGrid, MultiGrid) into one, and use multigrid as a boolean attribute to specify the behavior of the class. I haven't modified the test files, so it will for sure fail since I changed some method signatures. In addition, I notice HexGrid is a subtype of Grid, and this modification will cause some of its behaviors to fail potentially.

What I intend to do next, if the current hierarchy looks good to you, is to change multigrid into grid_type. Allowed values of grid_type would be single, multi, or hex, so that HexGrid would be "merged" to Grid class as well. Just a side node, I adopt this style of code because I am somewhat influenced by the style of sklearn.

I also had this question as I read the source code. The only reason I can think of is probably to avoid concurrency issues/code optimization done by compiler. Is there any good reason to write the code this way?

[Corvince]

Great PR! However, as the changes would break every model with a Grid, we have to be clever on how to to the transition.

I think calls to SingleGrid and MultiGrid should call the appropriate version of Grid and issue a Depreciation warning. Same for some of the removed functions.

I added a bunch of code comments, sometimes they are specific to your PR, but some a more general remarks on the grid functions.

I also opened a topic on the mailing list to discuss how to proceed with you proposed changes

[Corvince]

Summary line and explanation of the method is missing

[Corvince]

I think this will fail for multigrid? As old_agent would be a set. It is also not clear how "replace" should work for a multigrid. Should it replace a single agent? All agents? Or can't it be used with MultiGrid?

[Corvince]

I think the function should also work with other sequences, e.g. ["random", "random"]

[Corvince]

Comment and code got me confused. You don't pick a random cell here, you just make a set (why not list?) of empty cells and then randomly select one further down

[Corvince]

The agents pos attribute should also be set to None

[kumom]

Thanks a lot for the comments! I was rushing a bit to show the hierarchy. I was also a bit conservative due to backward compatibility problem like you said. I haven't really taken a good look at any other classes yet, so I am hesitant about merging HexGrid as well. I am implementing a model involving network and I will try to use NetworkGrid to test out the functions there. I will look into the code more carefully next week and push new code to the branch.

[kumom]

btw, is there any motivation to name a network NetworkGrid? I was a bit confused by the name, but since I did not really look at the code and for backward compatibility I left this class intact.

[Corvince]

Really quick, so you get this info if you want to work on this on the weekend:

I tested your branch and there are several severe bugs. I suggest adding

class SingleGrid(Grid):
    """Depreciated class."""
    def __init__(self, width, height, torus):
        super().__init__(width, height, torus, multigrid=False)


class MultiGrid(Grid):
    """Depreciated class."""
    def __init__(self, width, height, torus):
        super().__init__(width, height, torus, multigrid=True)

to your file and then run the tests to see a lot of errors (not just related to missing functions). I already made some changes. I wanted to PR my changes into your repo, but since you did not fork from projectmesa, I wasn't able to tell GitHub where to PR. So here is a gist of my file based on your changes.

[Corvince]

As noted already there are several bugs in this implementation. See this PR to your repo for which ones and how I solved them.

I would suggest the following approach: Make sure the current tests work (by adding in deprecated functions and/or resolve bugs). Then add new tests based on the current implementation.

Again, I really like your changes/new structure but its not ready for a merge yet. I would also like to see a return of the get_neighborhood function (including center).

[kumom]

As noted already there are several bugs in this implementation. See this PR to your repo for which ones and how I solved them.

I would suggest the following approach: Make sure the current tests work (by adding in deprecated functions and/or resolve bugs). Then add new tests based on the current implementation.

Again, I really like your changes/new structure but its not ready for a merge yet. I would also like to see a return of the get_neighborhood function (including center).

Thanks a million for the efforts! I know the current code was committed under rush (sorry for that) and I intended to work on it this week.

[kumom]

I don't understand why I pass the tests locally but fail in Travis...I really don't want to push a lot to annoy you, but I am not sure why this happens.

tests/test_batchrunner.py ....                                           [  4%]
tests/test_datacollector.py ....                                         [  9%]
tests/test_examples.py .                                                 [ 10%]
tests/test_grid.py ...............................                       [ 45%]
tests/test_lifespan.py ..                                                [ 47%]
tests/test_main.py .                                                     [ 48%]
tests/test_scaffold.py F                                                 [ 50%]
tests/test_space.py .............................                        [ 82%]
tests/test_time.py ........                                              [ 92%]
tests/test_usersettableparam.py ....                                     [ 96%]
tests/test_visualization.py ...                                          [100%]Coverage.py warning: No data was collected. (no-data-collected)

In addition, the failure is

=================================== FAILURES ===================================
________________ ScaffoldTest.test_scaffold_creates_project_dir ________________

self = <test_scaffold.ScaffoldTest testMethod=test_scaffold_creates_project_dir>

    def test_scaffold_creates_project_dir(self):
        with self.runner.isolated_filesystem():
            assert not os.path.isdir("example_project")
            self.runner.invoke(cli, ['startproject', '--no-input'])
>           assert os.path.isdir("example_project")
E           AssertionError: assert False
E            +  where False = <function isdir at 0x103c989d8>('example_project')
E            +    where <function isdir at 0x103c989d8> = <module 'posixpath' from '/Users/kumom/anaconda3/lib/python3.6/posixpath.py'>.isdir
E            +      where <module 'posixpath' from '/Users/kumom/anaconda3/lib/python3.6/posixpath.py'> = os.path

tests/test_scaffold.py:21: AssertionError
----------------------------- Captured stdout call -----------------------------
A valid repository for "/Users/kumom/anaconda3/lib/python3.6/site-packages/mesa/cookiecutter-mesa" could not be found in the following locations:
/Users/kumom/anaconda3/lib/python3.6/site-packages/mesa/cookiecutter-mesa
/Users/kumom/anaconda3/lib/python3.6/site-packages/mesa/cookiecutter-mesa

However, I have passed the local tests when I did the PR #622, and I haven't updated/downgraded the package in any way. Why would this error suddenly pops up?

[Corvince]

Exciting new commits! I will review your changes and give you more feedback in the next days.

RE tests: Maybe try to clean your local .pytest_cache folders or something like this? Also not sure which tests you pass locally

[jackiekazil]

@kumom this is a great PR. At first glance / skim, I have to say I agree with Corvince. If you aren't a part of the dev group -- can you join us there for the discussion? (Otherwise, we will report back here.) https://groups.google.com/forum/#!topic/projectmesa-dev/H7IUeol8iRM

[kumom]

@kumom this is a great PR. At first glance / skim, I have to say I agree with Corvince. If you aren't a part of the dev group -- can you join us there for the discussion? (Otherwise, we will report back here.) https://groups.google.com/forum/#!topic/projectmesa-dev/H7IUeol8iRM

I am in the group already. Thanks a lot for the comments.

[kumom]

As noted already there are several bugs in this implementation. See this PR to your repo for which ones and how I solved them.

I would suggest the following approach: Make sure the current tests work (by adding in deprecated functions and/or resolve bugs). Then add new tests based on the current implementation.

Again, I really like your changes/new structure but its not ready for a merge yet. I would also like to see a return of the get_neighborhood function (including center).

After cleaning the cache, I passes all the tests (with the buggy code)...

============================= test session starts ==============================
platform darwin -- Python 3.6.5, pytest-4.0.2, py-1.5.3, pluggy-0.8.1
rootdir: /Users/kumom/Documents/Projects/mesa, inifile:
plugins: remotedata-0.2.1, openfiles-0.3.0, doctestplus-0.1.3, cov-2.6.1, arraydiff-0.2
collected 88 items

tests/test_batchrunner.py ....                                           [  4%]
tests/test_datacollector.py ....                                         [  9%]
tests/test_examples.py .                                                 [ 10%]
tests/test_grid.py ...............................                       [ 45%]
tests/test_lifespan.py ..                                                [ 47%]
tests/test_main.py .                                                     [ 48%]
tests/test_scaffold.py .                                                 [ 50%]
tests/test_space.py .............................                        [ 82%]
tests/test_time.py ........                                              [ 92%]
tests/test_usersettableparam.py ....                                     [ 96%]
tests/test_visualization.py ...                                          [100%]Coverage.py warning: No data was collected. (no-data-collected)


---------- coverage: platform darwin, python 3.6.5-final-0 -----------
Name                                                    Stmts   Miss  Cover
---------------------------------------------------------------------------
mesa/__init__.py                                            9      9     0%
mesa/agent.py                                              10     10     0%
mesa/batchrunner.py                                       152    152     0%
mesa/datacollection.py                                     78     78     0%
mesa/main.py                                               25     25     0%
mesa/model.py                                              28     28     0%
mesa/space.py                                             326    326     0%
mesa/time.py                                               61     61     0%
mesa/visualization/ModularVisualization.py                120    120     0%
mesa/visualization/TextVisualization.py                    40     40     0%
mesa/visualization/UserParam.py                            50     50     0%
mesa/visualization/__init__.py                              1      1     0%
mesa/visualization/modules/CanvasGridVisualization.py      25     25     0%
mesa/visualization/modules/ChartVisualization.py           25     25     0%
mesa/visualization/modules/HexGridVisualization.py         28     28     0%
mesa/visualization/modules/NetworkVisualization.py         16     16     0%
mesa/visualization/modules/TextVisualization.py             5      5     0%
mesa/visualization/modules/__init__.py                      6      6     0%
---------------------------------------------------------------------------
TOTAL                                                    1005   1005     0%

[Corvince]

Here is something that's up to a discussion:

You have removed the get_neighbors and get_neighborhood functions and provide a single neighbors function. Although you can kind of get around with the get_agents and include_empty arguments, I think the former functions are much clearer.
This might come down to personal taste, but my personal expectations would be:
neighborhood: A list of cells that surround an agent (and, per default, include the own cell). Might or might not return agents
neighbors: A list of agents that are within my neighborhood (and in a multigrid possibly on my own cell)

I think the use cases for both functions are quite distinct and I would like to keep them separate. Also your function defaults would return a list of cell coordinates, which are not empty. I think that is kind of odd.

But as I said this is just my personal opinion/expectation. Feel free to discuss with me

[Corvince]

After cleaning the cache, I passes all the tests (with the buggy code)...

Thats strange. When I run your changes locally the same tests fail as on travis

[dmasad]

@kumom Thanks for working on this -- my memory is that the split into single/multi-grid was a path-dependent artifact and not a well-thought-out design choice, so this is probably long overdue.

I agree with @Corvince that it makes sense to have separate methods for getting neighbors (agents) and neighborhood (coordinates). Even if one is just a thin wrapper around the other, it's one less thing for a modeler to need to implement.

I'm also seeing the same test failures on my end as Travis is showing.

[kumom]

I am still debugging the issue with pytest, and I think I have figured out why all tests are passed - because mesa installed via pip was imported. However, after I ran pip3 uninstall mesa and did pytest --cov=mesa tests/ again, I received the following error

__________________ ERROR collecting tests/test_batchrunner.py __________________
ImportError while importing test module '/Users/kumom/Documents/Projects/mesa/tests/test_batchrunner.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
tests/test_batchrunner.py:8: in <module>
    from mesa import Agent, Model
E   ImportError: cannot import name 'Agent'

I ran this under a new folder cloned from projectmesa/mesa. Why would this happen? Am I doing something wrong with pytest?

[Corvince]

Pytest ist fine, but you still have to install Mesa. Instead of pip install Mesa, which pulls it from pypi, try to install it in the root dir (projectmesa/Mesa) with pip install -e .[dev]

[kumom]

Pytest ist fine, but you still have to install Mesa. Instead of pip install Mesa, which pulls it from pypi, try to install it in the root dir (projectmesa/Mesa) with pip install -e .[dev]

Thanks a lot! Problem solved already

[EwoutH]

@Corvince At some point in developing the PropertyGrid in #1898 I also considered merging them. Instead of a binary "single" or "multi", I considered a capacity keyword which could be 1 (equivalent to Single), inf (equivalent to Multi) or any integer in between (new functionality).

Do you think ideally we should pursue merging these classes? If so, do you think it's worth it?

The reasons I didn't where 1) backwards compatibility and 2) performance (some functions are faster if there can be only 1 agent anywhere or if there can be infinite somewhere).

• See Proposal: Adding Cell Properties to Grids in Mesa #1889 for some background on the capacity idea.

[Corvince]

I think that's a pretty solid idea and I think it would be possible to do this in a backwards compatible way. I would guess the devil will be in the details, for example for grid access do you return a single agent or a list of a single agent? The latter would be more consistent but possibly confusing (why return a list if there can only be one agent)