Reconcile tournaments #1042

drvinceknight · 2017-06-09T18:23:44Z

This would introduce backwards incompatibility at the Tournament call level. This PR would be for v3.0.0

Status quo:

To create a standard tournament:

>>> std = axl.Tournament(players=players, turns=200)

To create a probabilistic ending tournament:

>>> prob_end = axl.ProbEndTournament(players=players, prob_end=.2)

To create a standard spatial tournament:

>>> spatial = axl.SpatialTournament(players=players, turns=200, edges=edges)

To create a probabilistic ending tournament:

>>> prob_end_spatial = axl.ProbEndSpatialTournament(players=players, prob_end=.2, edges=edges)

This PR changes that and removes the ProbEndTournament, SpatialTournament, ProbEndSpatialTournament classes so to create the above we would have:

>>> std = axl.Tournament(players=players, turns=200)
>>> prob_end = axl.Tournament(players=players, prob_end=.2)
>>> spatial = axl.Tournament(players=players, turns=200, edges=edges)
>>> prob_end_spatial = axl.Tournament(players=players, prob_end=.2, edges=edges)

I've done this by carrying out various things:

c695781 Pass the responsibility of sampling a match length in a prob end Match from the MatchGenerator to the Match class. The main consideration here is that I've considered turns and prob_end as stopping conditions for a match:
- If one of them is None the other is used. (If turns is None and prob_end is not None then we have a probabilistic ending match)
- If both are None then a default of 20 turns is used. (Currently no default length is in place, I've picked 20 out of a hat, could use 200 to be in line with tournaments?)
- If both are not None then both are used (so in essence we have a finite match of max length turns but with a probability of finishing earlier).
99e8ed9 Refactor the match_generator module: it had 4 different generators (and 1 parent class), they have all been put in the MatchGenerator class which passes the match parameters to the Match class. The one thing this does is create an edges generator if no edges are passed to it. In this case the edges are the edges of the complete graph (so everyone plays everyone else).
0e859a7 Refactor the rest of the code: removing all the other tournament classes.

One final change is a potential fix of a bug. Currently, in a prob end tournament, the match length between any two strategies is randomly sampled and that sampled length is repeated. In cc4d537 I change this so that a new Match is created at each repetition which in turn means a new length is sampled. I've done this as a separate commit so that it's isolated but I feel that this should be the way it is (I've written a test for it too).

Advantages

This cleans up the code quite a lot (it's basically moving a couple of things and then a bunch of deletions)
This simplifies the creation of tournaments (users don't need to figure out which tournament class to use)
I think this might actually remove the need for Tournament Transformers? #678 (tournament transformers?)
EDIT: Also make things simpler for https://github.com/Axelrod-Python/axelrod-api? (No need to consider separate "interfaces" for different types of tournaments. Just take the input parameters and it'll know what to do.)

Disadvantages

Backwards incompatibility, (basically anyone using ProbEndSpatialTournament etc no longer can). So this would be for v3.0.0.

Let me know what you think :)

This is with a view to reconciling all tournament types in to one type of tournament.

Put all parameters needed for match generation in a single match generator: - edges - noise - turns - prob_end

Simply pass the parameters to Tournament: - edges - prob_end - turns - noise I had to adjust some things: - Refactor of property based tests (all the types still exists but they just call a tournament) - Refactor of fingerprint (just use the Tournament with an edges argument) - Refactor of some tests

Make minor modifications to the doctests.

This ensures that when repeating matches they resample the match length for each repetition.

meatballs · 2017-06-09T18:51:12Z

I like this a lot! I'll have a proper look through the code....

meatballs

I'm suggesting a change rather than requesting one! Feel free to ignore

meatballs · 2017-06-09T19:04:17Z

axelrod/match.py

+            else:
+                turns = sample_length(prob_end)
+        elif turns is None:
+            turns = 20


I know it's my usual reaction to conditional statements but, especially with nesting, I'd be tempted to use a dict here. Something like:

turns_definition = { (True, True): 20, (True, False): sample_length(prob_end), (False, True): turns, (False, False): min(turns, sample_length(prob_end)) } turns = turns_definition[(turns is None, prob_end is None)]

I've just realised I need to move this logic anyway to the play method (for the cache to work as we'd want), I'll see if I see a way of getting a dict to work there (although it'd need to be different to this so that it avoids unnecessary random samples). 👍

marcharper · 2017-06-10T03:04:23Z

axelrod/tournament.py

-                 name: str = 'axelrod', game: Game = None, turns: int = 200,
-                 repetitions: int = 10,
-                 noise: float = 0, with_morality: bool = True) -> None:
+                 match_generator: MatchGenerator = MatchGenerator,


Is match_generator still needed as a parameter?

Would it be better for Tournament to take a dictionary of match parameters?

Is match_generator still needed as a parameter?

I'd be happy to remove it. The only reason we really have it is to allow for the making new tournaments idea: http://axelrod.readthedocs.io/en/stable/tutorials/advanced/making_tournaments.html

But that's such a niche thing I'd be happy to take it out. I'll take it out and we can see what we think :)

Would it be better for Tournament to take a dictionary of match parameters?

I don't think so. It's essentially a dictionary anyway (just kwargs) and as is is more helpful to the user (autocompletion etc...).

marcharper · 2017-06-10T03:05:00Z

axelrod/tournament.py

+        self.edges = edges
+
+        if turns is None and prob_end is None:
+            turns = 200


Maybe we should have a constant DEFAULT_TURNS somewhere

Have change default match length to 200 in: - Matches - Tournaments - Moran Processes Adjusted Moran process tests as a result of it.

Also removed the tutorial on creating new ones. This could be added in again (potentially modifying it by showing how to create new tournaments).

All matches have both a turns and a prob_end attribute. Standard matches have a finite turns and a prob_end of 0. Prob end matches have an infinite number of turns and a prob_end > 0. The play method of the match is where a Match "decides" how many turns to play for. This ensures that the cache will work properly. This also implies that the len of an infinite match is simply not defined and raises an error. Also changed how the length of a prob end match is sampled: do not sample the length if not required.

Have included a test and also adjusted the tests for the fingerprints.

drvinceknight · 2017-06-10T15:12:06Z

I've pushed a few more changes to this:

4ce4098 adds axelrod.default_turns and then uses that throughout as a default (including the Moran process - which changed some tests). I didn't go with axelrod.DEFAULTTURNS but only because I never really know when to use the "shouty" variables (happy to change it).
4ce4098 removes the match_generator as an argument in Tournament and also removes the corresponding tutorial.
ae2c878 changes how the Match decides how long it's going to play for. This is now done in the play method. So every Match has a turns and a prob_end attribute which are never None (but for example can be turns=float('inf') and prob_end=.2) and that's just used in the play method.
As a result of that, d61c957 changes the Tournament back to using the Match class properly (by not creating new Match instances for each repetition).

The other commits are minor things (@meatballs in 1c162eb I moved over to use a dictionary for the input selection instead of nested if statements 👍).

meatballs · 2017-06-10T15:16:12Z

I never really know when to use the "shouty" variables

When they intended to be used as a constant (which python doesn't have) rather than an ordinary variable. This would be a good example of using shoutiness correctly!

meatballs · 2017-06-10T15:16:56Z

I moved over to use a dictionary for the input selection instead of nested if statements

You've no idea how happy that makes me!

drvinceknight · 2017-06-10T15:18:24Z

This would be a good example of using shoutiness correctly!

Ok cool. I'll change it!

- 1 warning in games.rst because I had actually pointed at incorrect label. - Other doctests needed to be fixed: - Some returned to prior state (they had been changed when I was sampling more often than necessary) - Changed a random seed due to default match length being 200.

More Pythonic this way.

meatballs · 2017-06-10T19:07:48Z

Nice piece of work!

marcharper · 2017-06-10T20:05:42Z

LGTM but I do have one suggestion. For the tournament class with_morality seems out of place -- I get that it's a pass through to results set but I think it would be better as an invokable method on the results set rather than something that Tournament should be concerned with.

drvinceknight · 2017-06-10T20:16:06Z

I get that it's a pass through to results set but I think it would be better as an invokable method on the results set rather than something that Tournament should be concerned with.

This wouldn't really work as all the calculations for the result set are done as the interactions are read in (and due to memory constraints, the interactions are discarded). Invoking it on the result set would imply an entire reread of all the interactions. I agree that it's out of place. One option would be to make it no longer default and just have it as something that's always being calculated. I think having it as a optional is legacy.

drvinceknight · 2017-06-10T20:18:04Z

One option would be to make it no longer default and just have it as something that's always being calculated.

Another (perhaps better) option is not to make it part of the Tournament.__init__ but an option that can be passed to Tournament.play.

marcharper · 2017-06-10T21:14:48Z

I'm hitting the button, let's table the morality discussion

@marcharper

On #1042 @marcharper mentioned that it was perhaps worth moving this argument out of the Tournament.init. I took a look and turns out that this argument is not being used anymore (I'm not sure since when). Probably since the move to doing the result calculations on disk.

drvinceknight added 7 commits June 8, 2017 19:52

Add ability for a match to sample the length.

c695781

This is with a view to reconciling all tournament types in to one type of tournament.

Collapse all match generators.

99e8ed9

Put all parameters needed for match generation in a single match generator: - edges - noise - turns - prob_end

Adjust the doctests.

4985602

Make minor modifications to the doctests.

Correct types and change default tests.

2a28788

Fix/change how prob end match lengths are repeated

cc4d537

This ensures that when repeating matches they resample the match length for each repetition.

Add mypy dir to gitignore.

216ff18

meatballs requested changes Jun 9, 2017

View reviewed changes

marcharper reviewed Jun 10, 2017

View reviewed changes

drvinceknight added 6 commits June 10, 2017 15:55

Add default_turns and change moran default

4ce4098

Have change default match length to 200 in: - Matches - Tournaments - Moran Processes Adjusted Moran process tests as a result of it.

Remove match generator argument from tournament.

380eb0e

Also removed the tutorial on creating new ones. This could be added in again (potentially modifying it by showing how to create new tournaments).

Use match cache properly in tournament.

d61c957

Have included a test and also adjusted the tests for the fingerprints.

Use dictionary to set default args in Match

1c162eb

Close file in test.

d5c9dd9

meatballs approved these changes Jun 10, 2017

View reviewed changes

drvinceknight added 3 commits June 10, 2017 16:25

s/default_turns/DEFAULT_TURNS

f28f237

Tweak repetitions in match.play.

5ae0f9d

More Pythonic this way.

marcharper merged commit 3b1a4d6 into master Jun 10, 2017

drvinceknight mentioned this pull request Jun 10, 2017

Reconcile Moran processes #1043

Merged

drvinceknight deleted the reconcile-tournaments branch June 10, 2017 22:24

drvinceknight mentioned this pull request Jun 10, 2017

Tournament Transformers? #678

Closed

drvinceknight mentioned this pull request Jun 10, 2017

Remove morality_metrics as arg for Tournament. #1044

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Reconcile tournaments #1042

Reconcile tournaments #1042

drvinceknight commented Jun 9, 2017 •

edited

Loading

meatballs commented Jun 9, 2017

meatballs left a comment

meatballs Jun 9, 2017 •

edited

Loading

drvinceknight Jun 10, 2017

marcharper Jun 10, 2017

drvinceknight Jun 10, 2017 •

edited

Loading

marcharper Jun 10, 2017

drvinceknight Jun 10, 2017

drvinceknight commented Jun 10, 2017

meatballs commented Jun 10, 2017

meatballs commented Jun 10, 2017

drvinceknight commented Jun 10, 2017

meatballs commented Jun 10, 2017

marcharper commented Jun 10, 2017

drvinceknight commented Jun 10, 2017 •

edited

Loading

drvinceknight commented Jun 10, 2017

marcharper commented Jun 10, 2017

Reconcile tournaments #1042

Reconcile tournaments #1042

Conversation

drvinceknight commented Jun 9, 2017 • edited Loading

meatballs commented Jun 9, 2017

meatballs left a comment

Choose a reason for hiding this comment

meatballs Jun 9, 2017 • edited Loading

Choose a reason for hiding this comment

drvinceknight Jun 10, 2017

Choose a reason for hiding this comment

marcharper Jun 10, 2017

Choose a reason for hiding this comment

drvinceknight Jun 10, 2017 • edited Loading

Choose a reason for hiding this comment

marcharper Jun 10, 2017

Choose a reason for hiding this comment

drvinceknight Jun 10, 2017

Choose a reason for hiding this comment

drvinceknight commented Jun 10, 2017

meatballs commented Jun 10, 2017

meatballs commented Jun 10, 2017

drvinceknight commented Jun 10, 2017

meatballs commented Jun 10, 2017

marcharper commented Jun 10, 2017

drvinceknight commented Jun 10, 2017 • edited Loading

drvinceknight commented Jun 10, 2017

marcharper commented Jun 10, 2017

drvinceknight commented Jun 9, 2017 •

edited

Loading

meatballs Jun 9, 2017 •

edited

Loading

drvinceknight Jun 10, 2017 •

edited

Loading

drvinceknight commented Jun 10, 2017 •

edited

Loading