ML strategies #803

marcharper · 2017-01-02T00:01:30Z

This PR adds several new strategies.

Two strategies from the literature: Winner12 and Winner21
A new class of strategies based on hidden Markov models
Newly trained versions of players based on finite state machines, HMM, EvolvedLookerUp, PSOGambler, and Evolved ANN

Of particular note:

EvolvedFSM16, a 16 node FSM player that is the new "best" strategy
EvolvedFSM16Noise05, a 16 node FSM player that is the new best strategy for noisy tournaments (and quite good in the noise free tournaments)
PSOGamblerMem1, a memory one strategy trained with the PSO algorithm
A revised evolved looker up that performs well
EvolvedANN5 (with a smaller inner layer)
EvolvedHMM, a hidden markov model based strategy

It's likely that these strategy are not the best possible, and that better versions can be evolved in the future. I trained many strategies, some of which are not included, particularly:

Preliminary models for the Moran process (not added to the standard list, and likely to be improved)
Models trained to win rather than achieve a high net score

There are also other conceivable training modes, e.g. maximum total score (self + opponent). The point is: we may decide to add or remove trained strategies in the future.

Also in this PR:

Some of the strategy classes were refactored to allow better training. See the code in the axelrod-evolver repository for reference and training code.
Model data for the trained strategies is stored in a subdirectory allowing for easy addition of new strategies, and to prevent storing data directly in strategy files. The exception is HMM (for which I've only included one strategy).
To the point, since there are a lot of e.g. lookerup models, the strategy classes are created at runtime.

drvinceknight · 2017-01-02T08:11:55Z

Oooof this is a biggy!

drvinceknight

This is a first shallow sweep, there's a lot here: could take me a little while but it looks like awesome work!

drvinceknight · 2017-01-02T08:17:30Z

axelrod/strategies/worse_and_worse.py

    """

    name = 'Worse and Worse'
    classifier = {
-        'memory_depth': float('inf'),


It needs to know the current turn so does that not equate to knowing how long the game has been? (Thus infinite memory).

I'm actually not sure about this one so I've changed it back to float('inf') for now. The question is whether using the round number counts as using history or not, let's discuss elsewhere.

drvinceknight · 2017-01-02T08:18:45Z

axelrod/tests/unit/test_hmm.py

@@ -0,0 +1,123 @@
+"""Tests for Finite State Machine Strategies."""


hidden markov model strategies

drvinceknight · 2017-01-02T08:21:22Z

docs/reference/bibliography.rst

@@ -22,6 +22,10 @@ documentation.
 .. [Li2009] Li, J. & Kendall, G. (2009). A Strategy with Novel Evolutionary Features for the Iterated Prisoner’s Dilemma. Evolutionary Computation 17(2): 257–274.
 .. [Li2011] Li, J., Hingston, P., Member, S., & Kendall, G. (2011). Engineering Design of Strategies for Winning Iterated Prisoner ’ s Dilemma Competitions, 3(4), 348–360.
 .. [Li2014] Li, J. and Kendall, G. (2016). The Effect of Memory Size on the Evolutionary Stability of Strategies in Iterated Prisoner's Dilemma. IEEE Transactions on Evolutionary Computation, 18(6) 819-826
+.. [Mathieu2015] Mathieu, P. and Delahaye, J. New Winning Strategies


When the docs build does this look right? (Just doesn't match the one line formatting used for all others).

Citation updated

drvinceknight · 2017-01-02T08:22:35Z

setup.py

    url='http://axelrod.readthedocs.org/',
    license='The MIT License (MIT)',
    description='Reproduce the Axelrod iterated prisoners dilemma tournament',
+    include_package_data=True,


If you haven't, could you check that this pip installs? (eg pip installing in to a virtual env from the local dir)

It does locally, and the tests would fail on travis otherwise.

I don't think travis specifically tests this (just fyi) but appveyor does test the setup install (because we had some windows problems with that at some point...).

drvinceknight · 2017-01-02T08:23:49Z

axelrod/load_data_.py

@@ -0,0 +1,45 @@
+import pkg_resources


Could we have unit tests for these please.

Not sure how to better test these than simply loading the data and testing the strategies. I added a few integrity checks in ANN and LookerUp to make sure the data is of the expected length.

I was thinking that we could have a tests/unit/test_load_data.py file that just checks that these functions run and that the data is of the expected format separately.

I'm not sure that we'd be testing anything further in that case -- if the format or data types are wrong the strategies will fail when constructed or played.

I agree that this wouldn't test anything further, it just consolidates things: for example in the future these ml strategies could be changed to no longer read the data (hypothetically), their tests adjusted and an error creeping in to these reader functions. That's a weird case but I think my point holds?

But if the players no longer read the data then the data and these functions are unnecessary (and their coverage will disappear). So wouldn't we just delete the data and these functions in that case, unless something else is using them? And if something else is using them then a bad change will still break those things.

Very good point, I'd still say too many tests is better than too phew but I won't insist. :) 👍

Maybe type annotations are a good check here so when start annotating we'll get a little extra coverage.

drvinceknight · 2017-01-02T08:31:29Z

axelrod/strategies/hmm.py

+    """Implementation of a basic Hidden Markov Model. We assume that the
+    transition matrix is conditioned on the opponent's last action, so there
+    are two transition matrices. Emission distributions are stored as Bernoulli
+    probabilities for each state. This is essentially a stochastic FSM.


Names - SimpleHMM: ...

This one isn't a strategy but I updated the HMM Players with names

Sorry I got over zealous :)

drvinceknight · 2017-01-02T08:31:43Z

axelrod/strategies/hmm.py

+
+
+class HMMPlayer(Player):
+    """Abstract base class for Hidden Markov Model players."""


Same comment names ...

drvinceknight · 2017-01-02T08:31:53Z

axelrod/strategies/hmm.py

+        self.hmm.state = self.initial_state
+
+
+class EvolvedHMM5(HMMPlayer):


drvinceknight · 2017-01-02T08:33:36Z

axelrod/tests/unit/test_ann.py

@@ -14,7 +14,7 @@ class TestEvolvedANN(TestPlayer):
    expected_classifier = {
        'memory_depth': float('inf'),
        'stochastic': False,
-        'makes_use_of': set(["length"]),


Is the number of turns no longer a feature?

Nope! Just the round number.

drvinceknight · 2017-01-02T08:37:18Z

axelrod/tests/unit/test_hmm.py

+    def test_malformed_params(self):
+        # Test a malformed table
+        t_C = [[1, 0.5], [0, 1]]
+        self.assertFalse(is_stochastic_matrix(t_C))


Re my request for a test for this function, could it be pulled out of here and tested independently? (With a True assertion as well).

drvinceknight · 2017-01-03T09:12:18Z

axelrod/data/ann_weights.csv

@@ -0,0 +1,5 @@
+# name, features, hidden_layer_size, weights...


Small suggestion: remove the # and list full header (potentially useful for other analysis?). I expect this would need to be done on the axelrod-evolver repo and not for this PR.

I think we should cross this bridge later. The number of columns isn't constant so there isn't really a proper header.

Fine to leave for later, you could have the max number of columns with headers and have NANs in the other ones? (Not suggesting that for now).

drvinceknight

This is a fantastic addition to the library @marcharper :)

My requests are mainly docstrings and more tests as well as a couple of questions.

drvinceknight · 2017-01-03T09:13:25Z

axelrod/load_data_.py

+import pkg_resources
+
+
+def load_file(filename, directory):


Docstring for completeness. Also numpy style for this and the rest?

drvinceknight · 2017-01-03T09:17:48Z

axelrod/moran.py

+
+        If the mutation_rate is 0, the population will eventually fixate on
+        exactly one player type. In this case a StopIteration exception is
+        raised and the play stops. If mutation_rate is not zero, then the


the mutation_rate

drvinceknight · 2017-01-03T09:24:50Z

axelrod/strategies/ann.py

-            hidden_layer_size
-        )
-        ANN.__init__(self, i2h, h2o, bias)
+        num_features, num_hidden, weights = nn_weights['1']


Could we change the 1 in the dataset and here to be 10? This would be for readability and corresponding to the size of the hidden layer.

drvinceknight · 2017-01-03T09:25:20Z

axelrod/strategies/ann.py

+
+    Names:
+
+     - EvolvedANN5: : Original name by Marc Harper.


drvinceknight · 2017-01-03T09:26:33Z

axelrod/strategies/ann.py

@@ -158,7 +160,8 @@ def strategy(self, opponent):

 class EvolvedANN(ANN):
    """
-    A strategy based on a pre-trained neural network.
+    A strategy based on a pre-trained neural network with 17 features and a


Would it be worth including a bullet point list of the 17 features here?

drvinceknight · 2017-01-03T09:45:39Z

axelrod/strategies/hmm.py

+        for m in [self.hmm.transitions_C, self.hmm.transitions_D]:
+            for row in m:
+                values.update(row)
+        if not values.issubset({0, 1}):


return not values.issubset({0, 1}).

(This is stylistic, I don't feel strongly about it.)

drvinceknight · 2017-01-03T09:46:40Z

axelrod/strategies/hmm.py

+
+    Names
+
+        - EvolvedHMM5


: Original name...

drvinceknight · 2017-01-03T09:48:06Z

axelrod/tests/unit/test_hmm.py

+        player = self.player([[1]], [[1]], [0], initial_state=0)
+        player.hmm.state = -1
+        player.reset()
+        self.assertFalse(player.hmm.state == -1)


Could we have a specific test for the EvolvedHMM5.

drvinceknight · 2017-01-03T09:49:35Z

axelrod/tests/unit/test_hmm.py

+        player = self.player([[1]], [[1]], [0], initial_state=0)
+        player.hmm.state = -1
+        player.reset()
+        self.assertFalse(player.hmm.state == -1)


I think assertNotEqual would be better here.

And/or even assertEqual with 0.

drvinceknight · 2017-01-03T09:56:00Z

axelrod/strategies/meta.py

@@ -19,7 +19,7 @@ class MetaPlayer(Player):
    classifier = {
        'memory_depth': float('inf'),  # Long memory
        'stochastic': True,
-        'makes_use_of': set(),
+        'makes_use_of': {'game', 'length'},


Could you walk me through this one please?

The default player set has members that use both the game and the match length.

meatballs · 2017-01-03T11:17:32Z

axelrod/load_data_.py

+import pkg_resources
+
+
+def load_file(filename, directory):


Wouldn't these be better as a set of pandas dataframes? There would be far less code and it would be quicker too.

I know it's another dependency, but we're already dependent on numpy, so the precedent has been set.

Yep! I prefer using pandas actually if the extra dependency is ok with @drvinceknight .

I'm not averse to the extra dependency. 👍

Could change the output of the results_set.summarize to be a data frame too (not for this PR, another issue :)).

Cool. I think anyone that uses anaconda or can pip install numpy should have access to rest of the scientific stack (for sure at least scipy and pandas).

I'm punting on this one since the number of columns isn't constant in all cases.

Fine by me.

I'm not entirely sure use dfs would make things simpler in this case, the data as is would need to be pivoted for the df to be advantageous or the data could be stored with rows corresponding to "genes" and columns to different strategies... Perhaps not a bad idea (but I don't think necessary for this PR).

…onent plays

marcharper · 2017-01-05T06:29:17Z

I believe that I addressed all the comments sufficiently, let me know if you agree!

drvinceknight

All looks good to me! Nice job :)

marcharper added the ready-for-review label Jan 2, 2017

drvinceknight requested changes Jan 2, 2017

View reviewed changes

drvinceknight reviewed Jan 3, 2017

View reviewed changes

drvinceknight requested changes Jan 3, 2017

View reviewed changes

meatballs reviewed Jan 3, 2017

View reviewed changes

marcharper force-pushed the ml_strategies branch from db87d10 to a1a4969 Compare January 5, 2017 05:59

marcharper added 23 commits January 4, 2017 22:12

Retrain EvolvedANN, move data to subdirectory

7b49994

Fix data file inclusion in setup.py

fe9c174

Newly trained ML strategies

658d0c6

Add Evolved ANN for the Moran process

bf41394

New LookerUps

4cee37b

Alias EvolvedLookerUp

acb7349

Generalize LookerUp to m1 plays, m2 opponent plays, and n initial opp…

32c7823

…onent plays

Additional strategies, Fix lookerup table check

5b4137b

Updated LookerUp tables

129e20c

Added Winner12 and Winner21

4cfb912

Fix several tests

8e3114f

Remove some tables from the global player list

6f15d48

Improve data loading for ML strategies

d234bc8

Update several tests and refactor EvolvedLookerUp player creation

bf7e16b

Various updates to ML strategies

963c62d

Refactor data loading

82193fb

Use more efficient cooperations counts in ANN

79813aa

Update Gambler names and tests

b1f27bf

Evolved Finite State Machines

b1e81ec

Refactor ML strategies

bad6abf

Add lookerup initial actions as parameters to __init__

0039073

Add initial sequence to lookerup data loading

f50107b

Hidden Markov Model Player

e6a974d

marcharper added 5 commits January 4, 2017 22:12

Updates to ML strategies and training data.

dd94bfd

Remove untested Moran-trained strategies for now.

1cec7f4

Fix tests

4379123

Fixes from first pass review and more tests

81a643f

Second pass on review: docs and tests

17abaaf

marcharper force-pushed the ml_strategies branch from a1a4969 to 17abaaf Compare January 5, 2017 06:12

drvinceknight approved these changes Jan 5, 2017

View reviewed changes

drvinceknight added the ready-to-merge label Jan 5, 2017

Merge branch 'master' into ml_strategies

0ae1dc0

meatballs merged commit 611bfb7 into master Jan 5, 2017

meatballs deleted the ml_strategies branch January 5, 2017 10:40

		@@ -0,0 +1,123 @@
		"""Tests for Finite State Machine Strategies."""



		class HMMPlayer(Player):
		"""Abstract base class for Hidden Markov Model players."""

		self.hmm.state = self.initial_state


		class EvolvedHMM5(HMMPlayer):

		@@ -0,0 +1,5 @@
		# name, features, hidden_layer_size, weights...

ML strategies #803

ML strategies #803

Conversation

marcharper commented Jan 2, 2017 • edited Loading

drvinceknight commented Jan 2, 2017

drvinceknight left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

drvinceknight left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

marcharper commented Jan 5, 2017

drvinceknight left a comment

Choose a reason for hiding this comment

marcharper commented Jan 2, 2017 •

edited

Loading