Up to python3.6 in containers (#895)

* Change game container to python3

* containers all up to python3, but workers not being made

* Added plain python image for testing errors

* cleanup

* fixing test imports

* pipenvs to python 3 for containers

* removed aimmo-game-creator dependancy from worker

* Remove vscode settings

* Merge branch 'development' into up_to_python3_in_containers

# Conflicts:
#	aimmo-game-creator/Pipfile.lock
#	aimmo-game/Pipfile.lock

* try to fix tests

* Use only compare priorities in priority queue

* making code python 3 compatible

* passed container template to workers (python 3 works! 😄)

* python 3 changes and introduce multistage builds to docker file for python3 tests

* fixed broken tabbing

* game creator tests now run inside containers, fixed broken code

* added new command parser for run.py to help with testing

* added method stub and comments as reminder that tests aren't done yet

* run aimmo-game tests in a job in docker

* python setup.py test in Dockerfile

* add other docker containers to test in travis

* use the int division operator instead of the float division operator

* setup all_test.py to run the tests in containers by default

* Merge remote-tracking branch 'origin/up_to_python3_in_containers' into up_to_python3_in_containers

* make integration tests run using containers

* update travis script

* fix logic in all_tests.py

* fixes for k8s minikube workflow

* WIP for making it work with python 3.6.5

* use thread pools instead of green pool, found pod creation issue

* Trying to understand why trying to list pods hangs indefinitely

* removed Greenpool from our code and docuementation

* Identified source of problem and added prints to mark it clearly

* Switched to multiprocessing library to try and resolve issue

* Revert "Switched to multiprocessing library to try and resolve issue"

This reverts commit 49fc3d5.

* updated run turn to also used threadpools (pods still hang)

* Removed Rlock from gamestate to try remove deadlock

* kubernetes client not fully compatible with python3.7

* More progress on fixing issues with python3 upgrade

* 3_in_containers: Auto stash before merge of "up_to_python3_in_containers" and "origin/up_to_python3_in_containers"

* stop infinite recursion error in urllib3 with python 3.6

* don’t explicitly fix urllib3

* remove some logging statements

* change worker to use python 3.6.5

* remove debug logger statements

* use aiohttp instead of flask, socket io not working

* Merge remote-tracking branch 'origin/up_to_python3_in_containers' into up_to_python3_in_containers

* fix async await callback issue

* Revert "3_in_containers: Auto stash before merge of "up_to_python3_in_containers" and "origin/up_to_python3_in_containers""

This reverts commit 18d88e0.

* Merge branch 'up_to_python3_in_containers' of https://github.com/ocadotechnology/aimmo into up_to_python3_in_containers

* use pytest for aimmo-game

* Fix tests and python3

* adding verbosity to python aimmo game test

* add verbose output to pytest

* only search for tests inside of the tests folder

* remove -s options for aimmo game pytest

* Merge branch 'development' into up_to_python3_in_containers

* Cleanup

* Merge branch 'up_to_python3_in_containers' of https://github.com/ocadotechnology/aimmo into up_to_python3_in_containers

* ignore tests in codeclimate

* fix a few linting issues

* pep8

* refactor of all_tests.py as pep8 broke it

* pep8++

* include docker image tests in coverage

* add build stage for coverage to docker images

* Add coverage tool to containers

* Coverage has been added to containers for testing

* add coveralls to new jobs in travis

* add coveralls to coverage test build stage in docker images

* removed eventlet from worker coverage as it is unable to install

The tests do still run

* try to get coveralls working

* coveralls still not sending coverage data

* added coveralls to travis test stages instead of docker images

* publishing container ports to try allow it to send coverage data

* fix port exposing error

* Factorise duplicate get avatar id code

* add coverage report command as it is not collecting data

* Merge remote-tracking branch 'origin/up_to_python3_in_containers' into up_to_python3_in_containers

* Remove port publish and coverage report as neither solver the problem

* Fixed factorising

* Merge branch 'up_to_python3_in_containers' of https://github.com/ocadotechnology/aimmo into up_to_python3_in_containers

* More codeclimate fixes + ignoring docstrings

* add coverage config for coverage.py to see if it captures results

* Merge remote-tracking branch 'origin/up_to_python3_in_containers' into up_to_python3_in_containers

* Fix cyclomatic complexity issue.

* Revert "add coverage config for coverage.py to see if it captures results"

This reverts commit fec2094.

* Merge branch 'up_to_python3_in_containers' of https://github.com/ocadotechnology/aimmo into up_to_python3_in_containers

* trying only solution i've managed to find for this problem

* Added environment variable for detecting if we want coverage

* changed to using coverage api instead of the command line approach

* add bash to coverage tester

* fix syntax error

* removing cleanup line so coveralls can use .coverage

* output results from coverage on travis

* Removing coverage print (coverage is being collected properly)

coverage data is being collected, coveralls seems to be submitting data correctly. however nothing shows up on coverall.io for the container builds.

* pep8++

* bump version to 0.4.1

* Merge branch 'development' into up_to_python3_in_containers

* remove duplicate from pipfile

* appends ', printed' to users return statement to get their prints

* fixed some issues with collecting user prints

* pep8++

* fixed broken test

.travis.yml

@@ -3,7 +3,9 @@ dist: trusty
 group: edge
 language: python
 python:
-  - 2.7
+  - "2.7"
+services:
+  - docker
 env:
   global:
     - secure: "TNhj8oXrtBzCkkJDA8qbDmqUggK9Kbsy8Itgi0+VmXB+3bVRornZMS38ppaFz+BVOTdL80ZvN2s/OPV106QjFv0Hx1MmWAw4kNST2QBtxFXBHRKYtr7NtBN0jr11el1fG83YdpeZYQbc5aqbJ4OPz2GpfhGfDVhVGPjMFMKXI5XbTbbl+HCEL67ywozt964LhpuuXTaX7jgYFiJUtcwkYRUaDYY3ryJvMSOx95AjKyRMNC5JlgAqbJuYsOTm1eVZtfQ1jYVvd/NuAHOMDNpZWvcIaxTuc3k4XZh4UPryuRJWfjgjIq6kua7Q6ho6W2GbDgN2b/9lIldkTR8QfLSnCNLJIg6KJZ2gmIQg7u+nZHemdugo9XkvfmKXfB/t3HChFX1HNtS4gSeIn874IynLHx3UJ1lxm7BdDbF4Jjijffj5uWDGqVj3/Myd2jdFTJCoLJXvYI7la6ouMzXW5aDFhy2UXK2A3q7aBbaD64+U1R7YPGIyvfAd7NCF11vtRvJGI/fNjO5S1EuSacrQm7CiXu0rd0L5EOSU85XNTQsWN6xxJKEcc8Hx9YLRXkmR7gK9LoEPTUwFbfVXBUvnOsZav3MOBBxzj4+eLxkx2B1vbY2Lx5yPAqyWwi3vet46NEZUIKgqK+xRYQKj6dj3OF1gx7LOcyhpyevdpTZotiEx0C4=" # SNAP_API_AUTH
@@ -34,7 +36,6 @@ jobs:
       before_install:
         - nvm install node
         - nvm use node
-        - gem install sass --version 3.3.4
       install:
         - pip install .
         - pip install coveralls
@@ -44,7 +45,37 @@ jobs:
         - node djangoBundler.js
         - popd
       script:
-        - python all_tests.py --coverage
+        - python all_tests.py --no-docker-container-tests --coverage
+      after_success:
+        - coveralls
+    - name: "aimmo-game"
+      before_install:
+        - cd aimmo-game
+        - docker build -t ocadotechnology/aimmo-game:test .
+      install:
+        - pip install coveralls
+      script:
+        - docker run -it -p 5000:5000 ocadotechnology/aimmo-game:test
+      after_success:
+        - coveralls
+    - name: "aimmo-game-creator"
+      before_install:
+        - cd aimmo-game-creator
+        - docker build -t  ocadotechnology/aimmo-game-creator:test .
+      install:
+        - pip install coveralls
+      script:
+        - docker run -it -p 5000:5000 ocadotechnology/aimmo-game-creator:test
+      after_success:
+        - coveralls
+    - name: "aimmo-game-worker"
+      before_install:
+        - cd aimmo-game-worker
+        - docker build -t ocadotechnology/aimmo-game-worker:test .
+      install:
+        - pip install coveralls
+      script:
+        - docker run -it -p 5000:5000 ocadotechnology/aimmo-game-worker:test
       after_success:
         - coveralls
     - name: "Javascript Tests"

Pipfile

@@ -9,6 +9,7 @@ pylint = "<2.0.0"
 [packages]
 "e1839a8" = {editable = true, path = "."}
 kubernetes = "*"
+eventlet = "*"
 
 [requires]
 python_version = "2.7"

aimmo-game-creator/Dockerfile

@@ -1,15 +1,28 @@
-FROM python:2-alpine
-
+FROM python:3.6 as builder
 MAINTAINER code@ocado.com
-
-RUN apk add --no-cache gcc musl-dev python-dev libffi-dev openssl-dev
-
+# RUN apk add --no-cache gcc musl-dev python-dev libffi-dev openssl-dev
 RUN pip install pipenv
-
 COPY ["Pipfile", "Pipfile.lock", "setup.py", "./"]
-
+RUN pipenv install coverage
 RUN pipenv install --system --deploy
 
+FROM python:3.6-alpine as base
+COPY --from=builder /usr/local/lib/python3.6/site-packages /usr/local/lib/python3.6/site-packages
 COPY . .
 
-CMD ["python", "./service.py"]
+
+FROM base as runner
+ENV WORKER_MANAGER=kubernetes
+CMD ["python", "./service.py", "0.0.0.0"]
+
+
+FROM base as tester
+ENV WORKER_MANAGER=kubernetes
+CMD ["python", "setup.py", "test"]
+
+FROM base as coverage_tester
+ENV WORKER_MANAGER=kubernetes
+ENV WITH_COVERAGE='True'
+RUN apk add bash
+COPY --from=builder /usr/local/bin/coverage /usr/local/bin/coverage
+CMD python setup.py test

aimmo-game-creator/Pipfile

@@ -7,6 +7,7 @@ name = "pypi"
 kubernetes = "*"
 aimmo-game-creator = {editable = true, path = "."}
 docker = "*"
+coverage = "*"
 
 [requires]
-python_version = "2.7"
+python_version = "3.6.7"

aimmo-game-creator/game_manager.py

@@ -5,8 +5,9 @@
 from abc import ABCMeta, abstractmethod
 
 import requests
-from eventlet.greenpool import GreenPool
 from eventlet.semaphore import Semaphore
+from concurrent import futures
+from concurrent.futures import ALL_COMPLETED
 import kubernetes
 import docker
 import json
@@ -60,23 +61,23 @@ class GameManager(object):
     def __init__(self, games_url):
         self._data = _GameManagerData()
         self.games_url = games_url
-        self._pool = GreenPool(size=3)
         super(GameManager, self).__init__()
 
     @abstractmethod
     def create_game(self, game_id, game_data):
         """Creates a new game"""
 
-        raise NotImplemented
+        raise NotImplementedError
 
     @abstractmethod
     def delete_game(self, game_id):
         """Deletes the given game"""
 
-        raise NotImplemented
+        raise NotImplementedError
 
-    def recreate_game(self, game_id, game_data):
+    def recreate_game(self, game_to_add):
         """Deletes and recreates the given game"""
+        game_id, game_data = game_to_add
         LOGGER.info("Deleting game {}".format(game_data["name"]))
         try:
             self.delete_game(game_id)
@@ -104,15 +105,12 @@ def update(self):
                 id: games[id]
                 for id in self._data.add_new_games(games.keys())
             }
-            LOGGER.debug("Need to add games: {}".format(games_to_add))
 
             # Add missing games
-            self._parallel_map(self.recreate_game, games_to_add.keys(), games_to_add.values())
-
+            self._parallel_map(self.recreate_game, games_to_add.items())
             # Delete extra games
             known_games = set(games.keys())
             removed_game_ids = self._data.remove_unknown_games(known_games)
-            LOGGER.debug("Removing games: {}".format(removed_game_ids))
             self._parallel_map(self.delete_game, removed_game_ids)
 
     def get_persistent_state(self, player_id):
@@ -126,8 +124,9 @@ def run(self):
             LOGGER.info("Sleeping")
             time.sleep(10)
 
-    def _parallel_map(self, func, *iterable_args):
-        list(self._pool.imap(func, *iterable_args))
+    def _parallel_map(self, func, iterable_args):
+        with futures.ThreadPoolExecutor() as executor:
+            _ = executor.map(func, iterable_args)
 
 
 class LocalGameManager(GameManager):
@@ -154,7 +153,6 @@ def setup_container_environment_variables(template, game_data):
             template['environment']['CONTAINER_TEMPLATE'] = os.environ['CONTAINER_TEMPLATE']
 
         assert (game_id not in self.games)
-        game_data = {str(k): str(v) for k, v in game_data.items()}
         port = str(6001 + int(game_id) * 1000)
         client = docker.from_env()
 
@@ -244,6 +242,7 @@ def _create_game_rc(self, game_id, environment_variables):
         environment_variables['IMAGE_SUFFIX'] = os.environ.get('IMAGE_SUFFIX', 'latest')
         environment_variables['K8S_NAMESPACE'] = K8S_NAMESPACE
         environment_variables['WORKER_MANAGER'] = 'kubernetes'
+        environment_variables['EXTERNAL_PORT'] = "5000"
 
         rc = self._make_rc(environment_variables, game_id)
         self.api.create_namespaced_replication_controller(K8S_NAMESPACE, rc)

aimmo-game-creator/setup.py

@@ -1,5 +1,17 @@
 # -*- coding: utf-8 -*-
 from setuptools import find_packages, setup
+import coverage
+import sys
+import os
+
+
+withcoverage = os.environ.get('WITH_COVERAGE')
+
+if withcoverage == 'True':
+    print("starting code coverage engine")
+    coveragedatafile = ".coverage"
+    cov = coverage.Coverage(config_file=False)
+    cov.start()
 
 
 setup(
@@ -16,3 +28,8 @@
     test_suite='tests',
     zip_safe=False,
 )
+
+if withcoverage == 'True':
+    print("saving coverage stats")
+    cov.save()
+    print("exiting program")

aimmo-game-creator/tests/test_game_manager.py

@@ -1,6 +1,5 @@
 from __future__ import absolute_import
 
-import cPickle as pickle
 import unittest
 from json import dumps
 
@@ -40,11 +39,11 @@ def _generate_response(self, num_games):
         return {
             str(i): {
                 "name": "Game {}".format(i),
-                "settings": pickle.dumps({
+                "settings": {
                     "test": i,
                     "test2": "Settings {}".format(i),
-                })
-            } for i in xrange(num_games)
+                }
+            } for i in range(num_games)
         }
 
     def __call__(self, url, request):
@@ -61,20 +60,17 @@ def test_correct_url_requested(self):
         with HTTMock(mocker):
             self.game_manager.update()
         self.assertEqual(len(mocker.urls_requested), 1)
-        self.assertRegexpMatches(mocker.urls_requested[0], "http://test/*")
+        self.assertRegex(mocker.urls_requested[0], "http://test/*")
 
     def test_games_added(self):
         mocker = RequestMock(3)
         with HTTMock(mocker):
             self.game_manager.update()
         self.assertEqual(len(self.game_manager.final_games), 3)
         self.assertEqual(len(list(self.game_manager._data.get_games())), 3)
-        for i in xrange(3):
+        for i in range(3):
             self.assertIn(str(i), self.game_manager.final_games)
-            self.assertEqual(
-                pickle.loads(str(self.game_manager.added_games[str(i)]["settings"])),
-                {"test": i, "test2": "Settings {}".format(i)}
-            )
+            self.assertEqual(self.game_manager.added_games[str(i)]["settings"], {"test": i, "test2": "Settings {}".format(i)})
             self.assertEqual(self.game_manager.added_games[str(i)]["name"], "Game {}".format(i))
 
     def test_remove_games(self):
@@ -89,7 +85,7 @@ def test_added_games_given_correct_url(self):
         mocker = RequestMock(3)
         with HTTMock(mocker):
             self.game_manager.update()
-        for i in xrange(3):
+        for i in range(3):
             self.assertEqual(
                 self.game_manager.added_games[str(i)]["GAME_API_URL"],
                 "http://test/{}/".format(i)

aimmo-game-worker/Dockerfile

@@ -1,15 +1,28 @@
-FROM python:2-alpine
-
+FROM python:3.6 as builder
 MAINTAINER code@ocado.com
 
-RUN apk add --no-cache gcc musl-dev python-dev libffi-dev openssl-dev
-
 RUN pip install pipenv
-
 COPY ["Pipfile", "Pipfile.lock", "setup.py", "./"]
-
+RUN pipenv install coverage
 RUN pipenv install --system --deploy
 
+FROM python:3.6-alpine as base
+COPY --from=builder /usr/local/lib/python3.6/site-packages /usr/local/lib/python3.6/site-packages
 COPY . .
 
-CMD python service.py 0.0.0.0 $PORT $DATA_URL
+FROM base as runner
+ENV WORKER_MANAGER=kubernetes
+ENV FLASK_ENV='development'
+RUN apk add --no-cache bash
+CMD python ./service.py 0.0.0.0 $PORT $DATA_URL
+
+FROM base as tester
+ENV WORKER_MANAGER=kubernetes
+CMD python setup.py test
+
+FROM base as coverage_tester
+ENV WORKER_MANAGER=kubernetes
+ENV WITH_COVERAGE='True'
+RUN apk add bash
+COPY --from=builder /usr/local/bin/coverage /usr/local/bin/coverage
+CMD python setup.py test

aimmo-game-worker/Pipfile

@@ -7,6 +7,7 @@ name = "pypi"
 kubernetes = "*"
 aimmo-game-worker = {editable = true, path = "."}
 restrictedpython = "==4.0.b7"
+coverage = "*"
 
 [requires]
-python_version = "2.7"
+python_version = "3.6.7"