Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix kubernetes mode #906

Merged
merged 32 commits into from
Nov 29, 2018
Merged

Fix kubernetes mode #906

merged 32 commits into from
Nov 29, 2018

Conversation

TheseusGrey
Copy link
Contributor

@TheseusGrey TheseusGrey commented Nov 23, 2018

Description

After merging both the python 3 and the c-002 branches together, the workers stopped being created in Kubernetes mode despite work fine in their respective branches. This was due to a small bug in the python 3 code and has now been fixed.


This change is Reviewable

Copy link
Member

@faucomte97 faucomte97 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed 2 of 3 files at r1, 2 of 4 files at r2, 7 of 8 files at r3.
Reviewable status: all files reviewed, 1 unresolved discussion (waiting on @dent50cent)


aimmo-game/simulation/game_runner.py, line 64 at r3 (raw file):

    async def update_simulation(self, player_id_to_serialised_actions):
        LOGGER.debug(player_id_to_serialised_actions)

Remove logger please :)

Copy link
Contributor Author

@TheseusGrey TheseusGrey left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewable status: all files reviewed, 1 unresolved discussion (waiting on @faucomte97)


aimmo-game/simulation/game_runner.py, line 64 at r3 (raw file):

Previously, faucomte97 (Florian Aucomte) wrote…

Remove logger please :)

Done.

Copy link
Contributor

@OlafSzmidt OlafSzmidt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed 2 of 3 files at r1, 2 of 4 files at r2, 7 of 8 files at r3.
Reviewable status: 10 of 11 files reviewed, 3 unresolved discussions (waiting on @OlafSzmidt, @faucomte97, and @dent50cent)


aimmo-game/tests/functional/test_damage_pickups_and_effects.py, line 27 at r3 (raw file):

        pickup_created = DamageBoostPickup(self.cell)
        self.cell.pickup = pickup_created
        loop = asyncio.get_event_loop()

Move this and every other line below into setUP instead?


aimmo-game/tests/functional/test_health_pickups_and_effects.py, line 33 at r3 (raw file):

        self.cell.pickup = HealthPickup(self.cell)

        loop = asyncio.get_event_loop()

Repetition in setUP for event loops

Copy link
Contributor Author

@TheseusGrey TheseusGrey left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewable status: 10 of 11 files reviewed, 3 unresolved discussions (waiting on @OlafSzmidt and @faucomte97)


aimmo-game/tests/functional/test_damage_pickups_and_effects.py, line 27 at r3 (raw file):

Previously, OlafSzmidt (Olaf Szmidt) wrote…

Move this and every other line below into setUP instead?

Done.


aimmo-game/tests/functional/test_health_pickups_and_effects.py, line 33 at r3 (raw file):

Previously, OlafSzmidt (Olaf Szmidt) wrote…

Repetition in setUP for event loops

Done.

Copy link
Member

@faucomte97 faucomte97 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:lgtm:

Reviewed 1 of 3 files at r4.
Reviewable status: 9 of 11 files reviewed, 2 unresolved discussions (waiting on @OlafSzmidt and @faucomte97)

Copy link
Contributor

@OlafSzmidt OlafSzmidt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:lgtm:

Reviewed 3 of 3 files at r4.
Reviewable status: :shipit: complete! all files reviewed, all discussions resolved

Copy link
Contributor

@mrniket mrniket left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewable status: 10 of 24 files reviewed, 1 unresolved discussion (waiting on @OlafSzmidt, @faucomte97, and @mrniket)

a discussion (no related file):
WIP (feel free to resolve this whenever it's ready for review again)


Copy link
Contributor Author

@TheseusGrey TheseusGrey left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewable status: 10 of 28 files reviewed, 1 unresolved discussion (waiting on @OlafSzmidt, @faucomte97, and @mrniket)

a discussion (no related file):

Previously, mrniket (Niket Shah) wrote…

WIP (feel free to resolve this whenever it's ready for review again)

Done.


Copy link
Contributor

@mrniket mrniket left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed 10 of 14 files at r5, 7 of 8 files at r6.
Reviewable status: 27 of 28 files reviewed, 7 unresolved discussions (waiting on @dent50cent)


.travis.yml, line 39 at r6 (raw file):

        apt:
          packages:
            - docker-ce

does this make it work correctly with buildkit or do we need to set the environment variable as well?


.travis.yml, line 93 at r6 (raw file):

        apt:
          packages:
            - docker-ce

I don't think the Javascript tests need the docker addon


.travis.yml, line 129 at r6 (raw file):

        apt:
          packages:
            - docker-ce

the deploy step shouldn't need the docker add-on either


aimmo-game-creator/hooks/build, line 2 at r6 (raw file):

#!/bin/bash
docker build --target runner -t $IMAGE_NAME .

new line


aimmo-game-worker/hooks/build, line 2 at r6 (raw file):

#!/bin/bash
docker build --target runner -t $IMAGE_NAME .

new line


aimmo-game/hooks/build, line 3 at r6 (raw file):

#!/bin/bash

docker build --target runner -t $IMAGE_NAME .

new line


aimmo_runner/docker_scripts.py, line 19 at r6 (raw file):

def create_docker_client(raw_env_settings):

have we tested minikube with build kit as well?

Copy link
Contributor

@mrniket mrniket left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed 1 of 8 files at r6.
Reviewable status: all files reviewed, 7 unresolved discussions (waiting on @dent50cent)

Copy link
Contributor Author

@TheseusGrey TheseusGrey left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewable status: all files reviewed, 7 unresolved discussions (waiting on @mrniket)


.travis.yml, line 39 at r6 (raw file):

Previously, mrniket (Niket Shah) wrote…

does this make it work correctly with buildkit or do we need to set the environment variable as well?

They aren't using buildkit on Travis (see latest aimmo worker travis tests) . However it would still run the tests correctly despite that, the main use for buildkit is when we need to ignore the test stage. but i can set an environment variable to see if that will enable it


.travis.yml, line 93 at r6 (raw file):

Previously, mrniket (Niket Shah) wrote…

I don't think the Javascript tests need the docker addon

Removed it.


.travis.yml, line 129 at r6 (raw file):

Previously, mrniket (Niket Shah) wrote…

the deploy step shouldn't need the docker add-on either

Removed it.


aimmo-game-creator/hooks/build, line 2 at r6 (raw file):

Previously, mrniket (Niket Shah) wrote…

new line

Done.


aimmo-game-worker/hooks/build, line 2 at r6 (raw file):

Previously, mrniket (Niket Shah) wrote…

new line

Done.


aimmo-game/hooks/build, line 3 at r6 (raw file):

Previously, mrniket (Niket Shah) wrote…

new line

Done.


aimmo_runner/docker_scripts.py, line 19 at r6 (raw file):

Previously, mrniket (Niket Shah) wrote…

have we tested minikube with build kit as well?

No, I updated the docker scripts so that it adds the buildkit env variable when creating the client for building the images for Minikube. It should now use buildkit in both cases.

Copy link
Contributor

@mrniket mrniket left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed 5 of 5 files at r7.
Reviewable status: all files reviewed, 3 unresolved discussions (waiting on @dent50cent)


.travis.yml, line 39 at r6 (raw file):

Previously, dent50cent wrote…

They aren't using buildkit on Travis (see latest aimmo worker travis tests) . However it would still run the tests correctly despite that, the main use for buildkit is when we need to ignore the test stage. but i can set an environment variable to see if that will enable it

If it's as easy as setting an environment variable I would say it's worth a try, as it could lead to confusion later on


aimmo_runner/docker_scripts.py, line 19 at r6 (raw file):

Previously, dent50cent wrote…

No, I updated the docker scripts so that it adds the buildkit env variable when creating the client for building the images for Minikube. It should now use buildkit in both cases.

if we are setting it for both cases of the if statement then we can just set the buildkit env variable after the if statement :)

also does it work with minikube with this setting?


aimmo_runner/docker_scripts.py, line 51 at r7 (raw file):

        client = create_docker_client(raw_env_settings)
    else:
        client = create_docker_client('driver does not support')

can we use a boolean here instead? maybe it makes sense to calc raw_env_settings in the create_docker_client method?

Copy link
Contributor Author

@TheseusGrey TheseusGrey left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewable status: 26 of 28 files reviewed, 3 unresolved discussions (waiting on @mrniket)


.travis.yml, line 39 at r6 (raw file):

Previously, mrniket (Niket Shah) wrote…

If it's as easy as setting an environment variable I would say it's worth a try, as it could lead to confusion later on

i added the env variable, however i it turns out travis is using docker 17.09 and so it doesn't support buildkit, i think it might be because of where we install it from?


aimmo_runner/docker_scripts.py, line 19 at r6 (raw file):

Previously, mrniket (Niket Shah) wrote…

if we are setting it for both cases of the if statement then we can just set the buildkit env variable after the if statement :)

also does it work with minikube with this setting?

I tested it in Minikube and everything seemed to work fine.


aimmo_runner/docker_scripts.py, line 51 at r7 (raw file):

Previously, mrniket (Niket Shah) wrote…

can we use a boolean here instead? maybe it makes sense to calc raw_env_settings in the create_docker_client method?

Done.

Copy link
Contributor Author

@TheseusGrey TheseusGrey left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewable status: 26 of 28 files reviewed, 3 unresolved discussions (waiting on @mrniket)


.travis.yml, line 39 at r6 (raw file):

Previously, dent50cent wrote…

i added the env variable, however i it turns out travis is using docker 17.09 and so it doesn't support buildkit, i think it might be because of where we install it from?

follow-up, I removed the docker-ce addon section in the travis.yml, and then checked the travis logs for building the image and it built using buildkit, it looks like the apt addon gets docker 17.09 but we already had 18.09 when we specify docker in the sevices

Copy link
Contributor

@mrniket mrniket left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed 2 of 2 files at r8.
Reviewable status: :shipit: complete! all files reviewed, all discussions resolved


.travis.yml, line 39 at r6 (raw file):

Previously, dent50cent wrote…

follow-up, I removed the docker-ce addon section in the travis.yml, and then checked the travis logs for building the image and it built using buildkit, it looks like the apt addon gets docker 17.09 but we already had 18.09 when we specify docker in the sevices

awesome!

Copy link
Contributor

@mrniket mrniket left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:lgtm:

Reviewable status: :shipit: complete! all files reviewed, all discussions resolved

@TheseusGrey TheseusGrey merged commit 54e2596 into development Nov 29, 2018
@TheseusGrey TheseusGrey deleted the fix_kubernetes_mode branch November 29, 2018 15:10
faucomte97 pushed a commit that referenced this pull request Dec 21, 2018
* trying to find where the avatar admin display is breaking (#851)

* trying to find where it's breaking

* restart codeclimate - blank commit

* Setup Versioning (#849)

* test version 0.1.0

* Merge branch 'master' into versions

* try wihtout multiple deployment provider setup

* Remove if

* add job name to deploy stage

* Github versioning with releases

* Merge branch 'master' into versions

* test deploy step

* remove file option

* Make sure travis releases are prereleases

* Test deploy to PyPi

* specify branch for PyPi

* again

* only run deploy step once

* Merge branch 'master' into versions

* Merge branch 'development' into versions

* test release stage

* replace before_deploy with script

* remove extra indentation

* close second if statement

* Make bash script more resilient

* fix bash script syntax error

* Try using export

(commentted out python tests to make travis build faster for testing)

* Comment out job name

* comment out the whole test job

* move tag script into it’s own file

* test duplicate providers with different branch conditions

* test PyPi deploy with tags: true

* try bash script condition for PyPi

* fix bash script error

* Test deploy on tags

* try seeing if setting Travis tag gives use the corrent pypi number

* fix bash export statement

* push git tag before release

* Target correct commit on github releases

* bump up version number

* get ready for PR, switch branch from versions to development

* Update contributing guidelines

* Update versioning pipeline details

* Merge development into versions

* C-001 solution (#848)

* get docker for mac working

change ingresses namespace

change get_game_url_base_path

use docker for desktop ip

* begun creation of roles

* C-001 fixed

Attempted to replicate the security flaw C-001 as per the test report after implementing RBAC. The console shows a 503 upon attempting to replicate the issue, the user's consoles shows a 111 (connection refused). This leads to the conclusion the the issue has been resolved, however further testing may be needed.

* Moved role/account yamls to their own folder

Put them all in an rbac folder to keep the in one easy to find place

* Added a comment to describe the process

added a string to describe how to set up docker-desktop with extensions, and make use of RBAC. These steps may need to be automated, or included in the setup process. Minor issues still exist however.

* added reference to source of information

just so other's can find it easily if needed.

* Update worker account

* updated game_creator_role and minor changes

moved the game creator to the nginx namespace and updated it's role to reflect this, the ClusterRoleBinding gives the permissions of the game creator's permissions scope of the whole cluster (hence why it was moved out of reach of the worker pods, just incase there is a way for workers to gain access outside of their pod)

* Fixed game creator rc yaml not applying role binding

* change role binding namespace

* updated version

* Delete nginx-ingress folder

* Revert "get docker for mac working"

This reverts commit 017620a.

# Conflicts:
#	aimmo_runner/minikube.py

* Automate applying the accounts and roles

* fix travis build fail attempt

* Testing build fail

* More travis build tests

* travis build fix attempt++

* Update travis kubernetes version

* change kubernetes to ealier patch version

Kubernetes needs to be version 1.9.4 not 1.9.6

* Tests

* various changes

* Merge branch 'development' into c-001

* Resolve merge conflicts

* Merge remote-tracking branch 'origin/c-001' into c-001

* Fix issues created from merge conflicts

* Replacing versioner in setup.py

* minor change

* change to restart code climate

* pep8

* Docker for local mode (#859)

* Reduce image sizes for docker images

* bump version

* Stop turn slowdown (#857)

* Identified a potential point of slowdown

Turn processing is already done concurrently, but data fetching for all the workers was not, so i have changed this in the hopes it will speed up turn times and the number of users increases.

* Rethinking approach

difficult to implement concurrent solution to the code in question, rethinking approach.

* Added comments to where things need changing

added a couple of comments to explain what need's to be done in various places that i've identified that could slow down turns.

* Turn times had been sped up

Solution has been to create a timed process populated with daemon threads, this forces the main source of the slowdown into a 2 second window. However this has highlighted a big issue, the requests take more than 2 seconds a significant proportion of the time and as a result many actions become wait actions, also there needs to be a degree of garbage collection to prevent the creation of a potentially server ending amount of processes

* Cleanup

game runs turns at consistent 3 seconds locally even with 10 people, which in the past seem to cause the game to crash. Avatars seem to be able to respond in the 2 second limit, however we may want to increase this duration to make it more reliable.

* Merge branch 'development' into stop_turn_slowdown

* update version

* Minor changes

* Grammar changes

* reviewable changes

* Bump version number

* Reviewable changes++

* Merge branch 'development' into stop_turn_slowdown

* pep8

* Merge remote-tracking branch 'origin/stop_turn_slowdown' into stop_turn_slowdown

* Merge master into development (#861)

* Update minikube status check (#846)

* Update logic to check whether minikube is already running.

* Updated except to check for CalledProcessError.

* Merge branch 'master' into minikube-run-start

* Merge branch 'master' into development

* Fix 804, encoding user code (#818)

* Fix #804 encoding issue

* Minor refactoring of boolean should_send_logs logic


Negate bool

* Change incorrect boolean logic

* Update minikube status check (#846)

* Update logic to check whether minikube is already running.

* Updated except to check for CalledProcessError.

* Merge branch 'master' into minikube-run-start

* added PR template (issue templates existed already). (#830)

* added PR template (issue templates existed already).

* Minor changes

* added a version checkbox to the checklist in the PR template

* minor changes

* Merge branch 'development' into add_git_templates

* Bumb version number

* Merge branch 'development' into add_git_templates

* Merge branch 'development' into add_git_templates

* bump version to 0.2.3

* grammar fixes

* Adding game name to avatar display admin list (#877)

* Create prometheus operator (#872)

* Begun creation of prometheus deployment

Created the role and namespace that the prometheus operator will sit in, need to do more research on the structure of these things to get setup prometheus within kubernetes.

* created prometheus deployment

we have a prometheus deployment that monitors the cluster, and can be queried for information on things like pod cpu usage

* Merge branch 'development' into create_metrics

* Fixing prometheus client not importing

* Need to solution to beginning exposing new metrics

* Added prometheus example and set up kubernetes deployment yamls

* Created prometheus operator and custom resources

After further investigation i found that what i need to be using is a prometheus operator (this is a deployment), see link here: https://github.com/coreos/prometheus-operator. This operator watches the service monitors created via a prometheus deployment (both are custom resource definitions) and passes them on to the prometheus server (which was deployed by the operator). 

Prometheus functions within kubernetes by scraping from http endpoints (services), defined by various config maps, applying rules to the scraped data, which can then be passed to a web UI such as Grafana or the prometheus query manager.

* bump version to 0.2.3

* Merge branch 'development' into create_prometheus_operator

* rebump version

* removed comment

* removed unused imports

* remove used imports

* removed broken line

* Merge branch 'development' into create_prometheus_operator

* bump version to 0.2.5

* Docker for local mode (#875)

* Reduce image sizes for docker images

* bump version

* Merge branch 'development' into docker_for_local_mode

# Conflicts:
#	version.txt

* working docker in local mode!

* expose external port to aimmo-game

* detach aimmo-game-worker running from aimmo-game process

* delete previously running containers before creating new ones

* make docker container naming consistent with k8s

* make docker host os agnostic

* remove unnecessary log statement

* only re install deps if they have changed

* change hosts

* add Portainer instructions to usage, and small intro paragraph

* try to get host.docker.internal working on both linux and mac

* Merge branch 'docker_for_local_mode' of https://github.com/ocadotechnology/aimmo into docker_for_local_mode

* get port from env variable

* os dependent host

* use a template for container kwargs

* pass CONTAINER_TEMPLATE into worker

* provide default for LOCALHOST_IP

* provide defaults for environment variables to do with docker

* fix tests by mocking docker

* Adding docker install to travis to try and fix tests.

* Remove logs and adjust blank lines.

* Added settings to make Docker for local work for Ubuntu machines.

* Merge branch 'development' into docker_for_local_mode

* Remove extra whitespaces

* Merge branch 'docker_for_local_mode' of https://github.com/ocadotechnology/aimmo into docker_for_local_mode

* Merge branch 'development' of https://github.com/ocadotechnology/aimmo into docker_for_local_mode

* Bumped up version by minor change.

* Move docker logic into a separate file

Since docker setup is not related to minikube anymore

* inject env variables for aimmo-game

* Merge branch 'development' into docker_for_local_mode

# Conflicts:
#	version.txt

* remove newline in version file

* remove docker entrypoint scripts

* change experimental settings back to original values

* add time.sleep back to run script

* Code review changes

* Docstring changes and delete stopped aimmo containers as well

* change port setting back to original value

* fix docker version

* add missing quote

* Don’t fix versions in Pipfile as they are in the Pipfile.lock

* Update aimmo-game/Pipfile.lock

* Conform to pep8

* remove bash and curl from dockerfile

* Merge branch 'development' into docker_for_local_mode

# Conflicts:
#	aimmo-game/Pipfile
#	aimmo-game/Pipfile.lock
#	aimmo-game/service.py
#	version.txt

* Update docker command to fix port issue (#879)

* Changed default to port environment variable.

* Fixed port setup for Linux (#880)

* Separated port setup for docker and kubernetes modes.

* Removed unnecessary port initialiser.

* C-002 solution implementation (#896)

* Added RestrictedPython and restricted globals to compile avatar code in restricted manner.

* No longer updates code at every turn

* Moved restricted code to separate file.

* Revert "Moved restricted code to separate file."

This reverts commit e1b6475.

* Cleaned up comments and logs.

* Removed additional unused imports.

* Added actions to globals and cleaned up other globals

* Added regex to clean logs of unwanted warnings.

* Added Avatar in module dict straight away to fix delayed turn issue.

* Enables use of += and other similar operations

* Removed broken regex

* Fixed tests

* Removed imports from avatar examples

* Added tests

* Try fixing tests on travis

* Try fixing tests on travis part 2

* Re-update pipfile package name

* Check to see if RestrictedPython in setup.py causes an issue

* Update class definition

* Added builtins import for future Python 3

* CodeClimate

* Revert "Enables use of += and other similar operations"

* Made tests format consistent

* Re-added RestrictedPython to dependencies

* Bumped up version

* Removed tests from CodeClimate exclude and renamed avatar file

* Up to python3.6 in containers (#895)

* Change game container to python3

* containers all up to python3, but workers not being made

* Added plain python image for testing errors

* cleanup

* fixing test imports

* pipenvs to python 3 for containers

* removed aimmo-game-creator dependancy from worker

* Remove vscode settings

* Merge branch 'development' into up_to_python3_in_containers

# Conflicts:
#	aimmo-game-creator/Pipfile.lock
#	aimmo-game/Pipfile.lock

* try to fix tests

* Use only compare priorities in priority queue

* making code python 3 compatible

* passed container template to workers (python 3 works! 😄)

* python 3 changes and introduce multistage builds to docker file for python3 tests

* fixed broken tabbing

* game creator tests now run inside containers, fixed broken code

* added new command parser for run.py to help with testing

* added method stub and comments as reminder that tests aren't done yet

* run aimmo-game tests in a job in docker

* python setup.py test in Dockerfile

* add other docker containers to test in travis

* use the int division operator instead of the float division operator

* setup all_test.py to run the tests in containers by default

* Merge remote-tracking branch 'origin/up_to_python3_in_containers' into up_to_python3_in_containers

* make integration tests run using containers

* update travis script

* fix logic in all_tests.py

* fixes for k8s minikube workflow

* WIP for making it work with python 3.6.5

* use thread pools instead of green pool, found pod creation issue

* Trying to understand why trying to list pods hangs indefinitely

* removed Greenpool from our code and docuementation

* Identified source of problem and added prints to mark it clearly

* Switched to multiprocessing library to try and resolve issue

* Revert "Switched to multiprocessing library to try and resolve issue"

This reverts commit 49fc3d5.

* updated run turn to also used threadpools (pods still hang)

* Removed Rlock from gamestate to try remove deadlock

* kubernetes client not fully compatible with python3.7

* More progress on fixing issues with python3 upgrade

* 3_in_containers: Auto stash before merge of "up_to_python3_in_containers" and "origin/up_to_python3_in_containers"

* stop infinite recursion error in urllib3 with python 3.6

* don’t explicitly fix urllib3

* remove some logging statements

* change worker to use python 3.6.5

* remove debug logger statements

* use aiohttp instead of flask, socket io not working

* Merge remote-tracking branch 'origin/up_to_python3_in_containers' into up_to_python3_in_containers

* fix async await callback issue

* Revert "3_in_containers: Auto stash before merge of "up_to_python3_in_containers" and "origin/up_to_python3_in_containers""

This reverts commit 18d88e0.

* Merge branch 'up_to_python3_in_containers' of https://github.com/ocadotechnology/aimmo into up_to_python3_in_containers

* use pytest for aimmo-game

* Fix tests and python3

* adding verbosity to python aimmo game test

* add verbose output to pytest

* only search for tests inside of the tests folder

* remove -s options for aimmo game pytest

* Merge branch 'development' into up_to_python3_in_containers

* Cleanup

* Merge branch 'up_to_python3_in_containers' of https://github.com/ocadotechnology/aimmo into up_to_python3_in_containers

* ignore tests in codeclimate

* fix a few linting issues

* pep8

* refactor of all_tests.py as pep8 broke it

* pep8++

* include docker image tests in coverage

* add build stage for coverage to docker images

* Add coverage tool to containers

* Coverage has been added to containers for testing

* add coveralls to new jobs in travis

* add coveralls to coverage test build stage in docker images

* removed eventlet from worker coverage as it is unable to install

The tests do still run

* try to get coveralls working

* coveralls still not sending coverage data

* added coveralls to travis test stages instead of docker images

* publishing container ports to try allow it to send coverage data

* fix port exposing error

* Factorise duplicate get avatar id code

* add coverage report command as it is not collecting data

* Merge remote-tracking branch 'origin/up_to_python3_in_containers' into up_to_python3_in_containers

* Remove port publish and coverage report as neither solver the problem

* Fixed factorising

* Merge branch 'up_to_python3_in_containers' of https://github.com/ocadotechnology/aimmo into up_to_python3_in_containers

* More codeclimate fixes + ignoring docstrings

* add coverage config for coverage.py to see if it captures results

* Merge remote-tracking branch 'origin/up_to_python3_in_containers' into up_to_python3_in_containers

* Fix cyclomatic complexity issue.

* Revert "add coverage config for coverage.py to see if it captures results"

This reverts commit fec2094.

* Merge branch 'up_to_python3_in_containers' of https://github.com/ocadotechnology/aimmo into up_to_python3_in_containers

* trying only solution i've managed to find for this problem

* Added environment variable for detecting if we want coverage

* changed to using coverage api instead of the command line approach

* add bash to coverage tester

* fix syntax error

* removing cleanup line so coveralls can use .coverage

* output results from coverage on travis

* Removing coverage print (coverage is being collected properly)

coverage data is being collected, coveralls seems to be submitting data correctly. however nothing shows up on coverall.io for the container builds.

* pep8++

* bump version to 0.4.1

* Merge branch 'development' into up_to_python3_in_containers

* remove duplicate from pipfile

* appends ', printed' to users return statement to get their prints

* fixed some issues with collecting user prints

* pep8++

* fixed broken test

* Fix kubernetes mode (#906)

* fixed default port not being given in dockerfile for games

* Changed simulation runner to make use of asyncio

* changed parallel_map to async_map

* Waiting for workers pt. 1

* aimmo now works in kubernetes mode again

* aimmo now works in kubernetes mode again (tests fails though)

* Merge remote-tracking branch 'origin/fix_kubernetes_mode' into fix_kubernetes_mode

* fixed broken tests

* pep8

* removed temporary logging

* moved get loop into setUp for tests

* bump version++

* make runner last build stage

* remove broken coverage tool

* removed broken coverage tool++

* change docker build target in tester

* docker now uses buildkit (requires docker 18.09)

* create build hook for docker cloud

* fix syntax error

* add image tag to build hook

* fix image name on build hook

* add build hook for worker and game-creator

* remove pwd from build hooks

* remove broken coverage tool from travis.yml

* change env variable to be str not int

* force setup.py version

* Revert "force setup.py version"

This reverts commit 306bee8.

* review changes

* add dockerbuildkit env variable to travis

* change the way docker is installed for travis test

* removed docker addon from travis.yml (installing older version)

* changed logic in docker scripts

* Add .vscode to the .gitignore (#908)

* .vscode folders no longer show up on git

* Merge branch 'development' into add_editor_settings_to_gitignore

* PrintCollector improvements (#911)

* Implement custom print collector

* added doc strings and minor cleanup

* catch syntax warning incorrect syntax warning

* tests

* bump version

* pep8

* pep8++

* Merge branch 'development' into PrintCollector_improvements

* minor changes + remove clean log function as no longer needed

* improvements to log output

* separated tests and added new test

* refactored log printing

* Upgrade react unity webgl to 7.x (#910)

* alphabetize npm packages

* upgrade react unity webgl to 7.0.7

* update code to work with react unity webgl 7.x

* update yarn.lock

* write more tests for GameView

* add docker to top Pipfile

* Merge branch 'development' into upgrade_react_unity_webgl

# Conflicts:
#	aimmo-game/Pipfile.lock

* bump version

* Merge branch 'development' into upgrade_react_unity_webgl

# Conflicts:
#	version.txt

* Update react and babel (#916)

* Upgrade react to 16.6.3 and related packages

* update react and babel

* upgrade babel deps

* Mock out unityContent

* bump version

* update code for babel pollyfill update

* worker no longer sends extra request to game for data (#914)

* worker no longer sends extra request to game for data

* pep8

* pep8++

* fix tests

* Merge branch 'development' into remove_redundant_request

* bump version

* move worker's code into worker object

* minor change to worker

* Merge branch 'development' into remove_redundant_request

* fix broken tests

* minor change

* Created template tag to find user's playable game (#919)

* Added template tag for getting a user's playable games.

* Add a loading spinner for the game until the first game state is received (#921)

* update rxjs patch version

* update yarn lock

* fix typo in Game/types.js

* Add loading spinner for when the game is loading

* Merge branch 'development' into better_loading_state_for_game_view

* Version bump

* renderLoadingBar renamed to renderLoadingScreen

* Optimise workers (#920)

* Added timeout to run_turn, and begun optimisation of worker

* update how output is captured from user's code

* Begun migration from Flask to aiohttp

* remove comments

* fix string append

* fix async_map

* timing startup times

* Merge branch 'development' into optimise_workers

* remove timeit's and comments+logging

* Merge branch 'development' into optimise_workers

* remove flask and requests

* removed addition to output log

* added doc string

* pep8

* pep8++

* pep8++

* Fixed doc string inconsistency

* minor changes

* removed unnecessary print

* removed potentially confusing comment

* update frontend deps

* print host and port to logs on startup

* bump version

* Merge branch 'development' into optimise_workers

* Update AIMMO UI Colors (#924)

* Update AIMMO UI Colors

* update snapshot and version number

* Merge branch 'development' into run_code_loading_button

# Conflicts:
#	version.txt

* Run code button states (#927)

* add updating state

* remove debug console log

* Merge branch 'development' into run_code_loading

* circular progress inherits it’s coor from the parent button

* disable button when updating

* implementation of runcodebutton states (tests failing)

* fix proptypes not working

* fix/add tests

* add test case

* add logic for buttons error state (read desc.)

This would occur when the frontend cannot communicate with the server, the way this is usually handled is via a timeout. I've added an epic & reducer that will set a timeoutStatus to true after 20 seconds of inactivity from the server (there's an error though). The game should "un-timeout" when it receives a request, this still needs to be done, along with the visuals of the button and the popup message.

* moved timeoutStatus into game

* timeout state changes to true when no response received after x seconds

* Simplified approach to just detect when gameState is not being received

* cleanup

* map timeoutStatus state to button prop

* cleanup++

* tests for error state

* bump version

* fix failing tests

* Merge branch 'master' of https://github.com/ocadotechnology/aimmo into development

# Conflicts:
#	.travis.yml
#	Pipfile.lock
#	aimmo-game-creator/Dockerfile
#	aimmo-game-worker/Dockerfile
#	aimmo-game/Dockerfile
#	aimmo-game/simulation/worker_managers/kubernetes_worker_manager.py
#	aimmo-game/simulation/worker_managers/worker_manager.py
#	aimmo/admin.py
#	aimmo_runner/minikube.py
#	game_frontend/yarn.lock
#	rbac/game_creator_role.yaml
#	version.txt

* Merge master into development (#931)

* Update minikube status check (#846)

* Update logic to check whether minikube is already running.

* Updated except to check for CalledProcessError.

* Merge branch 'master' into minikube-run-start

* Merge development into master (#862)

* trying to find where the avatar admin display is breaking (#851)

* trying to find where it's breaking

* restart codeclimate - blank commit

* Setup Versioning (#849)

* test version 0.1.0

* Merge branch 'master' into versions

* try wihtout multiple deployment provider setup

* Remove if

* add job name to deploy stage

* Github versioning with releases

* Merge branch 'master' into versions

* test deploy step

* remove file option

* Make sure travis releases are prereleases

* Test deploy to PyPi

* specify branch for PyPi

* again

* only run deploy step once

* Merge branch 'master' into versions

* Merge branch 'development' into versions

* test release stage

* replace before_deploy with script

* remove extra indentation

* close second if statement

* Make bash script more resilient

* fix bash script syntax error

* Try using export

(commentted out python tests to make travis build faster for testing)

* Comment out job name

* comment out the whole test job

* move tag script into it’s own file

* test duplicate providers with different branch conditions

* test PyPi deploy with tags: true

* try bash script condition for PyPi

* fix bash script error

* Test deploy on tags

* try seeing if setting Travis tag gives use the corrent pypi number

* fix bash export statement

* push git tag before release

* Target correct commit on github releases

* bump up version number

* get ready for PR, switch branch from versions to development

* Update contributing guidelines

* Update versioning pipeline details

* Merge development into versions

* C-001 solution (#848)

* get docker for mac working

change ingresses namespace

change get_game_url_base_path

use docker for desktop ip

* begun creation of roles

* C-001 fixed

Attempted to replicate the security flaw C-001 as per the test report after implementing RBAC. The console shows a 503 upon attempting to replicate the issue, the user's consoles shows a 111 (connection refused). This leads to the conclusion the the issue has been resolved, however further testing may be needed.

* Moved role/account yamls to their own folder

Put them all in an rbac folder to keep the in one easy to find place

* Added a comment to describe the process

added a string to describe how to set up docker-desktop with extensions, and make use of RBAC. These steps may need to be automated, or included in the setup process. Minor issues still exist however.

* added reference to source of information

just so other's can find it easily if needed.

* Update worker account

* updated game_creator_role and minor changes

moved the game creator to the nginx namespace and updated it's role to reflect this, the ClusterRoleBinding gives the permissions of the game creator's permissions scope of the whole cluster (hence why it was moved out of reach of the worker pods, just incase there is a way for workers to gain access outside of their pod)

* Fixed game creator rc yaml not applying role binding

* change role binding namespace

* updated version

* Delete nginx-ingress folder

* Revert "get docker for mac working"

This reverts commit 017620a.

# Conflicts:
#	aimmo_runner/minikube.py

* Automate applying the accounts and roles

* fix travis build fail attempt

* Testing build fail

* More travis build tests

* travis build fix attempt++

* Update travis kubernetes version

* change kubernetes to ealier patch version

Kubernetes needs to be version 1.9.4 not 1.9.6

* Tests

* various changes

* Merge branch 'development' into c-001

* Resolve merge conflicts

* Merge remote-tracking branch 'origin/c-001' into c-001

* Fix issues created from merge conflicts

* Replacing versioner in setup.py

* minor change

* change to restart code climate

* pep8

* Docker for local mode (#859)

* Reduce image sizes for docker images

* bump version

* Stop turn slowdown (#857)

* Identified a potential point of slowdown

Turn processing is already done concurrently, but data fetching for all the workers was not, so i have changed this in the hopes it will speed up turn times and the number of users increases.

* Rethinking approach

difficult to implement concurrent solution to the code in question, rethinking approach.

* Added comments to where things need changing

added a couple of comments to explain what need's to be done in various places that i've identified that could slow down turns.

* Turn times had been sped up

Solution has been to create a timed process populated with daemon threads, this forces the main source of the slowdown into a 2 second window. However this has highlighted a big issue, the requests take more than 2 seconds a significant proportion of the time and as a result many actions become wait actions, also there needs to be a degree of garbage collection to prevent the creation of a potentially server ending amount of processes

* Cleanup

game runs turns at consistent 3 seconds locally even with 10 people, which in the past seem to cause the game to crash. Avatars seem to be able to respond in the 2 second limit, however we may want to increase this duration to make it more reliable.

* Merge branch 'development' into stop_turn_slowdown

* update version

* Minor changes

* Grammar changes

* reviewable changes

* Bump version number

* Reviewable changes++

* Merge branch 'development' into stop_turn_slowdown

* pep8

* Merge remote-tracking branch 'origin/stop_turn_slowdown' into stop_turn_slowdown

* Merge master into development (#861)

* Update minikube status check (#846)

* Update logic to check whether minikube is already running.

* Updated except to check for CalledProcessError.

* Merge branch 'master' into minikube-run-start

* Merge branch 'master' into development

* Merge branch 'master' of https://github.com/ocadotechnology/aimmo into development

# Conflicts:
#	.travis.yml
#	Pipfile.lock
#	aimmo-game-creator/Dockerfile
#	aimmo-game-worker/Dockerfile
#	aimmo-game/Dockerfile
#	aimmo-game/simulation/worker_managers/kubernetes_worker_manager.py
#	aimmo-game/simulation/worker_managers/worker_manager.py
#	aimmo/admin.py
#	aimmo_runner/minikube.py
#	game_frontend/yarn.lock
#	rbac/game_creator_role.yaml
#	version.txt

* Merge branch 'development' into update-development

* merge master into development (#933)

* Update minikube status check (#846)

* Update logic to check whether minikube is already running.

* Updated except to check for CalledProcessError.

* Merge branch 'master' into minikube-run-start

* Merge development into master (#862)

* trying to find where the avatar admin display is breaking (#851)

* trying to find where it's breaking

* restart codeclimate - blank commit

* Setup Versioning (#849)

* test version 0.1.0

* Merge branch 'master' into versions

* try wihtout multiple deployment provider setup

* Remove if

* add job name to deploy stage

* Github versioning with releases

* Merge branch 'master' into versions

* test deploy step

* remove file option

* Make sure travis releases are prereleases

* Test deploy to PyPi

* specify branch for PyPi

* again

* only run deploy step once

* Merge branch 'master' into versions

* Merge branch 'development' into versions

* test release stage

* replace before_deploy with script

* remove extra indentation

* close second if statement

* Make bash script more resilient

* fix bash script syntax error

* Try using export

(commentted out python tests to make travis build faster for testing)

* Comment out job name

* comment out the whole test job

* move tag script into it’s own file

* test duplicate providers with different branch conditions

* test PyPi deploy with tags: true

* try bash script condition for PyPi

* fix bash script error

* Test deploy on tags

* try seeing if setting Travis tag gives use the corrent pypi number

* fix bash export statement

* push git tag before release

* Target correct commit on github releases

* bump up version number

* get ready for PR, switch branch from versions to development

* Update contributing guidelines

* Update versioning pipeline details

* Merge development into versions

* C-001 solution (#848)

* get docker for mac working

change ingresses namespace

change get_game_url_base_path

use docker for desktop ip

* begun creation of roles

* C-001 fixed

Attempted to replicate the security flaw C-001 as per the test report after implementing RBAC. The console shows a 503 upon attempting to replicate the issue, the user's consoles shows a 111 (connection refused). This leads to the conclusion the the issue has been resolved, however further testing may be needed.

* Moved role/account yamls to their own folder

Put them all in an rbac folder to keep the in one easy to find place

* Added a comment to describe the process

added a string to describe how to set up docker-desktop with extensions, and make use of RBAC. These steps may need to be automated, or included in the setup process. Minor issues still exist however.

* added reference to source of information

just so other's can find it easily if needed.

* Update worker account

* updated game_creator_role and minor changes

moved the game creator to the nginx namespace and updated it's role to reflect this, the ClusterRoleBinding gives the permissions of the game creator's permissions scope of the whole cluster (hence why it was moved out of reach of the worker pods, just incase there is a way for workers to gain access outside of their pod)

* Fixed game creator rc yaml not applying role binding

* change role binding namespace

* updated version

* Delete nginx-ingress folder

* Revert "get docker for mac working"

This reverts commit 017620a.

# Conflicts:
#	aimmo_runner/minikube.py

* Automate applying the accounts and roles

* fix travis build fail attempt

* Testing build fail

* More travis build tests

* travis build fix attempt++

* Update travis kubernetes version

* change kubernetes to ealier patch version

Kubernetes needs to be version 1.9.4 not 1.9.6

* Tests

* various changes

* Merge branch 'development' into c-001

* Resolve merge conflicts

* Merge remote-tracking branch 'origin/c-001' into c-001

* Fix issues created from merge conflicts

* Replacing versioner in setup.py

* minor change

* change to restart code climate

* pep8

* Docker for local mode (#859)

* Reduce image sizes for docker images

* bump version

* Stop turn slowdown (#857)

* Identified a potential point of slowdown

Turn processing is already done concurrently, but data fetching for all the workers was not, so i have changed this in the hopes it will speed up turn times and the number of users increases.

* Rethinking approach

difficult to implement concurrent solution to the code in question, rethinking approach.

* Added comments to where things need changing

added a couple of comments to explain what need's to be done in various places that i've identified that could slow down turns.

* Turn times had been sped up

Solution has been to create a timed process populated with daemon threads, this forces the main source of the slowdown into a 2 second window. However this has highlighted a big issue, the requests take more than 2 seconds a significant proportion of the time and as a result many actions become wait actions, also there needs to be a degree of garbage collection to prevent the creation of a potentially server ending amount of processes

* Cleanup

game runs turns at consistent 3 seconds locally even with 10 people, which in the past seem to cause the game to crash. Avatars seem to be able to respond in the 2 second limit, however we may want to increase this duration to make it more reliable.

* Merge branch 'development' into stop_turn_slowdown

* update version

* Minor changes

* Grammar changes

* reviewable changes

* Bump version number

* Reviewable changes++

* Merge branch 'development' into stop_turn_slowdown

* pep8

* Merge remote-tracking branch 'origin/stop_turn_slowdown' into stop_turn_slowdown

* Merge master into development (#861)

* Update minikube status check (#846)

* Update logic to check whether minikube is already running.

* Updated except to check for CalledProcessError.

* Merge branch 'master' into minikube-run-start

* Merge branch 'master' into development

* Merge branch 'development' into update-Development

* new line (#935)

* new line

* updating minor version

* Merge branch 'development' of https://github.com/ocadotechnology/aimmo into development

* Button hotfix (#936)

* add updating state

* remove debug console log

* Merge branch 'development' into run_code_loading

* circular progress inherits it’s coor from the parent button

* disable button when updating

* implementation of runcodebutton states (tests failing)

* fix proptypes not working

* fix/add tests

* add test case

* add logic for buttons error state (read desc.)

This would occur when the frontend cannot communicate with the server, the way this is usually handled is via a timeout. I've added an epic & reducer that will set a timeoutStatus to true after 20 seconds of inactivity from the server (there's an error though). The game should "un-timeout" when it receives a request, this still needs to be done, along with the visuals of the button and the popup message.

* moved timeoutStatus into game

* timeout state changes to true when no response received after x seconds

* Simplified approach to just detect when gameState is not being received

* cleanup

* map timeoutStatus state to button prop

* cleanup++

* tests for error state

* bump version

* fix failing tests

* Button hotfix

* bump version

* Merge branch 'development' into run_code_loading

* Merge branch 'development' of https://github.com/ocadotechnology/aimmo into development

* Merge branch 'master' into development

* Merge branch 'development' of https://github.com/ocadotechnology/aimmo into development

# Conflicts:
#	aimmo/admin.py
#	game_frontend/yarn.lock
#	version.txt
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants