Worker should not participate in game if Avatar initialization fails

[jishi9]

Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/flask/app.py", line 1836, in __call__
    return self.wsgi_app(environ, start_response)
  File "/usr/local/lib/python2.7/site-packages/flask/app.py", line 1820, in wsgi_app
    response = self.make_response(self.handle_exception(e))
  File "/usr/local/lib/python2.7/site-packages/flask/app.py", line 1403, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/local/lib/python2.7/site-packages/flask/app.py", line 1817, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/lib/python2.7/site-packages/flask/app.py", line 1477, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/local/lib/python2.7/site-packages/flask/app.py", line 1381, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/local/lib/python2.7/site-packages/flask/app.py", line 1475, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python2.7/site-packages/flask/app.py", line 1461, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/Users/wael.aljeshi/Documents/c4l/aimmo/aimmo-game-worker/service.py", line 38, in process_turn
    action = avatar.handle_turn(avatar_state, world_map)
AttributeError: 'NoneType' object has no attribute 'handle_turn'

[Spycho]

Currently, if you bug an Avatar (e.g. def __init__(self): raise Exception('test'), Kubernetes failes to start the pod, at which point it backs off and tries again, I think infinitely. It should probably give up and kill it after one failure? 3 failures, in case its not the player's fault? Is this a thing we can do with Kubernetes, @mikebryant? Presumably we'd also need to do something to aimmo-game to stop it just trying to spawn another avatar worker. Giving the user some feedback would be useful, which possibly means we should give up after 1 failure to avoid confusing matters?

[OlafSzmidt]

If not possible, we could add some logic to see if the avatar is properly initialized before handling the turn - but that's a very short term solution imho.

Kubernates seems have this feature added very recently. Am I right @mikebryant? It's not to be mistaken with restartPolicy which is described here.

The failure policy for jobs was added just 25 days ago, if this is any relevant to this issue but I think it's in alpha?

Edit: The solution seems to be in .spec.template.spec.backoffLimit! Although this is weird because the default is supposed to be set at 6 mins?

[OlafSzmidt]

@mikebryant @Spycho OlafSzmidt@472b7a9 ? Not sure if relevant.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[OlafSzmidt]

Still relevant.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[CelineBoudier]

Still relevant? @OlafSzmidt @mrniket

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[j4mesholland]

not stale

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[OlafSzmidt]

Is this applicable anymore? We removed worker recreation. What happens when the initial initialisation fails?

@NiallEgan

[NiallEgan]

If there is a problem creating the avatar (i.e. if there is an issue with the source code), this error is caught and returned to the front end. I think that this can be closed.

I think really this was dealt with in the logging / ensuring worker pods don't crash

[NiallEgan]

Closing this as it has been dealt with in other issues.