intermittent 503 errors in end-to-end tests

[ryandawsonuk]

Sometimes see issues like this when running make test in '/testing/scripts/' against minikube:

_____________________ TestRollingHttp.test_rolling_update8 _____________________

self = <test_rolling_updates.TestRollingHttp object at 0x7f6ea3646240>

    def test_rolling_update8(self):
        tester = RollingTests()
>       tester.test_rolling_update8()

test_rolling_updates.py:58: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <test_rolling_updates.RollingTests object at 0x7f6ea36465f8>

    def test_rolling_update8(self):
        run("kubectl delete sdep --all", shell=True)
        wait_for_shutdown("mymodel-mymodel-e2eb561")
        wait_for_shutdown("mymodel-mymodel-svc-orch-8e2a24b")
        run("kubectl apply -f ../resources/graph1svc.json", shell=True, check=True)
        wait_for_rollout("mymodel-mymodel-svc-orch-8e2a24b")
        wait_for_rollout("mymodel-mymodel-e2eb561")
        r = initial_rest_request("mymodel","seldon")
        assert r.status_code == 200
        assert r.json()["data"]["tensor"]["values"] == [1.0,2.0,3.0,4.0]
        run("kubectl apply -f ../resources/graph4svc.json", shell=True, check=True)
        i = 0
        for i in range(50):
            r = rest_request_ambassador("mymodel", "seldon", API_AMBASSADOR)
            print("Status code", r.status_code)
>           assert r.status_code == 200
E           assert 503 == 200
E            +  where 503 = <Response [503]>.status_code

test_rolling_updates.py:261: AssertionError

Failures are on the rolling update tests but not always the same test and sometimes they even all pass.

Seems to fit with emissary-ingress/emissary#1461

[ryandawsonuk]

Have tested with just 1 ambassador pod as suggested in the ambassador issue and the 503s do go away