Add Istio specific assertion in Helm Broker acceptance test

[polskikiel]

Description

Acceptance-test fails with the following error:
Waiting for function failed in given timeout 5s. Function was executed 5 times. Last error: while getting catalog: while getting catalog from broker with URL: http://helm-broker.kyma-system.svc.cluster.local: Status: 404; ErrorMessage: <nil>; Description: <nil>; ResponseError: invalid character 'p' after top-level value

It happens since 19.08 so removing HB from Kyma might be an issue.

[jasiu001]

Increasing timeout for test TestBrokerHasIstioRbacAuthorizationRules does not solve the issue.

The first problem is the URL on which request is sent is incorrect. After introduction (Cluster)AddonsConfiguration, the correct URL on which ServiceCatalog should look for classes is /cluster/v2/catalog but instead /v2/catalog is used. (fixed)

The second problem is probably connected with wrong RBAC configuration on helm broker or lack thereof. Sidecar on helm broker should reject the request with 403 status code, instead of this request is passed and code 200(or 404) is returned.
Rules that determine accessibility for HelmBroker are in here and here

The failing test is not deterministic.

To fully rule out a istio error after failed test we should check first if ServiceRole and ServiceRoleBinding exist and all authorization rules are inject to the helm broker's sidecar.
This tutorial could help to receive that information.
If it turns out that access rules are not incjected to sidecar maybe we should add some "wait" function.

When full istio rbac monitoring will be apply turn on TestBrokerHasIstioRbacAuthorizationRules test, disabled in this PR

[mszostok]

two things to do in that issue:

• [ESTIAMTED FOR 8SP]rewrite the current destination rule tester (make it more debugable)
• [TIMEBOXED: 3SP] investigate and create a separate issue with concrete actions for refactoring current acceptance tests, e.i.:
  • support for running given tests (e.g. by go run regex?)
  • remove tests that are not used anymore (e.g. acceptance test which are skipped since 1year)
  • etc.

[polskikiel]

As mentioned in the above #4409 #531 issues the test The ServiceBindingUsagePrefixing is supposed to be fixed within this issue because it's also istio related.

We may think about solving this issue globally for all test by implementing istio rule checker in the Octopus for all of the tests: kyma-incubator/octopus#34

I didn't make progress on how to assert istio rules. We should disable this test at this moment because it's failing at CI-Force with message:

binding_usage_test.go:525: Waiting for function failed in given timeout 2m0s. Function was executed 120 times. Last error: "acc-test-svc-id-a-zgrn2" not found

TestServiceBindingUsagePrefixing report after fail can be improved by displaying ApplicationMapping, ClusterServiceClasses and ClusterServiceBrokers

I created an issue about refactoring the acceptance-tests: #5639

[piotrmiskiewicz]

The istio RBAC rules sometimes does not work, logs from the Helm-Broker shows calls from the test:

{"level":"info","log":{"action":"catalog","message":"action response","resp:services:count":10,"service":"server","time":"2019-09-18T17:45:54.686Z"}}
{"level":"info","log":{"message":"completed handling request","method":"GET","remote":"127.0.0.1:54454","request":"/cluster/v2/catalog","request_id":"1c621d5f-b78a-4fbe-a869-136c4ad7ea70","size":22241,"status":200,"time":"2019-09-18T17:45:54.687Z","took":4327470}}
{"level":"info","log":{"message":"started handling request","method":"GET","remote":"127.0.0.1:54454","request":"/cluster/v2/catalog","request_id":"5aec33e1-25d1-47eb-a95a-96fbc6b2ec2d","time":"2019-09-18T17:45:55.691Z"}}
{"level":"info","log":{"action":"catalog","message":"action response","resp:services:count":10,"service":"server","time":"2019-09-18T17:45:55.699Z"}}
{"level":"info","log":{"message":"completed handling request","method":"GET","remote":"127.0.0.1:54454","request":"/cluster/v2/catalog","request_id":"5aec33e1-25d1-47eb-a95a-96fbc6b2ec2d","size":22241,"status":200,"time":"2019-09-18T17:45:55.7Z","took":8901482}}
{"level":"info","log":{"message":"started handling request","method":"GET","remote":"127.0.0.1:54454","request":"/cluster/v2/catalog","request_id":"8684c427-b5e7-4099-aa43-71035c1faca6","time":"2019-09-18T17:45:56.685Z"}}
{"level":"info","log":{"action":"catalog","message":"action response","resp:services:count":10,"service":"server","time":"2019-09-18T17:45:56.689Z"}}
{"level":"info","log":{"message":"completed handling request","method":"GET","remote":"127.0.0.1:54454","request":"/cluster/v2/catalog","request_id":"8684c427-b5e7-4099-aa43-71035c1faca6","size":22241,"status":200,"time":"2019-09-18T17:45:56.689Z","took":4352770}}
{"level":"info","log":{"message":"started handling request","method":"GET","remote":"127.0.0.1:54454","request":"/cluster/v2/catalog","request_id":"3cb82d43-6b97-432d-b86d-ddd8a43448bd","time":"2019-09-18T17:45:57.685Z"}}
{"level":"info","log":{"action":"catalog","message":"action response","resp:services:count":10,"service":"server","time":"2019-09-18T17:45:57.689Z"}}
{"level":"info","log":{"message":"completed handling request","method":"GET","remote":"127.0.0.1:54454","request":"/cluster/v2/catalog","request_id":"3cb82d43-6b97-432d-b86d-ddd8a43448bd","size":22241,"status":200,"time":"2019-09-18T17:45:57.697Z","took":12508855}}
{"level":"info","log":{"message":"started handling request","method":"GET","remote":"127.0.0.1:54454","request":"/cluster/v2/catalog","request_id":"c217ecc0-c67b-4a7d-8aed-22f4a4436979","time":"2019-09-18T17:45:58.686Z"}}
{"level":"info","log":{"action":"catalog","message":"action response","resp:services:count":10,"service":"server","time":"2019-09-18T17:45:58.701Z"}}
{"level":"info","log":{"message":"completed handling request","method":"GET","remote":"127.0.0.1:54454","request":"/cluster/v2/catalog","request_id":"c217ecc0-c67b-4a7d-8aed-22f4a4436979","size":22241,"status":200,"time":"2019-09-18T17:45:58.702Z","took":16262025}}
{"level":"info","log":{"message":"started handling request","method":"GET","remote":"127.0.0.1:54454","request":"/cluster/v2/catalog","request_id":"d49a7d3d-423d-4561-bf80-32498bc0ac26","time":"2019-09-18T17:45:59.686Z"}}
{"level":"info","log":{"action":"catalog","message":"action response","resp:services:count":10,"service":"server","time":"2019-09-18T17:45:59.702Z"}}
{"level":"info","log":{"message":"completed handling request","method":"GET","remote":"127.0.0.1:54454","request":"/cluster/v2/catalog","request_id":"d49a7d3d-423d-4561-bf80-32498bc0ac26","size":22241,"status":200,"time":"2019-09-18T17:45:59.703Z","took":16774299}}
{"level":"info","log":{"message":"started handling request","method":"GET","remote":"127.0.0.1:54454","request":"/cluster/v2/catalog","request_id":"905ae38d-8098-45c4-8e29-b9f6202f90e3","time":"2019-09-18T17:46:00.685Z"}}
{"level":"info","log":{"action":"catalog","message":"action response","resp:services:count":10,"service":"server","time":"2019-09-18T17:46:00.689Z"}}
{"level":"info","log":{"message":"completed handling request","method":"GET","remote":"127.0.0.1:54454","request":"/cluster/v2/catalog","request_id":"905ae38d-8098-45c4-8e29-b9f6202f90e3","size":22241,"status":200,"time":"2019-09-18T17:46:00.69Z","took":4383050}}
{"level":"info","log":{"message":"started handling request","method":"GET","remote":"127.0.0.1:54454","request":"/cluster/v2/catalog","request_id":"695a4b68-7383-4bc2-bb71-96fc029975ca","time":"2019-09-18T17:46:01.682Z"}}
{"level":"info","log":{"action":"catalog","message":"action response","resp:services:count":10,"service":"server","time":"2019-09-18T17:46:01.7Z"}}
{"level":"info","log":{"message":"completed handling request","method":"GET","remote":"127.0.0.1:54454","request":"/cluster/v2/catalog","request_id":"695a4b68-7383-4bc2-bb71-96fc029975ca","size":22241,"status":200,"time":"2019-09-18T17:46:01.702Z","took":19757193}}
{"level":"info","log":{"message":"started handling request","method":"GET","remote":"127.0.0.1:54454","request":"/cluster/v2/catalog","request_id":"ec6ad03a-d571-42ac-8726-11be54e6474d","time":"2019-09-18T17:46:02.687Z"}}
{"level":"info","log":{"action":"catalog","message":"action response","resp:services:count":10,"service":"server","time":"2019-09-18T17:46:02.691Z"}}
{"level":"info","log":{"message":"completed handling request","method":"GET","remote":"127.0.0.1:54454","request":"/cluster/v2/catalog","request_id":"ec6ad03a-d571-42ac-8726-11be54e6474d","size":22241,"status":200,"time":"2019-09-18T17:46:02.692Z","took":5428285}}
{"level":"info","log":{"message":"started handling request","method":"GET","remote":"127.0.0.1:54454","request":"/cluster/v2/catalog","request_id":"ccfe472b-de3f-4e6c-877f-277448a85a27","time":"2019-09-18T17:46:03.685Z"}}
{"level":"info","log":{"action":"catalog","message":"action response","resp:services:count":10,"service":"server","time":"2019-09-18T17:46:03.898Z"}}

[piotrmiskiewicz]

I've created a ticket about the Istio problem: #5694