Stabilize application-operator and application-connector tests

[franpog859]

Description

Stabilize application-operator and application-connector tests on nightly and weekly clusters. The tests fail on about 10% of runs. It's not something critical because it does not happen on the PRs so it does not block the implementation process. It's important where it comes to the CI-force side, where the test owners are being pinged a lot

The issue probably concerns the clusters configuration. I assume so because it does not happen very often and does happen hardly ever on PRs

Expected result

application-operator and application-connector tests do not fail on nightly and weekly clusters

Logs

• https://status.build.kyma-project.io/view/gcs/kyma-prow-logs/logs/post-master-kyma-gke-upgrade/1253645692753154049
• https://status.build.kyma-project.io/view/gcs/kyma-prow-logs/logs/post-master-kyma-gke-central-connector/1253620022069170176

[franpog859]

@adamwalach said:

Just wanted to mention that VMs used for weekly/nightly clusters are huge - we changed the configuration some time ago because it was even more unstable - right now on weekly we have 3 nodes with 12cpu/16GBram each.

I think it could help with the investigation process, thanks!

[franpog859]

Logs from CI after this commit:

• operator
• operator
• operator
• operator

Error Trace:    proxy_test.go:421
                   proxy_test.go:286
Error:          Not equal: 
                   expected: string("2.0")
                   actual  : <nil>(<nil>)
Test:           TestProxyService/oauth_spec_url_test

Error Trace:    proxy_test.go:421
                   proxy_test.go:379
Error:          Not equal: 
                   expected: string("2.0")
                   actual  : <nil>(<nil>)
Test:           TestProxyService/additional_query_params_in_spec_test

Error Trace:    gateway_events_test.go:100
Error:          Should be true
Test:           TestGatewayEvents/should_get_all_subscribed_events

TestProxyService/additional_query_parameters_test
util.go:28: 
--------------------------------
POST /operator-app-test-p8rx/v1/metadata/services HTTP/1.1
Host: application-registry-external-api:8081
--------------------------------
HTTP/1.1 500 Internal Server Error
Content-Length: 291
Content-Type: application/json;charset=UTF-8
Date: Tue, 26 May 2020 09:39:24 GMT
Server: envoy
X-Envoy-Upstream-Service-Time: 418

{"code":500,"error":"Creating service in Application failed, Creating service failed, Operation cannot be fulfilled on applications.applicationconnector.kyma-project.io \"operator-app-test-p8rx\": the object has been modified; please apply your changes to the latest version and try again"}
--------------------------------
util.go:12: Invalid response code

These may be helpful:

• PR where Gateway Tests are hardcoded
• Kubernetes issue with the Application Registry error message

---

• application connector

        POST /app-conn-tests-6xdc/v1/events HTTP/1.1
        Host: gateway.gke-upgrade-commit-332d309c9-2cuzjjlxsy.a.build.kyma-project.io
        --------------------------------
        HTTP/1.1 500 Internal Server Error
        Content-Length: 131
        Content-Type: application/json
        Date: Tue, 02 Jun 2020 09:44:25 GMT
        Server: istio-envoy
        X-Envoy-Upstream-Service-Time: 1031
        
        {"status":500,"type":"internal_server","message":"internal_server","moreInfo":"error sending cloudevent: 503 Service Unavailable"}
        --------------------------------
        require.go:794: 
        	Error Trace:	suite.go:242
                                application_access_test.go:31
        	Error:      	Received unexpected error:
        	            	failed to send Event. Status: 0, Error: 
        	Test:       	TestApplicationAccess

[franpog859]

After this PR the tests should be more stable. I suggest to close this issue and reopen it whenever the problem occurs again. If so, please, add the link and the important log in the comment! 👋

[franpog859]

• operator failed with timeout - logs

[franpog859]

Hey, @jakkab ! Feel free to add the logs with the errors in the comment. I'll help with fixing them

[koala7659]

The problem persists - logs

[adamwalach]

post-master-kyma-gke-upgrade failed
https://status.build.kyma-project.io/view/gcs/kyma-prow-logs/logs/post-master-kyma-gke-upgrade/1287731447897001984

[adamwalach]

I checked logs in gcp (stackdriver) and it looks like this problem is quite common:

[adamwalach]

Looks quite promising, there are no application-connector failures since the patch was merged:

filter: text:'application-connector-1" in status "Failed' in all containers in workload-kyma-prow cluster