Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Test/Server code out of sync, blocking PRs #196

Closed
Ark-kun opened this issue Nov 10, 2018 · 5 comments
Closed

Test/Server code out of sync, blocking PRs #196

Ark-kun opened this issue Nov 10, 2018 · 5 comments
Assignees
@Ark-kun @IronPan
Labels
area/testing priority/p0

Comments

@Ark-kun
Copy link
Contributor

Ark-kun commented Nov 10, 2018 •
edited
Loading

https://gubernator.k8s.io/build/kubernetes-jenkins/pr-logs/pull/kubeflow_pipelines/173/presubmit-e2e-test/87
https://gubernator.k8s.io/build/kubernetes-jenkins/pr-logs/pull/kubeflow_pipelines/168/presubmit-e2e-test/89
https://gubernator.k8s.io/build/kubernetes-jenkins/pr-logs/pull/kubeflow_pipelines/148/presubmit-e2e-test/105

https://gubernator.k8s.io/pr/kubeflow/pipelines/148
https://gubernator.k8s.io/pr/kubeflow/pipelines/172
https://gubernator.k8s.io/pr/kubeflow/pipelines/173

run-api-integration-tests:	--- FAIL: TestCLI (1.81s)
run-api-integration-tests:	    --- PASS: TestCLI/TestPipelineCreateGetDeleteSuccess (0.87s)
run-api-integration-tests:	    --- PASS: TestCLI/TestPipelineListFailureInvalidArgument (0.02s)
run-api-integration-tests:	    --- PASS: TestCLI/TestPipelineListSuccess (0.45s)
run-api-integration-tests:	    --- FAIL: TestCLI/TestPipelineUploadSuccess (0.46s)
run-api-integration-tests:	    	cli_test.go:107: 
run-api-integration-tests:	    			Error Trace:	cli_test.go:107
run-api-integration-tests:	    			Error:      	Expected nil, but got: &errors.errorString{s:"Issue calling the service. Use the '\''\'\'''\''--debug'\''\'\'''\'' flag to see the HTTP request/response. Raw error from the client: &{0001-01-01T00:00:00.000Z     []} (*pipeline_upload_model.APIPipeline) is not supported by the TextConsumer, can be resolved by supporting TextUnmarshaler interface\nFailed to upload pipeline. Params: '\''\'\'''\''&{0xc420363b40 0xc4205d5260 30000000000 context.Background.WithDeadline(2018-11-10 06:58:42.365578944 +0000 UTC m=+36.413746161 [34.589790414s]) <nil>}'\''\'\'''\''\ngithub.com/kubeflow/pipelines/backend/src/common/util.NewUserError\n\t/go/src/github.com/kubeflow/pipelines/backend/src/common/util/error.go:121\ngithub.com/kubeflow/pipelines/backend/src/common/client/api_server.(*PipelineUploadClient).Upload\n\t/go/src/github.com/kubeflow/pipelines/backend/src/common/client/api_server/pipeline_upload_client.go:79\ngithub.com/kubeflow/pipelines/backend/src/common/client/api_server.(*PipelineUploadClient).UploadFile\n\t/go/src/github.com/kubeflow/pipelines/backend/src/common/client/api_server/pipeline_upload_client.go:59\ngithub.com/kubeflow/pipelines/backend/src/cmd/ml/cmd.NewPipelineUploadCmd.func2\n\t/go/src/github.com/kubeflow/pipelines/backend/src/cmd/ml/cmd/pipeline.go:47\ngithub.com/kubeflow/pipelines/vendor/github.com/spf13/cobra.(*Command).execute\n\t/go/src/github.com/kubeflow/pipelines/vendor/github.com/spf13/cobra/command.go:762\ngithub.com/kubeflow/pipelines/vendor/github.com/spf13/cobra.(*Command).ExecuteC\n\t/go/src/github.com/kubeflow/pipelines/vendor/github.com/spf13/cobra/command.go:852\ngithub.com/kubeflow/pipelines/backend/test.(*CLIIntegrationTest).TestPipelineUploadSuccess\n\t/go/src/github.com/kubeflow/pipelines/backend/test/cli_test.go:106\nruntime.call32\n\t/usr/local/go/src/runtime/asm_amd64.s:573\nreflect.Value.call\n\t/usr/local/go/src/reflect/value.go:447\nreflect.Value.Call\n\t/usr/local/go/src/reflect/value.go:308\ngithub.com/kubeflow/pipelines/vendor/github.com/stretchr/testify/suite.Run.func2\n\t/go/src/github.com/kubeflow/pipelines/vendor/github.com/stretchr/testify/suite/suite.go:102\ntesting.tRunner\n\t/usr/local/go/src/testing/testing.go:777\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:2361"}
run-api-integration-tests:	    			Test:       	TestCLI/TestPipelineUploadSuccess
run-api-integration-tests:	=== RUN   TestExperimentAPI
run-api-integration-tests:	=== RUN   TestExperimentAPI/TestExperimentAPI
run-api-integration-tests:	--- FAIL: TestExperimentAPI (2.16s)
run-api-integration-tests:	    --- FAIL: TestExperimentAPI/TestExperimentAPI (2.16s)
run-api-integration-tests:	    	test_utils.go:110: 
run-api-integration-tests:	    			Error Trace:	test_utils.go:110
run-api-integration-tests:	    			            				experiment_api_test.go:146
run-api-integration-tests:	    			Error:      	Expected nil, but got: &util.UserError{internalError:(*errors.withStack)(0xc420461a40), externalMessage:"Failed to delete experiment. Raw error from the service: Issue calling the service. Use the '\''\'\'''\''--debug'\''\'\'''\'' flag to see the HTTP request/response. Raw error from the client: &{0 [] } (*experiment_model.APIStatus) is not supported by the TextConsumer, can be resolved by supporting TextUnmarshaler interface", externalStatusCode:0xd}
run-api-integration-tests:	    			Test:       	TestExperimentAPI/TestExperimentAPI
run-api-integration-tests:	    	test_utils.go:110: 
run-api-integration-tests:	    			Error Trace:	test_utils.go:110
run-api-integration-tests:	    			            				experiment_api_test.go:146
run-api-integration-tests:	    			Error:      	Expected nil, but got: &util.UserError{internalError:(*errors.withStack)(0xc42032fe20), externalMessage:"Failed to delete experiment. Raw error from the service: Issue calling the service. Use the '\''\'\'''\''--debug'\''\'\'''\'' flag to see the HTTP request/response. Raw error from the client: &{0 [] } (*experiment_model.APIStatus) is not supported by the TextConsumer, can be resolved by supporting TextUnmarshaler interface", externalStatusCode:0xd}
run-api-integration-tests:	    			Test:       	TestExperimentAPI/TestExperimentAPI
run-api-integration-tests:	    	test_utils.go:110: 
run-api-integration-tests:	    			Error Trace:	test_utils.go:110
run-api-integration-tests:	    			            				experiment_api_test.go:146
run-api-integration-tests:	    			Error:      	Expected nil, but got: &util.UserError{internalError:(*errors.withStack)(0xc420134440), externalMessage:"Failed to delete experiment. Raw error from the service: Issue calling the service. Use the '\''\'\'''\''--debug'\''\'\'''\'' flag to see the HTTP request/response. Raw error from the client: &{0 [] } (*experiment_model.APIStatus) is not supported by the TextConsumer, can be resolved by supporting TextUnmarshaler interface", externalStatusCode:0xd}
run-api-integration-tests:	    			Test:       	TestExperimentAPI/TestExperimentAPI
run-api-integration-tests:	=== RUN   TestJobApi
run-api-integration-tests:	=== RUN   TestJobApi/TestJobApis
run-api-integration-tests:	panic: runtime error: invalid memory address or nil pointer dereference [recovered]
run-api-integration-tests:		panic: runtime error: invalid memory address or nil pointer dereference
run-api-integration-tests:	[signal SIGSEGV: segmentation violation code=0x1 addr=0x40 pc=0x1111250]
run-api-integration-tests:	goroutine 193 [running]:
run-api-integration-tests:	testing.tRunner.func1(0xc4203720f0)
run-api-integration-tests:		/usr/local/go/src/testing/testing.go:742 +0x29d
run-api-integration-tests:	panic(0x1288e40, 0x1e20a30)
run-api-integration-tests:		/usr/local/go/src/runtime/panic.go:502 +0x229
run-api-integration-tests:	github.com/kubeflow/pipelines/backend/test.(*JobApiTestSuite).TestJobApis(0xc4200c34a0)
run-api-integration-tests:		/go/src/github.com/kubeflow/pipelines/backend/test/job_api_test.go:82 +0x270
run-api-integration-tests:	reflect.Value.call(0xc4207b8660, 0xc42000fe48, 0x13, 0x1431bb0, 0x4, 0xc42083ff80, 0x1, 0x1, 0xc42083fec8, 0x142e120, ...)
run-api-integration-tests:		/usr/local/go/src/reflect/value.go:447 +0x969
run-api-integration-tests:	reflect.Value.Call(0xc4207b8660, 0xc42000fe48, 0x13, 0xc42083ff80, 0x1, 0x1, 0x114aa19, 0xb, 0x0)
run-api-integration-tests:		/usr/local/go/src/reflect/value.go:308 +0xa4
run-api-integration-tests:	github.com/kubeflow/pipelines/vendor/github.com/stretchr/testify/suite.Run.func2(0xc4203720f0)
run-api-integration-tests:		/go/src/github.com/kubeflow/pipelines/vendor/github.com/stretchr/testify/suite/suite.go:102 +0x243
run-api-integration-tests:	testing.tRunner(0xc4203720f0, 0xc420454180)
run-api-integration-tests:		/usr/local/go/src/testing/testing.go:777 +0xd0
run-api-integration-tests:	created by testing.(*T).Run
run-api-integration-tests:		/usr/local/go/src/testing/testing.go:824 +0x2e0
run-api-integration-tests:	FAIL	github.com/kubeflow/pipelines/backend/test	4.580s
@Ark-kun Ark-kun changed the title CLI tests seem to be failing and blocking all check-ins CLI tests seem to be failing and blocking check-ins Nov 10, 2018
@Ark-kun Ark-kun mentioned this issue Nov 10, 2018
Closed
@Ark-kun
Copy link
Contributor Author

Ark-kun commented Nov 10, 2018

That issue got fixed when I explicitly re-run build-image step for all problematic PRs.

I do not fully understand how the unit test image has caused problem, since it should only contain the run_test.sh script and couple of Go packages.

@IronPan
Copy link
Member

IronPan commented Nov 11, 2018

I think these are integration test failures.

The issue might be that your build-image step is ran before #112 and the test is ran after that PR. The test and server code are out of sync.

Thus, I don't think we should have this change #165
It makes code being tested non deterministic, and non reproducible.

Please consider revert the change.

@IronPan IronPan mentioned this issue Nov 11, 2018
Closed
@IronPan
Copy link
Member

IronPan commented Nov 11, 2018

(The test code would use client library which is par of the source code)

@Ark-kun
Copy link
Contributor Author

Ark-kun commented Nov 11, 2018

Thus, I don't think we should have this change #165
It makes code being tested non deterministic, and non reproducible.

Please consider revert the change.

Turns out #165 does not fix the issue completely, but it's at least closer to what we should ideally have. Ideally, all our tests should test the code that Prow prepared before running the test.

Without #165, the desync happens in ~90% of cases. With #165 it happens in ~10% of cases.
The reason we got a desync now is due to the fact that cluster leakage and quota exhaustion caused many tests to fail repeatedly, thus increasing the time between build-image and test runs.

Without #165, the desync is unfixable - the tests will never succeed unless you merge master to the branch (which removes all LGTM and APPROVED labels). With #165 the bad desync can be fixed by re-running the build-image step.

The only way to 100% fix this is to never do git clone in our tests. Let's think about ways to accomplish that.

@Ark-kun
Copy link
Contributor Author

Ark-kun commented Nov 11, 2018

Here is how to fix this for backend unit tests:

  1. Build+push the backend unit test base image (go+gcloud+jstemmer). (No repo source code)
  2. Specify that image in Prow config.
  3. Use test/backend-unit-test/run_test.sh as test entry point

@vicaire vicaire changed the title CLI tests seem to be failing and blocking check-ins Test/Server code out of sync, blocking PRs Nov 12, 2018
@vicaire vicaire assigned Ark-kun and unassigned vicaire Nov 12, 2018
@Ark-kun Ark-kun mentioned this issue Dec 14, 2018
Merged
@Ark-kun Ark-kun closed this as completed Dec 14, 2018
Linchin pushed a commit to Linchin/pipelines that referenced this issue Apr 11, 2023
@jlewi @k8s-ci-robot
Only consider include_dirs on presubmit tests; install funcsigs. (kub…
579319c 
…eflow#195)

* Only consider include_dirs on presubmit tests.

* We all tests to be triggered only when certain files are modified.
* I think it only makes sense to apply this filter on presubmits; for
  postsubmits we should include always run a test.

Related to:

kubeflow/kubeflow#1048

* update readme.

* Try to fix kubeflow#196 by updating funcsigs.
HumairAK pushed a commit to red-hat-data-services/data-science-pipelines that referenced this issue Mar 11, 2024
@drewbutlerbb4
Tekton UI patch (kubeflow#196)
7a19833
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@Ark-kun Ark-kun

@IronPan IronPan

Labels
area/testing priority/p0
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@Ark-kun @IronPan @vicaire