Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add verify-codegen in travis CI #1070

Merged
merged 2 commits into from
Aug 19, 2019

Conversation

ohmystack
Copy link
Contributor

@ohmystack ohmystack commented Aug 16, 2019

This change is Reviewable

@k8s-ci-robot
Copy link

Hi @ohmystack. Thanks for your PR.

I'm waiting for a kubeflow member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@ohmystack
Copy link
Contributor Author

Mentioned here: #1069 (comment)

.travis.yml Outdated
@@ -24,6 +24,7 @@ script:
# And we can not use ** because goveralls uses filepath.Match
# to match ignore files and it does not support it.
- goveralls -service=travis-ci -v -package ./pkg/... -ignore "pkg/client/*/*.go,pkg/client/*/*/*.go,pkg/client/*/*/*/*.go,pkg/client/*/*/*/*/*.go,pkg/client/*/*/*/*/*/*.go,pkg/client/*/*/*/*/*/*/*.go,pkg/util/testutil/*.go,pkg/apis/tensorflow/*/zz_generated.*.go,pkg/apis/tensorflow/*/*_generated.go,pkg/apis/common/*/zz_generated.*.go,pkg/apis/common/*/*_generated.go"
- hack/verify-codegen.sh
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we verify the codegen before the go build commands? Since it can fail fast.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It makes sense. Done.

@gaocegege
Copy link
Member

/ok-to-test

Thanks for your contribution! 🎉 👍

@coveralls
Copy link

coveralls commented Aug 16, 2019

Coverage Status

Coverage remained the same at 76.744% when pulling ef7088e on ohmystack:travis-verify-codegen into d164c36 on kubeflow:master.

@TravisBuddy
Copy link

Travis tests have failed

Hey @ohmystack,
Please read the following log in order to understand the failure reason.
It'll be awesome if you fix what's wrong and commit the changes.

1st Build

View build log

goveralls -service=travis-ci -v -package ./pkg/... -ignore "pkg/client/*/*.go,pkg/client/*/*/*.go,pkg/client/*/*/*/*.go,pkg/client/*/*/*/*/*.go,pkg/client/*/*/*/*/*/*.go,pkg/client/*/*/*/*/*/*/*.go,pkg/util/testutil/*.go,pkg/apis/tensorflow/*/zz_generated.*.go,pkg/apis/tensorflow/*/*_generated.go,pkg/apis/common/*/zz_generated.*.go,pkg/apis/common/*/*_generated.go"
?   	github.com/kubeflow/tf-operator/pkg/apis/common/v1	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/apis/common/v1beta2	[no test files]
=== RUN   TestSetTypeNames
--- PASS: TestSetTypeNames (0.00s)
=== RUN   TestSetDefaultTFJob
--- PASS: TestSetDefaultTFJob (0.00s)
=== RUN   TestIsChieforMaster
--- PASS: TestIsChieforMaster (0.00s)
PASS
coverage: 20.9% of statements in github.com/kubeflow/tf-operator/pkg/apis/common/v1, github.com/kubeflow/tf-operator/pkg/apis/common/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/controller.v1beta2/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
ok  	github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1	0.031s	coverage: 20.9% of statements in github.com/kubeflow/tf-operator/pkg/apis/common/v1, github.com/kubeflow/tf-operator/pkg/apis/common/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/controller.v1beta2/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
=== RUN   TestSetTypeNames
--- PASS: TestSetTypeNames (0.00s)
=== RUN   TestSetDefaultTFJob
--- PASS: TestSetDefaultTFJob (0.00s)
=== RUN   TestIsChieforMaster
--- PASS: TestIsChieforMaster (0.00s)
PASS
coverage: 20.9% of statements in github.com/kubeflow/tf-operator/pkg/apis/common/v1, github.com/kubeflow/tf-operator/pkg/apis/common/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/controller.v1beta2/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
ok  	github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1beta2	0.032s	coverage: 20.9% of statements in github.com/kubeflow/tf-operator/pkg/apis/common/v1, github.com/kubeflow/tf-operator/pkg/apis/common/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/controller.v1beta2/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
=== RUN   TestValidateBetaTwoTFJobSpec
time="2019-08-16T10:05:28Z" level=error msg="TFJobSpec is not valid: Image is undefined in the container of Worker"
time="2019-08-16T10:05:28Z" level=error msg="TFJobSpec is not valid: There is no container named tensorflow in Worker"
--- PASS: TestValidateBetaTwoTFJobSpec (0.00s)
=== RUN   TestValidateV1TFJobSpec
time="2019-08-16T10:05:28Z" level=error msg="TFJobSpec is not valid: Image is undefined in the container of Worker"
time="2019-08-16T10:05:28Z" level=error msg="TFJobSpec is not valid: There is no container named tensorflow in Worker"
--- PASS: TestValidateV1TFJobSpec (0.00s)
PASS
coverage: 14.2% of statements in github.com/kubeflow/tf-operator/pkg/apis/common/v1, github.com/kubeflow/tf-operator/pkg/apis/common/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/controller.v1beta2/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
ok  	github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation	0.030s	coverage: 14.2% of statements in github.com/kubeflow/tf-operator/pkg/apis/common/v1, github.com/kubeflow/tf-operator/pkg/apis/common/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/controller.v1beta2/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
?   	github.com/kubeflow/tf-operator/pkg/client/clientset/versioned	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2/fake	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/informers/externalversions	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1beta2	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1beta2	[no test files]
=== RUN   TestGenGeneralName
--- PASS: TestGenGeneralName (0.00s)
PASS
coverage: 0.4% of statements in github.com/kubeflow/tf-operator/pkg/apis/common/v1, github.com/kubeflow/tf-operator/pkg/apis/common/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/controller.v1beta2/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
ok  	github.com/kubeflow/tf-operator/pkg/common/jobcontroller	0.022s	coverage: 0.4% of statements in github.com/kubeflow/tf-operator/pkg/apis/common/v1, github.com/kubeflow/tf-operator/pkg/apis/common/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/controller.v1beta2/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
?   	github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/testutil	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/unstructured	[no test files]
=== RUN   TestCreatePods
--- PASS: TestCreatePods (0.01s)
=== RUN   TestCreateService
time="2019-08-16T10:05:46Z" level=info msg="Controller test-tfjob created service empty_service"
--- PASS: TestCreateService (0.00s)
=== RUN   TestCreateServicesWithControllerRef
time="2019-08-16T10:05:46Z" level=info msg="Controller test-tfjob created service empty_service"
--- PASS: TestCreateServicesWithControllerRef (0.00s)
=== RUN   TestClaimServices
--- PASS: TestClaimServices (0.00s)
PASS
coverage: 41.1% of statements in github.com/kubeflow/tf-operator/pkg/apis/common/v1, github.com/kubeflow/tf-operator/pkg/apis/common/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/controller.v1beta2/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
ok  	github.com/kubeflow/tf-operator/pkg/control	0.058s	coverage: 41.1% of statements in github.com/kubeflow/tf-operator/pkg/apis/common/v1, github.com/kubeflow/tf-operator/pkg/apis/common/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/controller.v1beta2/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
=== RUN   TestNormalPath
time="2019-08-16T10:05:53Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:05:53Z" level=info msg="Creating Job controller"
time="2019-08-16T10:05:53Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-16T10:05:53Z" level=info msg="Need to create new pod: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2019-08-16T10:05:53Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=1, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:05:53Z" level=info msg="need to create new service: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2019-08-16T10:05:53Z" level=info msg="Need to create new pod: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2019-08-16T10:05:53Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:05:53Z" level=info msg="need to create new service: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2019-08-16T10:05:53Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (3.088861ms)" job=default.test-tfjob
time="2019-08-16T10:05:53Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:05:53Z" level=info msg="Creating Job controller"
time="2019-08-16T10:05:53Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-16T10:05:53Z" level=info msg="Ignoring inactive pod default/worker-1 in state Succeeded, deletion time <nil>"
time="2019-08-16T10:05:53Z" level=info msg="Ignoring inactive pod default/worker-2 in state Succeeded, deletion time <nil>"
time="2019-08-16T10:05:53Z" level=info msg="Ignoring inactive pod default/worker-3 in state Succeeded, deletion time <nil>"
time="2019-08-16T10:05:53Z" level=info msg="Ignoring inactive pod default/ps-0 in state Succeeded, deletion time <nil>"
time="2019-08-16T10:05:53Z" level=info msg="Ignoring inactive pod default/ps-1 in state Succeeded, deletion time <nil>"
time="2019-08-16T10:05:53Z" level=info msg="Ignoring inactive pod default/worker-0 in state Succeeded, deletion time <nil>"
time="2019-08-16T10:05:53Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:05:53Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=0, running=0, failed=0" job=default.test-tfjob uid=
E0816 10:05:53.404708    9620 event.go:259] Could not construct reference to: '&v1.TFJob{TypeMeta:v1.TypeMeta{Kind:"TFJob", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"test-tfjob", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, Spec:v1.TFJobSpec{ActiveDeadlineSeconds:(*int64)(nil), BackoffLimit:(*int32)(nil), CleanPodPolicy:(*v1.CleanPodPolicy)(0xc42040d310), TTLSecondsAfterFinished:(*int32)(nil), TFReplicaSpecs:map[v1.TFReplicaType]*v1.ReplicaSpec{"PS":(*v1.ReplicaSpec)(0xc4202ae000), "Worker":(*v1.ReplicaSpec)(0xc4202af8c0)}}, Status:v1.JobStatus{Conditions:[]v1.JobCondition(nil), ReplicaStatuses:map[v1.ReplicaType]*v1.ReplicaStatus{"PS":(*v1.ReplicaStatus)(0xc420425460), "Worker":(*v1.ReplicaStatus)(0xc420425490)}, StartTime:(*v1.Time)(0xc4204ee720), CompletionTime:(*v1.Time)(nil), LastReconcileTime:(*v1.Time)(nil)}}' due to: 'selfLink was empty, can't make reference'. Will not report event: 'Normal' 'TFJobSucceeded' 'TFJob test-tfjob successfully completed.'
time="2019-08-16T10:05:53Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (1.700986ms)" job=default.test-tfjob
time="2019-08-16T10:05:53Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:05:53Z" level=info msg="Creating Job controller"
time="2019-08-16T10:05:53Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-16T10:05:53Z" level=info msg="Need to create new pod: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-08-16T10:05:53Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=1, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:05:53Z" level=info msg="need to create new service: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-08-16T10:05:53Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (1.004242ms)" job=default.test-tfjob
time="2019-08-16T10:05:53Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:05:53Z" level=info msg="Creating Job controller"
time="2019-08-16T10:05:53Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-16T10:05:53Z" level=info msg="Need to create new pod: ps-0" job=default.test-tfjob replica-type=ps uid=
time="2019-08-16T10:05:53Z" level=info msg="Need to create new pod: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2019-08-16T10:05:53Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:05:53Z" level=info msg="need to create new service: ps-0" job=default.test-tfjob replica-type=ps uid=
time="2019-08-16T10:05:53Z" level=info msg="need to create new service: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2019-08-16T10:05:53Z" level=info msg="Need to create new pod: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-08-16T10:05:53Z" level=info msg="Need to create new pod: worker-1" job=default.test-tfjob replica-type=worker uid=
time="2019-08-16T10:05:53Z" level=info msg="Need to create new pod: worker-2" job=default.test-tfjob replica-type=worker uid=
time="2019-08-16T10:05:53Z" level=info msg="Need to create new pod: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2019-08-16T10:05:53Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:05:53Z" level=info msg="need to create new service: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-08-16T10:05:53Z" level=info msg="need to create new service: worker-1" job=default.test-tfjob replica-type=worker uid=
time="2019-08-16T10:05:53Z" level=info msg="need to create new service: worker-2" job=default.test-tfjob replica-type=worker uid=
time="2019-08-16T10:05:53Z" level=info msg="need to create new service: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2019-08-16T10:05:53Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (1.640392ms)" job=default.test-tfjob
time="2019-08-16T10:05:53Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:05:53Z" level=info msg="Creating Job controller"
time="2019-08-16T10:05:53Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-16T10:05:53Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:05:53Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:05:53Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (399.707µs)" job=default.test-tfjob
time="2019-08-16T10:05:53Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:05:53Z" level=info msg="Creating Job controller"
time="2019-08-16T10:05:53Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-16T10:05:53Z" level=info msg="Need to create new pod: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2019-08-16T10:05:53Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:05:53Z" level=info msg="need to create new service: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2019-08-16T10:05:53Z" level=info msg="Need to create new pod: worker-2" job=default.test-tfjob replica-type=worker uid=
time="2019-08-16T10:05:53Z" level=info msg="Need to create new pod: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2019-08-16T10:05:53Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:05:53Z" level=info msg="need to create new service: worker-2" job=default.test-tfjob replica-type=worker uid=
time="2019-08-16T10:05:53Z" level=info msg="need to create new service: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2019-08-16T10:05:53Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (1.10998ms)" job=default.test-tfjob
time="2019-08-16T10:05:53Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:05:53Z" level=info msg="Creating Job controller"
time="2019-08-16T10:05:53Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-16T10:05:53Z" level=info msg="Ignoring inactive pod default/worker-2 in state Succeeded, deletion time <nil>"
time="2019-08-16T10:05:53Z" level=info msg="Need to create new pod: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2019-08-16T10:05:53Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:05:53Z" level=info msg="need to create new service: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2019-08-16T10:05:53Z" level=info msg="Need to create new pod: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2019-08-16T10:05:53Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=3, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:05:53Z" level=info msg="need to create new service: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2019-08-16T10:05:53Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (1.28893ms)" job=default.test-tfjob
time="2019-08-16T10:05:53Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:05:53Z" level=info msg="Creating Job controller"
time="2019-08-16T10:05:53Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-16T10:05:53Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:05:53Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=4, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:05:53Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (392.45µs)" job=default.test-tfjob
--- PASS: TestNormalPath (0.03s)
=== RUN   TestRun
time="2019-08-16T10:05:53Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:05:53Z" level=info msg="Creating Job controller"
time="2019-08-16T10:05:53Z" level=info msg="Starting TFJob controller"
time="2019-08-16T10:05:53Z" level=info msg="Waiting for informer caches to sync"
time="2019-08-16T10:05:53Z" level=info msg="Starting 1 workers"
time="2019-08-16T10:05:53Z" level=info msg="Started workers"
time="2019-08-16T10:05:53Z" level=info msg="Shutting down workers"
--- PASS: TestRun (0.50s)
=== RUN   TestSyncPdb
time="2019-08-16T10:05:53Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:05:53Z" level=info msg="Creating Job controller"
--- PASS: TestSyncPdb (0.00s)
=== RUN   TestAddTFJob
time="2019-08-16T10:05:53Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:05:53Z" level=info msg="Creating Job controller"
time="2019-08-16T10:05:53Z" level=info msg="Starting TFJob controller"
time="2019-08-16T10:05:53Z" level=info msg="Waiting for informer caches to sync"
time="2019-08-16T10:05:53Z" level=info msg="TFJob test-tfjob is created." job=default.test-tfjob uid=
time="2019-08-16T10:05:54Z" level=info msg="Starting 1 workers"
time="2019-08-16T10:05:54Z" level=info msg="Started workers"
--- PASS: TestAddTFJob (0.10s)
=== RUN   TestCopyLabelsAndAnnotation
time="2019-08-16T10:05:54Z" level=info msg="Shutting down workers"
time="2019-08-16T10:05:54Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:05:54Z" level=info msg="Creating Job controller"
time="2019-08-16T10:05:54Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-16T10:05:54Z" level=info msg="Need to create new pod: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-08-16T10:05:54Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=1, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:05:54Z" level=info msg="need to create new service: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-08-16T10:05:54Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (345.785µs)" job=default.test-tfjob
--- PASS: TestCopyLabelsAndAnnotation (0.00s)
=== RUN   TestDeletePodsAndServices
time="2019-08-16T10:05:54Z" level=info msg="Starting TFJob controller"
time="2019-08-16T10:05:54Z" level=info msg="Waiting for informer caches to sync"
time="2019-08-16T10:05:54Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:05:54Z" level=info msg="Creating Job controller"
time="2019-08-16T10:05:54Z" level=info msg="Starting 1 workers"
time="2019-08-16T10:05:54Z" level=info msg="Started workers"
time="2019-08-16T10:05:54Z" level=info msg="Shutting down workers"
time="2019-08-16T10:05:54Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-16T10:05:54Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (482.214µs)" job=default.test-tfjob
time="2019-08-16T10:05:54Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:05:54Z" level=info msg="Creating Job controller"
time="2019-08-16T10:05:54Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-16T10:05:54Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (268.041µs)" job=default.test-tfjob
time="2019-08-16T10:05:54Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:05:54Z" level=info msg="Creating Job controller"
time="2019-08-16T10:05:54Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-16T10:05:54Z" level=info msg="Ignoring inactive pod default/ps-1 in state Succeeded, deletion time <nil>"
time="2019-08-16T10:05:54Z" level=info msg="Ignoring inactive pod default/worker-0 in state Succeeded, deletion time <nil>"
time="2019-08-16T10:05:54Z" level=info msg="Ignoring inactive pod default/worker-1 in state Succeeded, deletion time <nil>"
time="2019-08-16T10:05:54Z" level=info msg="Ignoring inactive pod default/worker-2 in state Succeeded, deletion time <nil>"
time="2019-08-16T10:05:54Z" level=info msg="Ignoring inactive pod default/worker-3 in state Succeeded, deletion time <nil>"
time="2019-08-16T10:05:54Z" level=info msg="Ignoring inactive pod default/ps-0 in state Succeeded, deletion time <nil>"
time="2019-08-16T10:05:54Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (1.692408ms)" job=default.test-tfjob
time="2019-08-16T10:05:54Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:05:54Z" level=info msg="Creating Job controller"
time="2019-08-16T10:05:54Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-16T10:05:54Z" level=info msg="Ignoring inactive pod default/worker-1 in state Succeeded, deletion time <nil>"
time="2019-08-16T10:05:54Z" level=info msg="Ignoring inactive pod default/worker-2 in state Succeeded, deletion time <nil>"
time="2019-08-16T10:05:54Z" level=info msg="Ignoring inactive pod default/worker-3 in state Succeeded, deletion time <nil>"
time="2019-08-16T10:05:54Z" level=info msg="Ignoring inactive pod default/ps-0 in state Succeeded, deletion time <nil>"
time="2019-08-16T10:05:54Z" level=info msg="Ignoring inactive pod default/ps-1 in state Succeeded, deletion time <nil>"
time="2019-08-16T10:05:54Z" level=info msg="Ignoring inactive pod default/worker-0 in state Succeeded, deletion time <nil>"
time="2019-08-16T10:05:54Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (1.435608ms)" job=default.test-tfjob
--- PASS: TestDeletePodsAndServices (0.01s)
=== RUN   TestCleanupTFJob
time="2019-08-16T10:05:54Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:05:54Z" level=info msg="Creating Job controller"
time="2019-08-16T10:05:54Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-16T10:05:54Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (384.18µs)" job=default.test-tfjob
time="2019-08-16T10:05:54Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:05:54Z" level=info msg="Creating Job controller"
time="2019-08-16T10:05:54Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-16T10:05:54Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (271.617µs)" job=default.test-tfjob
time="2019-08-16T10:05:54Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:05:54Z" level=info msg="Creating Job controller"
time="2019-08-16T10:05:56Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-16T10:05:56Z" level=info msg="Ignoring inactive pod default/worker-1 in state Succeeded, deletion time <nil>"
time="2019-08-16T10:05:56Z" level=info msg="Ignoring inactive pod default/worker-2 in state Succeeded, deletion time <nil>"
time="2019-08-16T10:05:56Z" level=info msg="Ignoring inactive pod default/worker-3 in state Succeeded, deletion time <nil>"
time="2019-08-16T10:05:56Z" level=info msg="Ignoring inactive pod default/ps-0 in state Succeeded, deletion time <nil>"
time="2019-08-16T10:05:56Z" level=info msg="Ignoring inactive pod default/ps-1 in state Succeeded, deletion time <nil>"
time="2019-08-16T10:05:56Z" level=info msg="Ignoring inactive pod default/worker-0 in state Succeeded, deletion time <nil>"
time="2019-08-16T10:05:56Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (782.765µs)" job=default.test-tfjob
--- PASS: TestCleanupTFJob (2.01s)
=== RUN   TestActiveDeadlineSeconds
time="2019-08-16T10:05:56Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:05:56Z" level=info msg="Creating Job controller"
time="2019-08-16T10:05:56Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-16T10:05:56Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:05:56Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=4, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:05:56Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:05:56Z" level=info msg="Creating Job controller"
time="2019-08-16T10:05:58Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
--- PASS: TestActiveDeadlineSeconds (2.00s)
=== RUN   TestBackoffForOnFailure
time="2019-08-16T10:05:58Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:05:58Z" level=info msg="Creating Job controller"
time="2019-08-16T10:05:58Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-16T10:05:58Z" level=warning msg="The restart policy of replica PS of the job test-tfjob is not OnFailure or Always. Not counted in backoff limit." job=default.test-tfjob uid=
time="2019-08-16T10:05:58Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (451.289µs)" job=default.test-tfjob
--- PASS: TestBackoffForOnFailure (0.00s)
=== RUN   TestAddPod
time="2019-08-16T10:05:58Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:05:58Z" level=info msg="Creating Job controller"
time="2019-08-16T10:05:58Z" level=info msg="Starting TFJob controller"
time="2019-08-16T10:05:58Z" level=info msg="Waiting for informer caches to sync"
time="2019-08-16T10:05:58Z" level=info msg="Starting 1 workers"
time="2019-08-16T10:05:58Z" level=info msg="Started workers"
--- PASS: TestAddPod (0.10s)
=== RUN   TestClusterSpec
time="2019-08-16T10:05:58Z" level=info msg="Shutting down workers"
--- PASS: TestClusterSpec (0.00s)
=== RUN   TestRestartPolicy
--- PASS: TestRestartPolicy (0.00s)
=== RUN   TestExitCode
time="2019-08-16T10:05:58Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:05:58Z" level=info msg="Creating Job controller"
time="2019-08-16T10:05:58Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-16T10:05:58Z" level=info msg="Ignoring inactive pod default/worker-0 in state Failed, deletion time <nil>"
time="2019-08-16T10:05:58Z" level=info msg="Pod: default.worker-0 exited with code 130" job=default.test-tfjob replica-type=worker uid=
E0816 10:05:58.155200    9620 event.go:259] Could not construct reference to: '&v1.TFJob{TypeMeta:v1.TypeMeta{Kind:"TFJob", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"test-tfjob", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, Spec:v1.TFJobSpec{ActiveDeadlineSeconds:(*int64)(nil), BackoffLimit:(*int32)(nil), CleanPodPolicy:(*v1.CleanPodPolicy)(0xc4206163a0), TTLSecondsAfterFinished:(*int32)(nil), TFReplicaSpecs:map[v1.TFReplicaType]*v1.ReplicaSpec{"Worker":(*v1.ReplicaSpec)(0xc4208342c0)}}, Status:v1.JobStatus{Conditions:[]v1.JobCondition(nil), ReplicaStatuses:map[v1.ReplicaType]*v1.ReplicaStatus{"Worker":(*v1.ReplicaStatus)(0xc420588ee0)}, StartTime:(*v1.Time)(nil), CompletionTime:(*v1.Time)(nil), LastReconcileTime:(*v1.Time)(nil)}}' due to: 'selfLink was empty, can't make reference'. Will not report event: 'Normal' 'ExitedWithCode' 'Pod: default.worker-0 exited with code 130'
time="2019-08-16T10:05:58Z" level=info msg="Need to restart the pod: default.worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-08-16T10:05:58Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=1, running=0, failed=1" job=default.test-tfjob uid=
E0816 10:05:58.155321    9620 event.go:259] Could not construct reference to: '&v1.TFJob{TypeMeta:v1.TypeMeta{Kind:"TFJob", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"test-tfjob", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, Spec:v1.TFJobSpec{ActiveDeadlineSeconds:(*int64)(nil), BackoffLimit:(*int32)(nil), CleanPodPolicy:(*v1.CleanPodPolicy)(0xc4206163a0), TTLSecondsAfterFinished:(*int32)(nil), TFReplicaSpecs:map[v1.TFReplicaType]*v1.ReplicaSpec{"Worker":(*v1.ReplicaSpec)(0xc4208342c0)}}, Status:v1.JobStatus{Conditions:[]v1.JobCondition(nil), ReplicaStatuses:map[v1.ReplicaType]*v1.ReplicaStatus{"Worker":(*v1.ReplicaStatus)(0xc420588ee0)}, StartTime:(*v1.Time)(0xc42080a780), CompletionTime:(*v1.Time)(nil), LastReconcileTime:(*v1.Time)(nil)}}' due to: 'selfLink was empty, can't make reference'. Will not report event: 'Warning' 'TFJobRestarting' 'TFJob test-tfjob is restarting because 1 Worker replica(s) failed.'
time="2019-08-16T10:05:58Z" level=info msg="need to create new service: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-08-16T10:05:58Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (610.92µs)" job=default.test-tfjob
--- PASS: TestExitCode (0.00s)
=== RUN   TestAddService
time="2019-08-16T10:05:58Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:05:58Z" level=info msg="Creating Job controller"
time="2019-08-16T10:05:58Z" level=info msg="Starting TFJob controller"
time="2019-08-16T10:05:58Z" level=info msg="Waiting for informer caches to sync"
time="2019-08-16T10:05:58Z" level=info msg="Starting TFJob controller"
time="2019-08-16T10:05:58Z" level=info msg="Waiting for informer caches to sync"
time="2019-08-16T10:05:58Z" level=info msg="Starting 1 workers"
time="2019-08-16T10:05:58Z" level=info msg="Started workers"
time="2019-08-16T10:05:58Z" level=info msg="Shutting down workers"
time="2019-08-16T10:05:58Z" level=info msg="Starting 1 workers"
time="2019-08-16T10:05:58Z" level=info msg="Started workers"
--- PASS: TestAddService (0.10s)
=== RUN   TestFailed
time="2019-08-16T10:05:58Z" level=info msg="Shutting down workers"
time="2019-08-16T10:05:58Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:05:58Z" level=info msg="Creating Job controller"
time="2019-08-16T10:05:58Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=3, running=0, failed=1" job=default.test-tfjob uid=
E0816 10:05:58.258264    9620 event.go:259] Could not construct reference to: '&v1.TFJob{TypeMeta:v1.TypeMeta{Kind:"TFJob", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"test-tfjob", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, Spec:v1.TFJobSpec{ActiveDeadlineSeconds:(*int64)(nil), BackoffLimit:(*int32)(nil), CleanPodPolicy:(*v1.CleanPodPolicy)(nil), TTLSecondsAfterFinished:(*int32)(nil), TFReplicaSpecs:map[v1.TFReplicaType]*v1.ReplicaSpec{"Worker":(*v1.ReplicaSpec)(0xc420834dc0)}}, Status:v1.JobStatus{Conditions:[]v1.JobCondition(nil), ReplicaStatuses:map[v1.ReplicaType]*v1.ReplicaStatus{"Worker":(*v1.ReplicaStatus)(0xc4207dcd10)}, StartTime:(*v1.Time)(0xc420865780), CompletionTime:(*v1.Time)(nil), LastReconcileTime:(*v1.Time)(nil)}}' due to: 'selfLink was empty, can't make reference'. Will not report event: 'Normal' 'TFJobFailed' 'TFJob test-tfjob has failed because 1 Worker replica(s) failed.'
--- PASS: TestFailed (0.00s)
=== RUN   TestStatus
time="2019-08-16T10:05:58Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:05:58Z" level=info msg="Creating Job controller"
time="2019-08-16T10:05:58Z" level=info msg="TFJob=test-tfjob, ReplicaType=Chief expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:05:58Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:05:58Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:05:58Z" level=info msg="Creating Job controller"
time="2019-08-16T10:05:58Z" level=info msg="TFJob=test-tfjob, ReplicaType=Chief expected=1, running=1, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:05:58Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=1, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:05:58Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:05:58Z" level=info msg="Creating Job controller"
time="2019-08-16T10:05:58Z" level=info msg="TFJob=test-tfjob, ReplicaType=Chief expected=1, running=0, failed=1" job=default.test-tfjob uid=
time="2019-08-16T10:05:58Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=1, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:05:58Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:05:58Z" level=info msg="Creating Job controller"
time="2019-08-16T10:05:58Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=1, running=0, failed=1" job=default.test-tfjob uid=
time="2019-08-16T10:05:58Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:05:58Z" level=info msg="Creating Job controller"
time="2019-08-16T10:05:58Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:05:58Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:05:58Z" level=info msg="Creating Job controller"
time="2019-08-16T10:05:58Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=1, running=1, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:05:58Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:05:58Z" level=info msg="Creating Job controller"
time="2019-08-16T10:05:58Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:05:58Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:05:58Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:05:58Z" level=info msg="Creating Job controller"
time="2019-08-16T10:05:58Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=2, failed=2" job=default.test-tfjob uid=
time="2019-08-16T10:05:58Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:05:58Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:05:58Z" level=info msg="Creating Job controller"
time="2019-08-16T10:05:58Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=2, running=0, failed=2" job=default.test-tfjob uid=
time="2019-08-16T10:05:58Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:05:58Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:05:58Z" level=info msg="Creating Job controller"
time="2019-08-16T10:05:58Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=3, running=3, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:05:58Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:05:58Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:05:58Z" level=info msg="Creating Job controller"
time="2019-08-16T10:05:58Z" level=info msg="TFJob=test-tfjob, ReplicaType=Chief expected=1, running=1, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:05:58Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=0, failed=4" job=default.test-tfjob uid=
time="2019-08-16T10:05:58Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:05:58Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:05:58Z" level=info msg="Creating Job controller"
time="2019-08-16T10:05:58Z" level=info msg="TFJob=test-tfjob, ReplicaType=Chief expected=1, running=1, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:05:58Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:05:58Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:05:58Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:05:58Z" level=info msg="Creating Job controller"
time="2019-08-16T10:05:58Z" level=info msg="TFJob=test-tfjob, ReplicaType=Chief expected=1, running=1, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:05:58Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:05:58Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=1, failed=1" job=default.test-tfjob uid=
time="2019-08-16T10:05:58Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:05:58Z" level=info msg="Creating Job controller"
time="2019-08-16T10:05:58Z" level=info msg="TFJob=test-tfjob, ReplicaType=Chief expected=1, running=0, failed=1" job=default.test-tfjob uid=
time="2019-08-16T10:05:58Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:05:58Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:05:58Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:05:58Z" level=info msg="Creating Job controller"
time="2019-08-16T10:05:58Z" level=info msg="TFJob=test-tfjob, ReplicaType=Chief expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:05:58Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=0, failed=4" job=default.test-tfjob uid=
time="2019-08-16T10:05:58Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:05:58Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:05:58Z" level=info msg="Creating Job controller"
time="2019-08-16T10:05:58Z" level=info msg="TFJob=test-tfjob, ReplicaType=Chief expected=1, running=0, failed=1" job=default.test-tfjob uid=
time="2019-08-16T10:05:58Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=0, failed=4" job=default.test-tfjob uid=
time="2019-08-16T10:05:58Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
--- PASS: TestStatus (0.02s)
=== RUN   TestGenOwnerReference
--- PASS: TestGenOwnerReference (0.00s)
=== RUN   TestGenLabels
--- PASS: TestGenLabels (0.00s)
=== RUN   TestConvertTFJobToUnstructured
--- PASS: TestConvertTFJobToUnstructured (0.00s)
PASS
coverage: 44.0% of statements in github.com/kubeflow/tf-operator/pkg/apis/common/v1, github.com/kubeflow/tf-operator/pkg/apis/common/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/controller.v1beta2/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
ok  	github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow	4.936s	coverage: 44.0% of statements in github.com/kubeflow/tf-operator/pkg/apis/common/v1, github.com/kubeflow/tf-operator/pkg/apis/common/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/controller.v1beta2/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
=== RUN   TestNormalPath
time="2019-08-16T10:06:04Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:06:04Z" level=info msg="Creating Job controller"
time="2019-08-16T10:06:04Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-16T10:06:04Z" level=info msg="Need to create new pod: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2019-08-16T10:06:04Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=1, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:06:04Z" level=info msg="need to create new service: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2019-08-16T10:06:04Z" level=info msg="Need to create new pod: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2019-08-16T10:06:04Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:06:04Z" level=info msg="need to create new service: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2019-08-16T10:06:04Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (1.994551ms)" job=default.test-tfjob
time="2019-08-16T10:06:04Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:06:04Z" level=info msg="Creating Job controller"
time="2019-08-16T10:06:04Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-16T10:06:04Z" level=info msg="Ignoring inactive pod default/worker-0 in state Succeeded, deletion time <nil>"
time="2019-08-16T10:06:04Z" level=info msg="Ignoring inactive pod default/worker-1 in state Succeeded, deletion time <nil>"
time="2019-08-16T10:06:04Z" level=info msg="Ignoring inactive pod default/worker-2 in state Succeeded, deletion time <nil>"
time="2019-08-16T10:06:04Z" level=info msg="Ignoring inactive pod default/worker-3 in state Succeeded, deletion time <nil>"
time="2019-08-16T10:06:04Z" level=info msg="Ignoring inactive pod default/ps-0 in state Succeeded, deletion time <nil>"
time="2019-08-16T10:06:04Z" level=info msg="Ignoring inactive pod default/ps-1 in state Succeeded, deletion time <nil>"
time="2019-08-16T10:06:04Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:06:04Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=0, running=0, failed=0" job=default.test-tfjob uid=
E0816 10:06:04.415111    9731 event.go:259] Could not construct reference to: '&v1beta2.TFJob{TypeMeta:v1.TypeMeta{Kind:"TFJob", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"test-tfjob", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, Spec:v1beta2.TFJobSpec{ActiveDeadlineSeconds:(*int64)(nil), BackoffLimit:(*int32)(nil), CleanPodPolicy:(*v1beta2.CleanPodPolicy)(0xc42040a2a0), TTLSecondsAfterFinished:(*int32)(nil), TFReplicaSpecs:map[v1beta2.TFReplicaType]*v1beta2.ReplicaSpec{"PS":(*v1beta2.ReplicaSpec)(0xc4202ac000), "Worker":(*v1beta2.ReplicaSpec)(0xc4202ad8c0)}}, Status:v1beta2.JobStatus{Conditions:[]v1beta2.JobCondition(nil), ReplicaStatuses:map[v1beta2.ReplicaType]*v1beta2.ReplicaStatus{"PS":(*v1beta2.ReplicaStatus)(0xc4204bafe0), "Worker":(*v1beta2.ReplicaStatus)(0xc4204bb010)}, StartTime:(*v1.Time)(0xc4202780c0), CompletionTime:(*v1.Time)(nil), LastReconcileTime:(*v1.Time)(nil)}}' due to: 'selfLink was empty, can't make reference'. Will not report event: 'Normal' 'TFJobSucceeded' 'TFJob test-tfjob successfully completed.'
time="2019-08-16T10:06:04Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (1.134441ms)" job=default.test-tfjob
time="2019-08-16T10:06:04Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:06:04Z" level=info msg="Creating Job controller"
time="2019-08-16T10:06:04Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-16T10:06:04Z" level=info msg="Need to create new pod: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-08-16T10:06:04Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=1, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:06:04Z" level=info msg="need to create new service: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-08-16T10:06:04Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (275.697µs)" job=default.test-tfjob
time="2019-08-16T10:06:04Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:06:04Z" level=info msg="Creating Job controller"
time="2019-08-16T10:06:04Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-16T10:06:04Z" level=info msg="Need to create new pod: ps-0" job=default.test-tfjob replica-type=ps uid=
time="2019-08-16T10:06:04Z" level=info msg="Need to create new pod: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2019-08-16T10:06:04Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:06:04Z" level=info msg="need to create new service: ps-0" job=default.test-tfjob replica-type=ps uid=
time="2019-08-16T10:06:04Z" level=info msg="need to create new service: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2019-08-16T10:06:04Z" level=info msg="Need to create new pod: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-08-16T10:06:04Z" level=info msg="Need to create new pod: worker-1" job=default.test-tfjob replica-type=worker uid=
time="2019-08-16T10:06:04Z" level=info msg="Need to create new pod: worker-2" job=default.test-tfjob replica-type=worker uid=
time="2019-08-16T10:06:04Z" level=info msg="Need to create new pod: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2019-08-16T10:06:04Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:06:04Z" level=info msg="need to create new service: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-08-16T10:06:04Z" level=info msg="need to create new service: worker-1" job=default.test-tfjob replica-type=worker uid=
time="2019-08-16T10:06:04Z" level=info msg="need to create new service: worker-2" job=default.test-tfjob replica-type=worker uid=
time="2019-08-16T10:06:04Z" level=info msg="need to create new service: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2019-08-16T10:06:04Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (1.728773ms)" job=default.test-tfjob
time="2019-08-16T10:06:04Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:06:04Z" level=info msg="Creating Job controller"
time="2019-08-16T10:06:04Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-16T10:06:04Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:06:04Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:06:04Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (719.03µs)" job=default.test-tfjob
time="2019-08-16T10:06:04Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:06:04Z" level=info msg="Creating Job controller"
time="2019-08-16T10:06:04Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-16T10:06:04Z" level=info msg="Need to create new pod: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2019-08-16T10:06:04Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:06:04Z" level=info msg="need to create new service: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2019-08-16T10:06:04Z" level=info msg="Need to create new pod: worker-2" job=default.test-tfjob replica-type=worker uid=
time="2019-08-16T10:06:04Z" level=info msg="Need to create new pod: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2019-08-16T10:06:04Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:06:04Z" level=info msg="need to create new service: worker-2" job=default.test-tfjob replica-type=worker uid=
time="2019-08-16T10:06:04Z" level=info msg="need to create new service: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2019-08-16T10:06:04Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (1.38181ms)" job=default.test-tfjob
time="2019-08-16T10:06:04Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:06:04Z" level=info msg="Creating Job controller"
time="2019-08-16T10:06:04Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-16T10:06:04Z" level=info msg="Ignoring inactive pod default/worker-2 in state Succeeded, deletion time <nil>"
time="2019-08-16T10:06:04Z" level=info msg="Need to create new pod: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2019-08-16T10:06:04Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:06:04Z" level=info msg="need to create new service: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2019-08-16T10:06:04Z" level=info msg="Need to create new pod: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2019-08-16T10:06:04Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=3, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:06:04Z" level=info msg="need to create new service: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2019-08-16T10:06:04Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (1.463741ms)" job=default.test-tfjob
time="2019-08-16T10:06:04Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:06:04Z" level=info msg="Creating Job controller"
time="2019-08-16T10:06:04Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-16T10:06:04Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=4, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:06:04Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:06:04Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (611.922µs)" job=default.test-tfjob
--- PASS: TestNormalPath (0.03s)
=== RUN   TestRun
time="2019-08-16T10:06:04Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:06:04Z" level=info msg="Creating Job controller"
time="2019-08-16T10:06:04Z" level=info msg="Starting TFJob controller"
time="2019-08-16T10:06:04Z" level=info msg="Waiting for informer caches to sync"
time="2019-08-16T10:06:04Z" level=info msg="Starting 1 workers"
time="2019-08-16T10:06:04Z" level=info msg="Started workers"
time="2019-08-16T10:06:04Z" level=info msg="Shutting down workers"
--- PASS: TestRun (0.50s)
=== RUN   TestSyncPdb
time="2019-08-16T10:06:04Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:06:04Z" level=info msg="Creating Job controller"
--- PASS: TestSyncPdb (0.00s)
=== RUN   TestAddTFJob
time="2019-08-16T10:06:04Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:06:04Z" level=info msg="Creating Job controller"
time="2019-08-16T10:06:04Z" level=info msg="Starting TFJob controller"
time="2019-08-16T10:06:04Z" level=info msg="Waiting for informer caches to sync"
time="2019-08-16T10:06:04Z" level=info msg="TFJob test-tfjob is created." job=default.test-tfjob uid=
time="2019-08-16T10:06:05Z" level=info msg="Starting 1 workers"
time="2019-08-16T10:06:05Z" level=info msg="Started workers"
--- PASS: TestAddTFJob (0.10s)
=== RUN   TestCopyLabelsAndAnnotation
time="2019-08-16T10:06:05Z" level=info msg="Shutting down workers"
time="2019-08-16T10:06:05Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:06:05Z" level=info msg="Creating Job controller"
time="2019-08-16T10:06:05Z" level=info msg="Starting TFJob controller"
time="2019-08-16T10:06:05Z" level=info msg="Waiting for informer caches to sync"
time="2019-08-16T10:06:05Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-16T10:06:05Z" level=info msg="Need to create new pod: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-08-16T10:06:05Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=1, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:06:05Z" level=info msg="need to create new service: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-08-16T10:06:05Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (926.735µs)" job=default.test-tfjob
--- PASS: TestCopyLabelsAndAnnotation (0.00s)
=== RUN   TestDeletePodsAndServices
time="2019-08-16T10:06:05Z" level=info msg="Starting 1 workers"
time="2019-08-16T10:06:05Z" level=info msg="Started workers"
time="2019-08-16T10:06:05Z" level=info msg="Shutting down workers"
time="2019-08-16T10:06:05Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:06:05Z" level=info msg="Creating Job controller"
time="2019-08-16T10:06:05Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-16T10:06:05Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=4, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:06:05Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:06:05Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (821.349µs)" job=default.test-tfjob
time="2019-08-16T10:06:05Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:06:05Z" level=info msg="Creating Job controller"
time="2019-08-16T10:06:05Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-16T10:06:05Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=4, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:06:05Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:06:05Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (883.102µs)" job=default.test-tfjob
time="2019-08-16T10:06:05Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:06:05Z" level=info msg="Creating Job controller"
time="2019-08-16T10:06:05Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-16T10:06:05Z" level=info msg="Ignoring inactive pod default/worker-1 in state Succeeded, deletion time <nil>"
time="2019-08-16T10:06:05Z" level=info msg="Ignoring inactive pod default/worker-2 in state Succeeded, deletion time <nil>"
time="2019-08-16T10:06:05Z" level=info msg="Ignoring inactive pod default/worker-3 in state Succeeded, deletion time <nil>"
time="2019-08-16T10:06:05Z" level=info msg="Ignoring inactive pod default/ps-0 in state Succeeded, deletion time <nil>"
time="2019-08-16T10:06:05Z" level=info msg="Ignoring inactive pod default/ps-1 in state Succeeded, deletion time <nil>"
time="2019-08-16T10:06:05Z" level=info msg="Ignoring inactive pod default/worker-0 in state Succeeded, deletion time <nil>"
time="2019-08-16T10:06:05Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:06:05Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:06:05Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (608.377µs)" job=default.test-tfjob
time="2019-08-16T10:06:05Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:06:05Z" level=info msg="Creating Job controller"
time="2019-08-16T10:06:05Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-16T10:06:05Z" level=info msg="Ignoring inactive pod default/worker-0 in state Succeeded, deletion time <nil>"
time="2019-08-16T10:06:05Z" level=info msg="Ignoring inactive pod default/worker-1 in state Succeeded, deletion time <nil>"
time="2019-08-16T10:06:05Z" level=info msg="Ignoring inactive pod default/worker-2 in state Succeeded, deletion time <nil>"
time="2019-08-16T10:06:05Z" level=info msg="Ignoring inactive pod default/worker-3 in state Succeeded, deletion time <nil>"
time="2019-08-16T10:06:05Z" level=info msg="Ignoring inactive pod default/ps-0 in state Succeeded, deletion time <nil>"
time="2019-08-16T10:06:05Z" level=info msg="Ignoring inactive pod default/ps-1 in state Succeeded, deletion time <nil>"
time="2019-08-16T10:06:05Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:06:05Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:06:05Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (660.617µs)" job=default.test-tfjob
--- PASS: TestDeletePodsAndServices (0.01s)
=== RUN   TestCleanupTFJob
time="2019-08-16T10:06:05Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:06:05Z" level=info msg="Creating Job controller"
time="2019-08-16T10:06:05Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-16T10:06:05Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:06:05Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=4, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:06:05Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (508.61µs)" job=default.test-tfjob
time="2019-08-16T10:06:05Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:06:05Z" level=info msg="Creating Job controller"
time="2019-08-16T10:06:05Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-16T10:06:05Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:06:05Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=4, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:06:05Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (558.947µs)" job=default.test-tfjob
time="2019-08-16T10:06:05Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:06:05Z" level=info msg="Creating Job controller"
time="2019-08-16T10:06:07Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-16T10:06:07Z" level=info msg="Ignoring inactive pod default/worker-2 in state Succeeded, deletion time <nil>"
time="2019-08-16T10:06:07Z" level=info msg="Ignoring inactive pod default/worker-3 in state Succeeded, deletion time <nil>"
time="2019-08-16T10:06:07Z" level=info msg="Ignoring inactive pod default/ps-0 in state Succeeded, deletion time <nil>"
time="2019-08-16T10:06:07Z" level=info msg="Ignoring inactive pod default/ps-1 in state Succeeded, deletion time <nil>"
time="2019-08-16T10:06:07Z" level=info msg="Ignoring inactive pod default/worker-0 in state Succeeded, deletion time <nil>"
time="2019-08-16T10:06:07Z" level=info msg="Ignoring inactive pod default/worker-1 in state Succeeded, deletion time <nil>"
time="2019-08-16T10:06:07Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:06:07Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:06:07Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (1.333994ms)" job=default.test-tfjob
--- PASS: TestCleanupTFJob (2.01s)
=== RUN   TestActiveDeadlineSeconds
time="2019-08-16T10:06:07Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:06:07Z" level=info msg="Creating Job controller"
time="2019-08-16T10:06:07Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-16T10:06:07Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:06:07Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=4, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:06:07Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:06:07Z" level=info msg="Creating Job controller"
time="2019-08-16T10:06:09Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
--- PASS: TestActiveDeadlineSeconds (2.00s)
=== RUN   TestBackoffForOnFailure
time="2019-08-16T10:06:09Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:06:09Z" level=info msg="Creating Job controller"
time="2019-08-16T10:06:09Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-16T10:06:09Z" level=warning msg="The restart policy of replica PS of the job test-tfjob is not OnFailure or Always. Not counted in backoff limit." job=default.test-tfjob uid=
time="2019-08-16T10:06:09Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (410.309µs)" job=default.test-tfjob
--- PASS: TestBackoffForOnFailure (0.00s)
=== RUN   TestAddPod
time="2019-08-16T10:06:09Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:06:09Z" level=info msg="Creating Job controller"
time="2019-08-16T10:06:09Z" level=info msg="Starting TFJob controller"
time="2019-08-16T10:06:09Z" level=info msg="Waiting for informer caches to sync"
time="2019-08-16T10:06:09Z" level=info msg="Starting 1 workers"
time="2019-08-16T10:06:09Z" level=info msg="Started workers"
--- PASS: TestAddPod (0.10s)
=== RUN   TestClusterSpec
--- PASS: TestClusterSpec (0.00s)
=== RUN   TestRestartPolicy
--- PASS: TestRestartPolicy (0.00s)
=== RUN   TestExitCode
time="2019-08-16T10:06:09Z" level=info msg="Shutting down workers"
time="2019-08-16T10:06:09Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:06:09Z" level=info msg="Creating Job controller"
time="2019-08-16T10:06:09Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-16T10:06:09Z" level=info msg="Starting TFJob controller"
time="2019-08-16T10:06:09Z" level=info msg="Waiting for informer caches to sync"
time="2019-08-16T10:06:09Z" level=info msg="Ignoring inactive pod default/worker-0 in state Failed, deletion time <nil>"
time="2019-08-16T10:06:09Z" level=info msg="Pod: default.worker-0 exited with code 130" job=default.test-tfjob replica-type=worker uid=
E0816 10:06:09.166161    9731 event.go:259] Could not construct reference to: '&v1beta2.TFJob{TypeMeta:v1.TypeMeta{Kind:"TFJob", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"test-tfjob", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, Spec:v1beta2.TFJobSpec{ActiveDeadlineSeconds:(*int64)(nil), BackoffLimit:(*int32)(nil), CleanPodPolicy:(*v1beta2.CleanPodPolicy)(0xc42052c390), TTLSecondsAfterFinished:(*int32)(nil), TFReplicaSpecs:map[v1beta2.TFReplicaType]*v1beta2.ReplicaSpec{"Worker":(*v1beta2.ReplicaSpec)(0xc420012840)}}, Status:v1beta2.JobStatus{Conditions:[]v1beta2.JobCondition(nil), ReplicaStatuses:map[v1beta2.ReplicaType]*v1beta2.ReplicaStatus{"Worker":(*v1beta2.ReplicaStatus)(0xc4201174c0)}, StartTime:(*v1.Time)(nil), CompletionTime:(*v1.Time)(nil), LastReconcileTime:(*v1.Time)(nil)}}' due to: 'selfLink was empty, can't make reference'. Will not report event: 'Normal' 'ExitedWithCode' 'Pod: default.worker-0 exited with code 130'
time="2019-08-16T10:06:09Z" level=info msg="Need to restart the pod: default.worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-08-16T10:06:09Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=1, running=0, failed=1" job=default.test-tfjob uid=
E0816 10:06:09.166329    9731 event.go:259] Could not construct reference to: '&v1beta2.TFJob{TypeMeta:v1.TypeMeta{Kind:"TFJob", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"test-tfjob", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, Spec:v1beta2.TFJobSpec{ActiveDeadlineSeconds:(*int64)(nil), BackoffLimit:(*int32)(nil), CleanPodPolicy:(*v1beta2.CleanPodPolicy)(0xc42052c390), TTLSecondsAfterFinished:(*int32)(nil), TFReplicaSpecs:map[v1beta2.TFReplicaType]*v1beta2.ReplicaSpec{"Worker":(*v1beta2.ReplicaSpec)(0xc420012840)}}, Status:v1beta2.JobStatus{Conditions:[]v1beta2.JobCondition(nil), ReplicaStatuses:map[v1beta2.ReplicaType]*v1beta2.ReplicaStatus{"Worker":(*v1beta2.ReplicaStatus)(0xc4201174c0)}, StartTime:(*v1.Time)(0xc4204ad9c0), CompletionTime:(*v1.Time)(nil), LastReconcileTime:(*v1.Time)(nil)}}' due to: 'selfLink was empty, can't make reference'. Will not report event: 'Warning' 'TFJobRestarting' 'TFJob test-tfjob is restarting because 1 Worker replica(s) failed.'
time="2019-08-16T10:06:09Z" level=info msg="need to create new service: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-08-16T10:06:09Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (839.179µs)" job=default.test-tfjob
--- PASS: TestExitCode (0.00s)
=== RUN   TestAddService
time="2019-08-16T10:06:09Z" level=info msg="Starting 1 workers"
time="2019-08-16T10:06:09Z" level=info msg="Started workers"
time="2019-08-16T10:06:09Z" level=info msg="Shutting down workers"
time="2019-08-16T10:06:09Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:06:09Z" level=info msg="Creating Job controller"
time="2019-08-16T10:06:09Z" level=info msg="Starting TFJob controller"
time="2019-08-16T10:06:09Z" level=info msg="Waiting for informer caches to sync"
time="2019-08-16T10:06:09Z" level=info msg="Starting 1 workers"
time="2019-08-16T10:06:09Z" level=info msg="Started workers"
--- PASS: TestAddService (0.10s)
=== RUN   TestFailed
time="2019-08-16T10:06:09Z" level=info msg="Shutting down workers"
time="2019-08-16T10:06:09Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:06:09Z" level=info msg="Creating Job controller"
time="2019-08-16T10:06:09Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=3, running=0, failed=1" job=default.test-tfjob uid=
E0816 10:06:09.270047    9731 event.go:259] Could not construct reference to: '&v1beta2.TFJob{TypeMeta:v1.TypeMeta{Kind:"TFJob", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"test-tfjob", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, Spec:v1beta2.TFJobSpec{ActiveDeadlineSeconds:(*int64)(nil), BackoffLimit:(*int32)(nil), CleanPodPolicy:(*v1beta2.CleanPodPolicy)(nil), TTLSecondsAfterFinished:(*int32)(nil), TFReplicaSpecs:map[v1beta2.TFReplicaType]*v1beta2.ReplicaSpec{"Worker":(*v1beta2.ReplicaSpec)(0xc420013340)}}, Status:v1beta2.JobStatus{Conditions:[]v1beta2.JobCondition(nil), ReplicaStatuses:map[v1beta2.ReplicaType]*v1beta2.ReplicaStatus{"Worker":(*v1beta2.ReplicaStatus)(0xc4205e2e00)}, StartTime:(*v1.Time)(0xc420879460), CompletionTime:(*v1.Time)(nil), LastReconcileTime:(*v1.Time)(nil)}}' due to: 'selfLink was empty, can't make reference'. Will not report event: 'Normal' 'TFJobFailed' 'TFJob test-tfjob has failed because 1 Worker replica(s) failed.'
--- PASS: TestFailed (0.00s)
=== RUN   TestStatus
time="2019-08-16T10:06:09Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:06:09Z" level=info msg="Creating Job controller"
time="2019-08-16T10:06:09Z" level=info msg="TFJob=test-tfjob, ReplicaType=Chief expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:06:09Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:06:09Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:06:09Z" level=info msg="Creating Job controller"
time="2019-08-16T10:06:09Z" level=info msg="TFJob=test-tfjob, ReplicaType=Chief expected=1, running=1, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:06:09Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=1, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:06:09Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:06:09Z" level=info msg="Creating Job controller"
time="2019-08-16T10:06:09Z" level=info msg="TFJob=test-tfjob, ReplicaType=Chief expected=1, running=0, failed=1" job=default.test-tfjob uid=
time="2019-08-16T10:06:09Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=1, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:06:09Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:06:09Z" level=info msg="Creating Job controller"
time="2019-08-16T10:06:09Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=1, running=0, failed=1" job=default.test-tfjob uid=
time="2019-08-16T10:06:09Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:06:09Z" level=info msg="Creating Job controller"
time="2019-08-16T10:06:09Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:06:09Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:06:09Z" level=info msg="Creating Job controller"
time="2019-08-16T10:06:09Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=1, running=1, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:06:09Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:06:09Z" level=info msg="Creating Job controller"
time="2019-08-16T10:06:09Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:06:09Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:06:09Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:06:09Z" level=info msg="Creating Job controller"
time="2019-08-16T10:06:09Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=2, failed=2" job=default.test-tfjob uid=
time="2019-08-16T10:06:09Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:06:09Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:06:09Z" level=info msg="Creating Job controller"
time="2019-08-16T10:06:09Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=2, running=0, failed=2" job=default.test-tfjob uid=
time="2019-08-16T10:06:09Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:06:09Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:06:09Z" level=info msg="Creating Job controller"
time="2019-08-16T10:06:09Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=3, running=3, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:06:09Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:06:09Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:06:09Z" level=info msg="Creating Job controller"
time="2019-08-16T10:06:09Z" level=info msg="TFJob=test-tfjob, ReplicaType=Chief expected=1, running=1, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:06:09Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=0, failed=4" job=default.test-tfjob uid=
time="2019-08-16T10:06:09Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:06:09Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:06:09Z" level=info msg="Creating Job controller"
time="2019-08-16T10:06:09Z" level=info msg="TFJob=test-tfjob, ReplicaType=Chief expected=1, running=1, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:06:09Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:06:09Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:06:09Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:06:09Z" level=info msg="Creating Job controller"
time="2019-08-16T10:06:09Z" level=info msg="TFJob=test-tfjob, ReplicaType=Chief expected=1, running=1, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:06:09Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:06:09Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=1, failed=1" job=default.test-tfjob uid=
time="2019-08-16T10:06:09Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:06:09Z" level=info msg="Creating Job controller"
time="2019-08-16T10:06:09Z" level=info msg="TFJob=test-tfjob, ReplicaType=Chief expected=1, running=0, failed=1" job=default.test-tfjob uid=
time="2019-08-16T10:06:09Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:06:09Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:06:09Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:06:09Z" level=info msg="Creating Job controller"
time="2019-08-16T10:06:09Z" level=info msg="TFJob=test-tfjob, ReplicaType=Chief expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:06:09Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=0, failed=4" job=default.test-tfjob uid=
time="2019-08-16T10:06:09Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:06:09Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:06:09Z" level=info msg="Creating Job controller"
time="2019-08-16T10:06:09Z" level=info msg="TFJob=test-tfjob, ReplicaType=Chief expected=1, running=0, failed=1" job=default.test-tfjob uid=
time="2019-08-16T10:06:09Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=0, failed=4" job=default.test-tfjob uid=
time="2019-08-16T10:06:09Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
--- PASS: TestStatus (0.02s)
=== RUN   TestGenOwnerReference
--- PASS: TestGenOwnerReference (0.00s)
=== RUN   TestGenLabels
--- PASS: TestGenLabels (0.00s)
=== RUN   TestConvertTFJobToUnstructured
--- PASS: TestConvertTFJobToUnstructured (0.00s)
PASS
coverage: 43.9% of statements in github.com/kubeflow/tf-operator/pkg/apis/common/v1, github.com/kubeflow/tf-operator/pkg/apis/common/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/controller.v1beta2/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
ok  	github.com/kubeflow/tf-operator/pkg/controller.v1beta2/tensorflow	4.932s	coverage: 43.9% of statements in github.com/kubeflow/tf-operator/pkg/apis/common/v1, github.com/kubeflow/tf-operator/pkg/apis/common/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/controller.v1beta2/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
?   	github.com/kubeflow/tf-operator/pkg/logger	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/util	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/util/k8sutil	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/util/signals	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/util/train	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/version	[no test files]
ignoring pkg/apis/common/v1/openapi_generated.go
ignoring pkg/apis/common/v1/zz_generated.deepcopy.go
ignoring pkg/apis/common/v1/zz_generated.defaults.go
ignoring pkg/apis/tensorflow/v1/openapi_generated.go
ignoring pkg/apis/tensorflow/v1/zz_generated.deepcopy.go
ignoring pkg/apis/tensorflow/v1/zz_generated.defaults.go
Job #2541.1
https://coveralls.io/jobs/52098762
hack/verify-codegen.sh
Generating deepcopy funcs
Generating deepcopy funcs
Generating clientset for tensorflow:v1beta2,v1 at github.com/kubeflow/tf-operator/pkg/client/clientset
Generating listers for tensorflow:v1beta2,v1 at github.com/kubeflow/tf-operator/pkg/client/listers
Generating informers for tensorflow:v1beta2,v1 at github.com/kubeflow/tf-operator/pkg/client/informers
Generating defaulters for tensorflow/v1beta2
Generating OpenAPI specification for common/v1beta2
hack/../hack/update-codegen.sh: line 49: /home/travis/gopath/bin/openapi-gen: No such file or directory
TravisBuddy Request Identifier: 82fb0eb0-c00d-11e9-aed2-2b81c1e31580

@TravisBuddy
Copy link

Travis tests have failed

Hey @ohmystack,
Please read the following log in order to understand the failure reason.
It'll be awesome if you fix what's wrong and commit the changes.

1st Build

View build log

hack/verify-codegen.sh
Generating deepcopy funcs
Generating deepcopy funcs
Generating clientset for tensorflow:v1beta2,v1 at github.com/kubeflow/tf-operator/pkg/client/clientset
Generating listers for tensorflow:v1beta2,v1 at github.com/kubeflow/tf-operator/pkg/client/listers
Generating informers for tensorflow:v1beta2,v1 at github.com/kubeflow/tf-operator/pkg/client/informers
Generating defaulters for tensorflow/v1beta2
Generating OpenAPI specification for common/v1beta2
hack/../hack/update-codegen.sh: line 49: /home/travis/gopath/bin/openapi-gen: No such file or directory
gometalinter --config=linter_config.json --vendor ./...
pkg/controller.v1/tensorflow/controller.go:121::warning: Errors unhandled.,LOW,HIGH (gosec)
pkg/controller.v1beta2/tensorflow/controller.go:114::warning: Errors unhandled.,LOW,HIGH (gosec)
pkg/controller.v1/tensorflow/controller.go:121:25:warning: error return value not checked (tfjobscheme.AddToScheme(scheme.Scheme)) (errcheck)
pkg/controller.v1beta2/tensorflow/controller.go:114:25:warning: error return value not checked (tfjobscheme.AddToScheme(scheme.Scheme)) (errcheck)
goveralls -service=travis-ci -v -package ./pkg/... -ignore "pkg/client/*/*.go,pkg/client/*/*/*.go,pkg/client/*/*/*/*.go,pkg/client/*/*/*/*/*.go,pkg/client/*/*/*/*/*/*.go,pkg/client/*/*/*/*/*/*/*.go,pkg/util/testutil/*.go,pkg/apis/tensorflow/*/zz_generated.*.go,pkg/apis/tensorflow/*/*_generated.go,pkg/apis/common/*/zz_generated.*.go,pkg/apis/common/*/*_generated.go"
?   	github.com/kubeflow/tf-operator/pkg/apis/common/v1	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/apis/common/v1beta2	[no test files]
=== RUN   TestSetTypeNames
--- PASS: TestSetTypeNames (0.00s)
=== RUN   TestSetDefaultTFJob
--- PASS: TestSetDefaultTFJob (0.00s)
=== RUN   TestIsChieforMaster
--- PASS: TestIsChieforMaster (0.00s)
PASS
coverage: 20.9% of statements in github.com/kubeflow/tf-operator/pkg/apis/common/v1, github.com/kubeflow/tf-operator/pkg/apis/common/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/controller.v1beta2/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
ok  	github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1	0.034s	coverage: 20.9% of statements in github.com/kubeflow/tf-operator/pkg/apis/common/v1, github.com/kubeflow/tf-operator/pkg/apis/common/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/controller.v1beta2/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
=== RUN   TestSetTypeNames
--- PASS: TestSetTypeNames (0.00s)
=== RUN   TestSetDefaultTFJob
--- PASS: TestSetDefaultTFJob (0.00s)
=== RUN   TestIsChieforMaster
--- PASS: TestIsChieforMaster (0.00s)
PASS
coverage: 20.9% of statements in github.com/kubeflow/tf-operator/pkg/apis/common/v1, github.com/kubeflow/tf-operator/pkg/apis/common/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/controller.v1beta2/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
ok  	github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1beta2	0.033s	coverage: 20.9% of statements in github.com/kubeflow/tf-operator/pkg/apis/common/v1, github.com/kubeflow/tf-operator/pkg/apis/common/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/controller.v1beta2/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
=== RUN   TestValidateBetaTwoTFJobSpec
time="2019-08-16T10:12:30Z" level=error msg="TFJobSpec is not valid: Image is undefined in the container of Worker"
time="2019-08-16T10:12:30Z" level=error msg="TFJobSpec is not valid: There is no container named tensorflow in Worker"
--- PASS: TestValidateBetaTwoTFJobSpec (0.00s)
=== RUN   TestValidateV1TFJobSpec
time="2019-08-16T10:12:30Z" level=error msg="TFJobSpec is not valid: Image is undefined in the container of Worker"
time="2019-08-16T10:12:30Z" level=error msg="TFJobSpec is not valid: There is no container named tensorflow in Worker"
--- PASS: TestValidateV1TFJobSpec (0.00s)
PASS
coverage: 14.2% of statements in github.com/kubeflow/tf-operator/pkg/apis/common/v1, github.com/kubeflow/tf-operator/pkg/apis/common/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/controller.v1beta2/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
ok  	github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation	0.031s	coverage: 14.2% of statements in github.com/kubeflow/tf-operator/pkg/apis/common/v1, github.com/kubeflow/tf-operator/pkg/apis/common/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/controller.v1beta2/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
?   	github.com/kubeflow/tf-operator/pkg/client/clientset/versioned	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2/fake	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/informers/externalversions	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1beta2	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1beta2	[no test files]
=== RUN   TestGenGeneralName
--- PASS: TestGenGeneralName (0.00s)
PASS
coverage: 0.4% of statements in github.com/kubeflow/tf-operator/pkg/apis/common/v1, github.com/kubeflow/tf-operator/pkg/apis/common/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/controller.v1beta2/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
ok  	github.com/kubeflow/tf-operator/pkg/common/jobcontroller	0.022s	coverage: 0.4% of statements in github.com/kubeflow/tf-operator/pkg/apis/common/v1, github.com/kubeflow/tf-operator/pkg/apis/common/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/controller.v1beta2/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
?   	github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/testutil	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/unstructured	[no test files]
=== RUN   TestCreatePods
--- PASS: TestCreatePods (0.01s)
=== RUN   TestCreateService
time="2019-08-16T10:12:48Z" level=info msg="Controller test-tfjob created service empty_service"
--- PASS: TestCreateService (0.00s)
=== RUN   TestCreateServicesWithControllerRef
time="2019-08-16T10:12:48Z" level=info msg="Controller test-tfjob created service empty_service"
--- PASS: TestCreateServicesWithControllerRef (0.00s)
=== RUN   TestClaimServices
--- PASS: TestClaimServices (0.00s)
PASS
coverage: 41.1% of statements in github.com/kubeflow/tf-operator/pkg/apis/common/v1, github.com/kubeflow/tf-operator/pkg/apis/common/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/controller.v1beta2/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
ok  	github.com/kubeflow/tf-operator/pkg/control	0.060s	coverage: 41.1% of statements in github.com/kubeflow/tf-operator/pkg/apis/common/v1, github.com/kubeflow/tf-operator/pkg/apis/common/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/controller.v1beta2/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
=== RUN   TestNormalPath
time="2019-08-16T10:12:55Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:12:55Z" level=info msg="Creating Job controller"
time="2019-08-16T10:12:55Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-16T10:12:55Z" level=info msg="Need to create new pod: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-08-16T10:12:55Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=1, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:12:55Z" level=info msg="need to create new service: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-08-16T10:12:55Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (2.587733ms)" job=default.test-tfjob
time="2019-08-16T10:12:55Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:12:55Z" level=info msg="Creating Job controller"
time="2019-08-16T10:12:55Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-16T10:12:55Z" level=info msg="Need to create new pod: ps-0" job=default.test-tfjob replica-type=ps uid=
time="2019-08-16T10:12:55Z" level=info msg="Need to create new pod: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2019-08-16T10:12:55Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:12:55Z" level=info msg="need to create new service: ps-0" job=default.test-tfjob replica-type=ps uid=
time="2019-08-16T10:12:55Z" level=info msg="need to create new service: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2019-08-16T10:12:55Z" level=info msg="Need to create new pod: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-08-16T10:12:55Z" level=info msg="Need to create new pod: worker-1" job=default.test-tfjob replica-type=worker uid=
time="2019-08-16T10:12:55Z" level=info msg="Need to create new pod: worker-2" job=default.test-tfjob replica-type=worker uid=
time="2019-08-16T10:12:55Z" level=info msg="Need to create new pod: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2019-08-16T10:12:55Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:12:55Z" level=info msg="need to create new service: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-08-16T10:12:55Z" level=info msg="need to create new service: worker-1" job=default.test-tfjob replica-type=worker uid=
time="2019-08-16T10:12:55Z" level=info msg="need to create new service: worker-2" job=default.test-tfjob replica-type=worker uid=
time="2019-08-16T10:12:55Z" level=info msg="need to create new service: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2019-08-16T10:12:55Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (1.16414ms)" job=default.test-tfjob
time="2019-08-16T10:12:55Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:12:55Z" level=info msg="Creating Job controller"
time="2019-08-16T10:12:55Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-16T10:12:55Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:12:55Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:12:55Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (448.689µs)" job=default.test-tfjob
time="2019-08-16T10:12:55Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:12:55Z" level=info msg="Creating Job controller"
time="2019-08-16T10:12:55Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-16T10:12:55Z" level=info msg="Need to create new pod: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2019-08-16T10:12:55Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:12:55Z" level=info msg="need to create new service: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2019-08-16T10:12:55Z" level=info msg="Need to create new pod: worker-2" job=default.test-tfjob replica-type=worker uid=
time="2019-08-16T10:12:55Z" level=info msg="Need to create new pod: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2019-08-16T10:12:55Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:12:55Z" level=info msg="need to create new service: worker-2" job=default.test-tfjob replica-type=worker uid=
time="2019-08-16T10:12:55Z" level=info msg="need to create new service: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2019-08-16T10:12:55Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (1.008401ms)" job=default.test-tfjob
time="2019-08-16T10:12:55Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:12:55Z" level=info msg="Creating Job controller"
time="2019-08-16T10:12:55Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-16T10:12:55Z" level=info msg="Ignoring inactive pod default/worker-2 in state Succeeded, deletion time <nil>"
time="2019-08-16T10:12:55Z" level=info msg="Need to create new pod: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2019-08-16T10:12:55Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:12:55Z" level=info msg="need to create new service: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2019-08-16T10:12:55Z" level=info msg="Need to create new pod: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2019-08-16T10:12:55Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=3, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:12:55Z" level=info msg="need to create new service: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2019-08-16T10:12:55Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (1.084968ms)" job=default.test-tfjob
time="2019-08-16T10:12:56Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:12:56Z" level=info msg="Creating Job controller"
time="2019-08-16T10:12:56Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-16T10:12:56Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=4, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:12:56Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:12:56Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (628.783µs)" job=default.test-tfjob
time="2019-08-16T10:12:56Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:12:56Z" level=info msg="Creating Job controller"
time="2019-08-16T10:12:56Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-16T10:12:56Z" level=info msg="Need to create new pod: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2019-08-16T10:12:56Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:12:56Z" level=info msg="need to create new service: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2019-08-16T10:12:56Z" level=info msg="Need to create new pod: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2019-08-16T10:12:56Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=1, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:12:56Z" level=info msg="need to create new service: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2019-08-16T10:12:56Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (522.548µs)" job=default.test-tfjob
time="2019-08-16T10:12:56Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:12:56Z" level=info msg="Creating Job controller"
time="2019-08-16T10:12:56Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-16T10:12:56Z" level=info msg="Ignoring inactive pod default/worker-1 in state Succeeded, deletion time <nil>"
time="2019-08-16T10:12:56Z" level=info msg="Ignoring inactive pod default/worker-2 in state Succeeded, deletion time <nil>"
time="2019-08-16T10:12:56Z" level=info msg="Ignoring inactive pod default/worker-3 in state Succeeded, deletion time <nil>"
time="2019-08-16T10:12:56Z" level=info msg="Ignoring inactive pod default/ps-0 in state Succeeded, deletion time <nil>"
time="2019-08-16T10:12:56Z" level=info msg="Ignoring inactive pod default/ps-1 in state Succeeded, deletion time <nil>"
time="2019-08-16T10:12:56Z" level=info msg="Ignoring inactive pod default/worker-0 in state Succeeded, deletion time <nil>"
time="2019-08-16T10:12:56Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:12:56Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=0, running=0, failed=0" job=default.test-tfjob uid=
E0816 10:12:56.006951    9911 event.go:259] Could not construct reference to: '&v1.TFJob{TypeMeta:v1.TypeMeta{Kind:"TFJob", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"test-tfjob", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, Spec:v1.TFJobSpec{ActiveDeadlineSeconds:(*int64)(nil), BackoffLimit:(*int32)(nil), CleanPodPolicy:(*v1.CleanPodPolicy)(0xc42035dfc0), TTLSecondsAfterFinished:(*int32)(nil), TFReplicaSpecs:map[v1.TFReplicaType]*v1.ReplicaSpec{"PS":(*v1.ReplicaSpec)(0xc42012bb80), "Worker":(*v1.ReplicaSpec)(0xc42034e000)}}, Status:v1.JobStatus{Conditions:[]v1.JobCondition(nil), ReplicaStatuses:map[v1.ReplicaType]*v1.ReplicaStatus{"PS":(*v1.ReplicaStatus)(0xc4200446e0), "Worker":(*v1.ReplicaStatus)(0xc420044770)}, StartTime:(*v1.Time)(0xc4204b2160), CompletionTime:(*v1.Time)(nil), LastReconcileTime:(*v1.Time)(nil)}}' due to: 'selfLink was empty, can't make reference'. Will not report event: 'Normal' 'TFJobSucceeded' 'TFJob test-tfjob successfully completed.'
time="2019-08-16T10:12:56Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (952.313µs)" job=default.test-tfjob
--- PASS: TestNormalPath (0.03s)
=== RUN   TestRun
time="2019-08-16T10:12:56Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:12:56Z" level=info msg="Creating Job controller"
time="2019-08-16T10:12:56Z" level=info msg="Starting TFJob controller"
time="2019-08-16T10:12:56Z" level=info msg="Waiting for informer caches to sync"
time="2019-08-16T10:12:56Z" level=info msg="Starting 1 workers"
time="2019-08-16T10:12:56Z" level=info msg="Started workers"
time="2019-08-16T10:12:56Z" level=info msg="Shutting down workers"
--- PASS: TestRun (0.50s)
=== RUN   TestSyncPdb
time="2019-08-16T10:12:56Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:12:56Z" level=info msg="Creating Job controller"
--- PASS: TestSyncPdb (0.00s)
=== RUN   TestAddTFJob
time="2019-08-16T10:12:56Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:12:56Z" level=info msg="Creating Job controller"
time="2019-08-16T10:12:56Z" level=info msg="TFJob test-tfjob is created." job=default.test-tfjob uid=
time="2019-08-16T10:12:56Z" level=info msg="Starting TFJob controller"
time="2019-08-16T10:12:56Z" level=info msg="Waiting for informer caches to sync"
time="2019-08-16T10:12:56Z" level=info msg="Starting 1 workers"
time="2019-08-16T10:12:56Z" level=info msg="Started workers"
--- PASS: TestAddTFJob (0.10s)
=== RUN   TestCopyLabelsAndAnnotation
time="2019-08-16T10:12:56Z" level=info msg="Shutting down workers"
time="2019-08-16T10:12:56Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:12:56Z" level=info msg="Creating Job controller"
time="2019-08-16T10:12:56Z" level=info msg="Starting TFJob controller"
time="2019-08-16T10:12:56Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-16T10:12:56Z" level=info msg="Need to create new pod: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-08-16T10:12:56Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=1, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:12:56Z" level=info msg="need to create new service: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-08-16T10:12:56Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (392.161µs)" job=default.test-tfjob
--- PASS: TestCopyLabelsAndAnnotation (0.00s)
=== RUN   TestDeletePodsAndServices
time="2019-08-16T10:12:56Z" level=info msg="Waiting for informer caches to sync"
time="2019-08-16T10:12:56Z" level=info msg="Starting 1 workers"
time="2019-08-16T10:12:56Z" level=info msg="Started workers"
time="2019-08-16T10:12:56Z" level=info msg="Shutting down workers"
time="2019-08-16T10:12:56Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:12:56Z" level=info msg="Creating Job controller"
time="2019-08-16T10:12:56Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-16T10:12:56Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (583.452µs)" job=default.test-tfjob
time="2019-08-16T10:12:56Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:12:56Z" level=info msg="Creating Job controller"
time="2019-08-16T10:12:56Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-16T10:12:56Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (606.936µs)" job=default.test-tfjob
time="2019-08-16T10:12:56Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:12:56Z" level=info msg="Creating Job controller"
time="2019-08-16T10:12:56Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-16T10:12:56Z" level=info msg="Ignoring inactive pod default/worker-3 in state Succeeded, deletion time <nil>"
time="2019-08-16T10:12:56Z" level=info msg="Ignoring inactive pod default/ps-0 in state Succeeded, deletion time <nil>"
time="2019-08-16T10:12:56Z" level=info msg="Ignoring inactive pod default/ps-1 in state Succeeded, deletion time <nil>"
time="2019-08-16T10:12:56Z" level=info msg="Ignoring inactive pod default/worker-0 in state Succeeded, deletion time <nil>"
time="2019-08-16T10:12:56Z" level=info msg="Ignoring inactive pod default/worker-1 in state Succeeded, deletion time <nil>"
time="2019-08-16T10:12:56Z" level=info msg="Ignoring inactive pod default/worker-2 in state Succeeded, deletion time <nil>"
time="2019-08-16T10:12:56Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (1.147055ms)" job=default.test-tfjob
time="2019-08-16T10:12:56Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:12:56Z" level=info msg="Creating Job controller"
time="2019-08-16T10:12:56Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-16T10:12:56Z" level=info msg="Ignoring inactive pod default/worker-0 in state Succeeded, deletion time <nil>"
time="2019-08-16T10:12:56Z" level=info msg="Ignoring inactive pod default/worker-1 in state Succeeded, deletion time <nil>"
time="2019-08-16T10:12:56Z" level=info msg="Ignoring inactive pod default/worker-2 in state Succeeded, deletion time <nil>"
time="2019-08-16T10:12:56Z" level=info msg="Ignoring inactive pod default/worker-3 in state Succeeded, deletion time <nil>"
time="2019-08-16T10:12:56Z" level=info msg="Ignoring inactive pod default/ps-0 in state Succeeded, deletion time <nil>"
time="2019-08-16T10:12:56Z" level=info msg="Ignoring inactive pod default/ps-1 in state Succeeded, deletion time <nil>"
time="2019-08-16T10:12:56Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (429.87µs)" job=default.test-tfjob
--- PASS: TestDeletePodsAndServices (0.01s)
=== RUN   TestCleanupTFJob
time="2019-08-16T10:12:56Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:12:56Z" level=info msg="Creating Job controller"
time="2019-08-16T10:12:56Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-16T10:12:56Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (259.68µs)" job=default.test-tfjob
time="2019-08-16T10:12:56Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:12:56Z" level=info msg="Creating Job controller"
time="2019-08-16T10:12:56Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-16T10:12:56Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (236.528µs)" job=default.test-tfjob
time="2019-08-16T10:12:56Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:12:56Z" level=info msg="Creating Job controller"
time="2019-08-16T10:12:58Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-16T10:12:58Z" level=info msg="Ignoring inactive pod default/worker-0 in state Succeeded, deletion time <nil>"
time="2019-08-16T10:12:58Z" level=info msg="Ignoring inactive pod default/worker-1 in state Succeeded, deletion time <nil>"
time="2019-08-16T10:12:58Z" level=info msg="Ignoring inactive pod default/worker-2 in state Succeeded, deletion time <nil>"
time="2019-08-16T10:12:58Z" level=info msg="Ignoring inactive pod default/worker-3 in state Succeeded, deletion time <nil>"
time="2019-08-16T10:12:58Z" level=info msg="Ignoring inactive pod default/ps-0 in state Succeeded, deletion time <nil>"
time="2019-08-16T10:12:58Z" level=info msg="Ignoring inactive pod default/ps-1 in state Succeeded, deletion time <nil>"
time="2019-08-16T10:12:58Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (596.067µs)" job=default.test-tfjob
--- PASS: TestCleanupTFJob (2.00s)
=== RUN   TestActiveDeadlineSeconds
time="2019-08-16T10:12:58Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:12:58Z" level=info msg="Creating Job controller"
time="2019-08-16T10:12:58Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-16T10:12:58Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=4, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:12:58Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:12:58Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:12:58Z" level=info msg="Creating Job controller"
time="2019-08-16T10:13:00Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
--- PASS: TestActiveDeadlineSeconds (2.00s)
=== RUN   TestBackoffForOnFailure
time="2019-08-16T10:13:00Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:13:00Z" level=info msg="Creating Job controller"
time="2019-08-16T10:13:00Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-16T10:13:00Z" level=warning msg="The restart policy of replica PS of the job test-tfjob is not OnFailure or Always. Not counted in backoff limit." job=default.test-tfjob uid=
time="2019-08-16T10:13:00Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (414.857µs)" job=default.test-tfjob
--- PASS: TestBackoffForOnFailure (0.00s)
=== RUN   TestAddPod
time="2019-08-16T10:13:00Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:13:00Z" level=info msg="Creating Job controller"
time="2019-08-16T10:13:00Z" level=info msg="Starting TFJob controller"
time="2019-08-16T10:13:00Z" level=info msg="Waiting for informer caches to sync"
time="2019-08-16T10:13:00Z" level=info msg="Starting 1 workers"
time="2019-08-16T10:13:00Z" level=info msg="Started workers"
--- PASS: TestAddPod (0.10s)
=== RUN   TestClusterSpec
time="2019-08-16T10:13:00Z" level=info msg="Shutting down workers"
--- PASS: TestClusterSpec (0.00s)
=== RUN   TestRestartPolicy
--- PASS: TestRestartPolicy (0.00s)
=== RUN   TestExitCode
time="2019-08-16T10:13:00Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:13:00Z" level=info msg="Creating Job controller"
time="2019-08-16T10:13:00Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-16T10:13:00Z" level=info msg="Ignoring inactive pod default/worker-0 in state Failed, deletion time <nil>"
time="2019-08-16T10:13:00Z" level=info msg="Pod: default.worker-0 exited with code 130" job=default.test-tfjob replica-type=worker uid=
E0816 10:13:00.740786    9911 event.go:259] Could not construct reference to: '&v1.TFJob{TypeMeta:v1.TypeMeta{Kind:"TFJob", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"test-tfjob", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, Spec:v1.TFJobSpec{ActiveDeadlineSeconds:(*int64)(nil), BackoffLimit:(*int32)(nil), CleanPodPolicy:(*v1.CleanPodPolicy)(0xc420a12350), TTLSecondsAfterFinished:(*int32)(nil), TFReplicaSpecs:map[v1.TFReplicaType]*v1.ReplicaSpec{"Worker":(*v1.ReplicaSpec)(0xc4209ddb80)}}, Status:v1.JobStatus{Conditions:[]v1.JobCondition(nil), ReplicaStatuses:map[v1.ReplicaType]*v1.ReplicaStatus{"Worker":(*v1.ReplicaStatus)(0xc4209d6f90)}, StartTime:(*v1.Time)(nil), CompletionTime:(*v1.Time)(nil), LastReconcileTime:(*v1.Time)(nil)}}' due to: 'selfLink was empty, can't make reference'. Will not report event: 'Normal' 'ExitedWithCode' 'Pod: default.worker-0 exited with code 130'
time="2019-08-16T10:13:00Z" level=info msg="Need to restart the pod: default.worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-08-16T10:13:00Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=1, running=0, failed=1" job=default.test-tfjob uid=
E0816 10:13:00.740916    9911 event.go:259] Could not construct reference to: '&v1.TFJob{TypeMeta:v1.TypeMeta{Kind:"TFJob", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"test-tfjob", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, Spec:v1.TFJobSpec{ActiveDeadlineSeconds:(*int64)(nil), BackoffLimit:(*int32)(nil), CleanPodPolicy:(*v1.CleanPodPolicy)(0xc420a12350), TTLSecondsAfterFinished:(*int32)(nil), TFReplicaSpecs:map[v1.TFReplicaType]*v1.ReplicaSpec{"Worker":(*v1.ReplicaSpec)(0xc4209ddb80)}}, Status:v1.JobStatus{Conditions:[]v1.JobCondition(nil), ReplicaStatuses:map[v1.ReplicaType]*v1.ReplicaStatus{"Worker":(*v1.ReplicaStatus)(0xc4209d6f90)}, StartTime:(*v1.Time)(0xc4209edb60), CompletionTime:(*v1.Time)(nil), LastReconcileTime:(*v1.Time)(nil)}}' due to: 'selfLink was empty, can't make reference'. Will not report event: 'Warning' 'TFJobRestarting' 'TFJob test-tfjob is restarting because 1 Worker replica(s) failed.'
time="2019-08-16T10:13:00Z" level=info msg="need to create new service: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-08-16T10:13:00Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (514.322µs)" job=default.test-tfjob
--- PASS: TestExitCode (0.00s)
=== RUN   TestAddService
time="2019-08-16T10:13:00Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:13:00Z" level=info msg="Creating Job controller"
time="2019-08-16T10:13:00Z" level=info msg="Starting TFJob controller"
time="2019-08-16T10:13:00Z" level=info msg="Waiting for informer caches to sync"
time="2019-08-16T10:13:00Z" level=info msg="Starting TFJob controller"
time="2019-08-16T10:13:00Z" level=info msg="Waiting for informer caches to sync"
time="2019-08-16T10:13:00Z" level=info msg="Starting 1 workers"
time="2019-08-16T10:13:00Z" level=info msg="Started workers"
time="2019-08-16T10:13:00Z" level=info msg="Shutting down workers"
time="2019-08-16T10:13:00Z" level=info msg="Starting 1 workers"
time="2019-08-16T10:13:00Z" level=info msg="Started workers"
--- PASS: TestAddService (0.10s)
=== RUN   TestFailed
time="2019-08-16T10:13:00Z" level=info msg="Shutting down workers"
time="2019-08-16T10:13:00Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:13:00Z" level=info msg="Creating Job controller"
time="2019-08-16T10:13:00Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=3, running=0, failed=1" job=default.test-tfjob uid=
E0816 10:13:00.844404    9911 event.go:259] Could not construct reference to: '&v1.TFJob{TypeMeta:v1.TypeMeta{Kind:"TFJob", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"test-tfjob", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, Spec:v1.TFJobSpec{ActiveDeadlineSeconds:(*int64)(nil), BackoffLimit:(*int32)(nil), CleanPodPolicy:(*v1.CleanPodPolicy)(nil), TTLSecondsAfterFinished:(*int32)(nil), TFReplicaSpecs:map[v1.TFReplicaType]*v1.ReplicaSpec{"Worker":(*v1.ReplicaSpec)(0xc420a54840)}}, Status:v1.JobStatus{Conditions:[]v1.JobCondition(nil), ReplicaStatuses:map[v1.ReplicaType]*v1.ReplicaStatus{"Worker":(*v1.ReplicaStatus)(0xc420a56d70)}, StartTime:(*v1.Time)(0xc420a7a800), CompletionTime:(*v1.Time)(nil), LastReconcileTime:(*v1.Time)(nil)}}' due to: 'selfLink was empty, can't make reference'. Will not report event: 'Normal' 'TFJobFailed' 'TFJob test-tfjob has failed because 1 Worker replica(s) failed.'
--- PASS: TestFailed (0.00s)
=== RUN   TestStatus
time="2019-08-16T10:13:00Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:13:00Z" level=info msg="Creating Job controller"
time="2019-08-16T10:13:00Z" level=info msg="TFJob=test-tfjob, ReplicaType=Chief expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:13:00Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:13:00Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:13:00Z" level=info msg="Creating Job controller"
time="2019-08-16T10:13:00Z" level=info msg="TFJob=test-tfjob, ReplicaType=Chief expected=1, running=1, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:13:00Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=1, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:13:00Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:13:00Z" level=info msg="Creating Job controller"
time="2019-08-16T10:13:00Z" level=info msg="TFJob=test-tfjob, ReplicaType=Chief expected=1, running=0, failed=1" job=default.test-tfjob uid=
time="2019-08-16T10:13:00Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=1, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:13:00Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:13:00Z" level=info msg="Creating Job controller"
time="2019-08-16T10:13:00Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=1, running=0, failed=1" job=default.test-tfjob uid=
time="2019-08-16T10:13:00Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:13:00Z" level=info msg="Creating Job controller"
time="2019-08-16T10:13:00Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:13:00Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:13:00Z" level=info msg="Creating Job controller"
time="2019-08-16T10:13:00Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=1, running=1, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:13:00Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:13:00Z" level=info msg="Creating Job controller"
time="2019-08-16T10:13:00Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:13:00Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:13:00Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:13:00Z" level=info msg="Creating Job controller"
time="2019-08-16T10:13:00Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=2, failed=2" job=default.test-tfjob uid=
time="2019-08-16T10:13:00Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:13:00Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:13:00Z" level=info msg="Creating Job controller"
time="2019-08-16T10:13:00Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=2, running=0, failed=2" job=default.test-tfjob uid=
time="2019-08-16T10:13:00Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:13:00Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:13:00Z" level=info msg="Creating Job controller"
time="2019-08-16T10:13:00Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=3, running=3, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:13:00Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:13:00Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:13:00Z" level=info msg="Creating Job controller"
time="2019-08-16T10:13:00Z" level=info msg="TFJob=test-tfjob, ReplicaType=Chief expected=1, running=1, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:13:00Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=0, failed=4" job=default.test-tfjob uid=
time="2019-08-16T10:13:00Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:13:00Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:13:00Z" level=info msg="Creating Job controller"
time="2019-08-16T10:13:00Z" level=info msg="TFJob=test-tfjob, ReplicaType=Chief expected=1, running=1, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:13:00Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:13:00Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:13:00Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:13:00Z" level=info msg="Creating Job controller"
time="2019-08-16T10:13:00Z" level=info msg="TFJob=test-tfjob, ReplicaType=Chief expected=1, running=1, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:13:00Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:13:00Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=1, failed=1" job=default.test-tfjob uid=
time="2019-08-16T10:13:00Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:13:00Z" level=info msg="Creating Job controller"
time="2019-08-16T10:13:00Z" level=info msg="TFJob=test-tfjob, ReplicaType=Chief expected=1, running=0, failed=1" job=default.test-tfjob uid=
time="2019-08-16T10:13:00Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:13:00Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:13:00Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:13:00Z" level=info msg="Creating Job controller"
time="2019-08-16T10:13:00Z" level=info msg="TFJob=test-tfjob, ReplicaType=Chief expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:13:00Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=0, failed=4" job=default.test-tfjob uid=
time="2019-08-16T10:13:00Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:13:00Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:13:00Z" level=info msg="Creating Job controller"
time="2019-08-16T10:13:00Z" level=info msg="TFJob=test-tfjob, ReplicaType=Chief expected=1, running=0, failed=1" job=default.test-tfjob uid=
time="2019-08-16T10:13:00Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=0, failed=4" job=default.test-tfjob uid=
time="2019-08-16T10:13:00Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
--- PASS: TestStatus (0.02s)
=== RUN   TestGenOwnerReference
--- PASS: TestGenOwnerReference (0.00s)
=== RUN   TestGenLabels
--- PASS: TestGenLabels (0.00s)
=== RUN   TestConvertTFJobToUnstructured
--- PASS: TestConvertTFJobToUnstructured (0.00s)
PASS
coverage: 43.9% of statements in github.com/kubeflow/tf-operator/pkg/apis/common/v1, github.com/kubeflow/tf-operator/pkg/apis/common/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/controller.v1beta2/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
ok  	github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow	4.934s	coverage: 43.9% of statements in github.com/kubeflow/tf-operator/pkg/apis/common/v1, github.com/kubeflow/tf-operator/pkg/apis/common/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/controller.v1beta2/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
=== RUN   TestNormalPath
time="2019-08-16T10:13:07Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:13:07Z" level=info msg="Creating Job controller"
time="2019-08-16T10:13:07Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-16T10:13:07Z" level=info msg="Need to create new pod: ps-0" job=default.test-tfjob replica-type=ps uid=
time="2019-08-16T10:13:07Z" level=info msg="Need to create new pod: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2019-08-16T10:13:07Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:13:07Z" level=info msg="need to create new service: ps-0" job=default.test-tfjob replica-type=ps uid=
time="2019-08-16T10:13:07Z" level=info msg="need to create new service: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2019-08-16T10:13:07Z" level=info msg="Need to create new pod: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-08-16T10:13:07Z" level=info msg="Need to create new pod: worker-1" job=default.test-tfjob replica-type=worker uid=
time="2019-08-16T10:13:07Z" level=info msg="Need to create new pod: worker-2" job=default.test-tfjob replica-type=worker uid=
time="2019-08-16T10:13:07Z" level=info msg="Need to create new pod: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2019-08-16T10:13:07Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:13:07Z" level=info msg="need to create new service: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-08-16T10:13:07Z" level=info msg="need to create new service: worker-1" job=default.test-tfjob replica-type=worker uid=
time="2019-08-16T10:13:07Z" level=info msg="need to create new service: worker-2" job=default.test-tfjob replica-type=worker uid=
time="2019-08-16T10:13:07Z" level=info msg="need to create new service: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2019-08-16T10:13:07Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (2.196727ms)" job=default.test-tfjob
time="2019-08-16T10:13:07Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:13:07Z" level=info msg="Creating Job controller"
time="2019-08-16T10:13:07Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-16T10:13:07Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:13:07Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:13:07Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (463.263µs)" job=default.test-tfjob
time="2019-08-16T10:13:07Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:13:07Z" level=info msg="Creating Job controller"
time="2019-08-16T10:13:07Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-16T10:13:07Z" level=info msg="Need to create new pod: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2019-08-16T10:13:07Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:13:07Z" level=info msg="need to create new service: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2019-08-16T10:13:07Z" level=info msg="Need to create new pod: worker-2" job=default.test-tfjob replica-type=worker uid=
time="2019-08-16T10:13:07Z" level=info msg="Need to create new pod: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2019-08-16T10:13:07Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:13:07Z" level=info msg="need to create new service: worker-2" job=default.test-tfjob replica-type=worker uid=
time="2019-08-16T10:13:07Z" level=info msg="need to create new service: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2019-08-16T10:13:07Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (906.73µs)" job=default.test-tfjob
time="2019-08-16T10:13:07Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:13:07Z" level=info msg="Creating Job controller"
time="2019-08-16T10:13:07Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-16T10:13:07Z" level=info msg="Ignoring inactive pod default/worker-2 in state Succeeded, deletion time <nil>"
time="2019-08-16T10:13:07Z" level=info msg="Need to create new pod: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2019-08-16T10:13:07Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:13:07Z" level=info msg="need to create new service: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2019-08-16T10:13:07Z" level=info msg="Need to create new pod: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2019-08-16T10:13:07Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=3, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:13:07Z" level=info msg="need to create new service: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2019-08-16T10:13:07Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (876.872µs)" job=default.test-tfjob
time="2019-08-16T10:13:07Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:13:07Z" level=info msg="Creating Job controller"
time="2019-08-16T10:13:07Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-16T10:13:07Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:13:07Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=4, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:13:07Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (543.286µs)" job=default.test-tfjob
time="2019-08-16T10:13:07Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:13:07Z" level=info msg="Creating Job controller"
time="2019-08-16T10:13:07Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-16T10:13:07Z" level=info msg="Need to create new pod: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2019-08-16T10:13:07Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:13:07Z" level=info msg="need to create new service: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2019-08-16T10:13:07Z" level=info msg="Need to create new pod: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2019-08-16T10:13:07Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=1, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:13:07Z" level=info msg="need to create new service: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2019-08-16T10:13:07Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (717.218µs)" job=default.test-tfjob
time="2019-08-16T10:13:07Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:13:07Z" level=info msg="Creating Job controller"
time="2019-08-16T10:13:07Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-16T10:13:07Z" level=info msg="Ignoring inactive pod default/ps-1 in state Succeeded, deletion time <nil>"
time="2019-08-16T10:13:07Z" level=info msg="Ignoring inactive pod default/worker-0 in state Succeeded, deletion time <nil>"
time="2019-08-16T10:13:07Z" level=info msg="Ignoring inactive pod default/worker-1 in state Succeeded, deletion time <nil>"
time="2019-08-16T10:13:07Z" level=info msg="Ignoring inactive pod default/worker-2 in state Succeeded, deletion time <nil>"
time="2019-08-16T10:13:07Z" level=info msg="Ignoring inactive pod default/worker-3 in state Succeeded, deletion time <nil>"
time="2019-08-16T10:13:07Z" level=info msg="Ignoring inactive pod default/ps-0 in state Succeeded, deletion time <nil>"
time="2019-08-16T10:13:07Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:13:07Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=0, running=0, failed=0" job=default.test-tfjob uid=
E0816 10:13:07.206376   10022 event.go:259] Could not construct reference to: '&v1beta2.TFJob{TypeMeta:v1.TypeMeta{Kind:"TFJob", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"test-tfjob", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, Spec:v1beta2.TFJobSpec{ActiveDeadlineSeconds:(*int64)(nil), BackoffLimit:(*int32)(nil), CleanPodPolicy:(*v1beta2.CleanPodPolicy)(0xc420548690), TTLSecondsAfterFinished:(*int32)(nil), TFReplicaSpecs:map[v1beta2.TFReplicaType]*v1beta2.ReplicaSpec{"PS":(*v1beta2.ReplicaSpec)(0xc420791080), "Worker":(*v1beta2.ReplicaSpec)(0xc420791340)}}, Status:v1beta2.JobStatus{Conditions:[]v1beta2.JobCondition(nil), ReplicaStatuses:map[v1beta2.ReplicaType]*v1beta2.ReplicaStatus{"PS":(*v1beta2.ReplicaStatus)(0xc4207a0760), "Worker":(*v1beta2.ReplicaStatus)(0xc4207a0790)}, StartTime:(*v1.Time)(0xc42079fd80), CompletionTime:(*v1.Time)(nil), LastReconcileTime:(*v1.Time)(nil)}}' due to: 'selfLink was empty, can't make reference'. Will not report event: 'Normal' 'TFJobSucceeded' 'TFJob test-tfjob successfully completed.'
time="2019-08-16T10:13:07Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (3.72009ms)" job=default.test-tfjob
time="2019-08-16T10:13:07Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:13:07Z" level=info msg="Creating Job controller"
time="2019-08-16T10:13:07Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-16T10:13:07Z" level=info msg="Need to create new pod: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-08-16T10:13:07Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=1, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:13:07Z" level=info msg="need to create new service: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-08-16T10:13:07Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (336.017µs)" job=default.test-tfjob
--- PASS: TestNormalPath (0.03s)
=== RUN   TestRun
time="2019-08-16T10:13:07Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:13:07Z" level=info msg="Creating Job controller"
time="2019-08-16T10:13:07Z" level=info msg="Starting TFJob controller"
time="2019-08-16T10:13:07Z" level=info msg="Waiting for informer caches to sync"
time="2019-08-16T10:13:07Z" level=info msg="Starting 1 workers"
time="2019-08-16T10:13:07Z" level=info msg="Started workers"
time="2019-08-16T10:13:07Z" level=info msg="Shutting down workers"
--- PASS: TestRun (0.50s)
=== RUN   TestSyncPdb
time="2019-08-16T10:13:07Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:13:07Z" level=info msg="Creating Job controller"
--- PASS: TestSyncPdb (0.00s)
=== RUN   TestAddTFJob
time="2019-08-16T10:13:07Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:13:07Z" level=info msg="Creating Job controller"
time="2019-08-16T10:13:07Z" level=info msg="TFJob test-tfjob is created." job=default.test-tfjob uid=
time="2019-08-16T10:13:07Z" level=info msg="Starting TFJob controller"
time="2019-08-16T10:13:07Z" level=info msg="Waiting for informer caches to sync"
time="2019-08-16T10:13:07Z" level=info msg="Starting 1 workers"
time="2019-08-16T10:13:07Z" level=info msg="Started workers"
--- PASS: TestAddTFJob (0.10s)
=== RUN   TestCopyLabelsAndAnnotation
time="2019-08-16T10:13:07Z" level=info msg="Shutting down workers"
time="2019-08-16T10:13:07Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:13:07Z" level=info msg="Creating Job controller"
time="2019-08-16T10:13:07Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-16T10:13:07Z" level=info msg="Need to create new pod: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-08-16T10:13:07Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=1, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:13:07Z" level=info msg="need to create new service: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-08-16T10:13:07Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (424.559µs)" job=default.test-tfjob
--- PASS: TestCopyLabelsAndAnnotation (0.00s)
=== RUN   TestDeletePodsAndServices
time="2019-08-16T10:13:07Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:13:07Z" level=info msg="Creating Job controller"
time="2019-08-16T10:13:07Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-16T10:13:07Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:13:07Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=4, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:13:07Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (596.212µs)" job=default.test-tfjob
time="2019-08-16T10:13:07Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:13:07Z" level=info msg="Creating Job controller"
time="2019-08-16T10:13:07Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-16T10:13:07Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=4, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:13:07Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:13:07Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (533.686µs)" job=default.test-tfjob
time="2019-08-16T10:13:07Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:13:07Z" level=info msg="Creating Job controller"
time="2019-08-16T10:13:07Z" level=info msg="Starting TFJob controller"
time="2019-08-16T10:13:07Z" level=info msg="Waiting for informer caches to sync"
time="2019-08-16T10:13:07Z" level=info msg="Starting 1 workers"
time="2019-08-16T10:13:07Z" level=info msg="Started workers"
time="2019-08-16T10:13:07Z" level=info msg="Shutting down workers"
time="2019-08-16T10:13:07Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-16T10:13:07Z" level=info msg="Ignoring inactive pod default/worker-2 in state Succeeded, deletion time <nil>"
time="2019-08-16T10:13:07Z" level=info msg="Ignoring inactive pod default/worker-3 in state Succeeded, deletion time <nil>"
time="2019-08-16T10:13:07Z" level=info msg="Ignoring inactive pod default/ps-0 in state Succeeded, deletion time <nil>"
time="2019-08-16T10:13:07Z" level=info msg="Ignoring inactive pod default/ps-1 in state Succeeded, deletion time <nil>"
time="2019-08-16T10:13:07Z" level=info msg="Ignoring inactive pod default/worker-0 in state Succeeded, deletion time <nil>"
time="2019-08-16T10:13:07Z" level=info msg="Ignoring inactive pod default/worker-1 in state Succeeded, deletion time <nil>"
time="2019-08-16T10:13:07Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:13:07Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:13:07Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (1.367834ms)" job=default.test-tfjob
time="2019-08-16T10:13:07Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:13:07Z" level=info msg="Creating Job controller"
time="2019-08-16T10:13:07Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-16T10:13:07Z" level=info msg="Ignoring inactive pod default/ps-1 in state Succeeded, deletion time <nil>"
time="2019-08-16T10:13:07Z" level=info msg="Ignoring inactive pod default/worker-0 in state Succeeded, deletion time <nil>"
time="2019-08-16T10:13:07Z" level=info msg="Ignoring inactive pod default/worker-1 in state Succeeded, deletion time <nil>"
time="2019-08-16T10:13:07Z" level=info msg="Ignoring inactive pod default/worker-2 in state Succeeded, deletion time <nil>"
time="2019-08-16T10:13:07Z" level=info msg="Ignoring inactive pod default/worker-3 in state Succeeded, deletion time <nil>"
time="2019-08-16T10:13:07Z" level=info msg="Ignoring inactive pod default/ps-0 in state Succeeded, deletion time <nil>"
time="2019-08-16T10:13:07Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:13:07Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:13:07Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (650.811µs)" job=default.test-tfjob
--- PASS: TestDeletePodsAndServices (0.01s)
=== RUN   TestCleanupTFJob
time="2019-08-16T10:13:07Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:13:07Z" level=info msg="Creating Job controller"
time="2019-08-16T10:13:07Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-16T10:13:07Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:13:07Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=4, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:13:07Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (494.432µs)" job=default.test-tfjob
time="2019-08-16T10:13:07Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:13:07Z" level=info msg="Creating Job controller"
time="2019-08-16T10:13:07Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-16T10:13:07Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:13:07Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=4, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:13:07Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (751.995µs)" job=default.test-tfjob
time="2019-08-16T10:13:07Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:13:07Z" level=info msg="Creating Job controller"
time="2019-08-16T10:13:09Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-16T10:13:09Z" level=info msg="Ignoring inactive pod default/worker-3 in state Succeeded, deletion time <nil>"
time="2019-08-16T10:13:09Z" level=info msg="Ignoring inactive pod default/ps-0 in state Succeeded, deletion time <nil>"
time="2019-08-16T10:13:09Z" level=info msg="Ignoring inactive pod default/ps-1 in state Succeeded, deletion time <nil>"
time="2019-08-16T10:13:09Z" level=info msg="Ignoring inactive pod default/worker-0 in state Succeeded, deletion time <nil>"
time="2019-08-16T10:13:09Z" level=info msg="Ignoring inactive pod default/worker-1 in state Succeeded, deletion time <nil>"
time="2019-08-16T10:13:09Z" level=info msg="Ignoring inactive pod default/worker-2 in state Succeeded, deletion time <nil>"
time="2019-08-16T10:13:09Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:13:09Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:13:09Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (774.653µs)" job=default.test-tfjob
--- PASS: TestCleanupTFJob (2.01s)
=== RUN   TestActiveDeadlineSeconds
time="2019-08-16T10:13:09Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:13:09Z" level=info msg="Creating Job controller"
time="2019-08-16T10:13:09Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-16T10:13:09Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:13:09Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=4, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:13:09Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:13:09Z" level=info msg="Creating Job controller"
time="2019-08-16T10:13:11Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
--- PASS: TestActiveDeadlineSeconds (2.00s)
=== RUN   TestBackoffForOnFailure
time="2019-08-16T10:13:11Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:13:11Z" level=info msg="Creating Job controller"
time="2019-08-16T10:13:11Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-16T10:13:11Z" level=warning msg="The restart policy of replica PS of the job test-tfjob is not OnFailure or Always. Not counted in backoff limit." job=default.test-tfjob uid=
time="2019-08-16T10:13:11Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (450.893µs)" job=default.test-tfjob
--- PASS: TestBackoffForOnFailure (0.00s)
=== RUN   TestAddPod
time="2019-08-16T10:13:11Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:13:11Z" level=info msg="Creating Job controller"
time="2019-08-16T10:13:11Z" level=info msg="Starting TFJob controller"
time="2019-08-16T10:13:11Z" level=info msg="Waiting for informer caches to sync"
time="2019-08-16T10:13:11Z" level=info msg="Starting 1 workers"
time="2019-08-16T10:13:11Z" level=info msg="Started workers"
--- PASS: TestAddPod (0.10s)
=== RUN   TestClusterSpec
time="2019-08-16T10:13:11Z" level=info msg="Shutting down workers"
--- PASS: TestClusterSpec (0.00s)
=== RUN   TestRestartPolicy
--- PASS: TestRestartPolicy (0.00s)
=== RUN   TestExitCode
time="2019-08-16T10:13:11Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:13:11Z" level=info msg="Creating Job controller"
time="2019-08-16T10:13:11Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-16T10:13:11Z" level=info msg="Ignoring inactive pod default/worker-0 in state Failed, deletion time <nil>"
time="2019-08-16T10:13:11Z" level=info msg="Pod: default.worker-0 exited with code 130" job=default.test-tfjob replica-type=worker uid=
E0816 10:13:11.947530   10022 event.go:259] Could not construct reference to: '&v1beta2.TFJob{TypeMeta:v1.TypeMeta{Kind:"TFJob", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"test-tfjob", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, Spec:v1beta2.TFJobSpec{ActiveDeadlineSeconds:(*int64)(nil), BackoffLimit:(*int32)(nil), CleanPodPolicy:(*v1beta2.CleanPodPolicy)(0xc42005c100), TTLSecondsAfterFinished:(*int32)(nil), TFReplicaSpecs:map[v1beta2.TFReplicaType]*v1beta2.ReplicaSpec{"Worker":(*v1beta2.ReplicaSpec)(0xc4203d18c0)}}, Status:v1beta2.JobStatus{Conditions:[]v1beta2.JobCondition(nil), ReplicaStatuses:map[v1beta2.ReplicaType]*v1beta2.ReplicaStatus{"Worker":(*v1beta2.ReplicaStatus)(0xc4205456e0)}, StartTime:(*v1.Time)(nil), CompletionTime:(*v1.Time)(nil), LastReconcileTime:(*v1.Time)(nil)}}' due to: 'selfLink was empty, can't make reference'. Will not report event: 'Normal' 'ExitedWithCode' 'Pod: default.worker-0 exited with code 130'
time="2019-08-16T10:13:11Z" level=info msg="Need to restart the pod: default.worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-08-16T10:13:11Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=1, running=0, failed=1" job=default.test-tfjob uid=
E0816 10:13:11.947646   10022 event.go:259] Could not construct reference to: '&v1beta2.TFJob{TypeMeta:v1.TypeMeta{Kind:"TFJob", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"test-tfjob", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, Spec:v1beta2.TFJobSpec{ActiveDeadlineSeconds:(*int64)(nil), BackoffLimit:(*int32)(nil), CleanPodPolicy:(*v1beta2.CleanPodPolicy)(0xc42005c100), TTLSecondsAfterFinished:(*int32)(nil), TFReplicaSpecs:map[v1beta2.TFReplicaType]*v1beta2.ReplicaSpec{"Worker":(*v1beta2.ReplicaSpec)(0xc4203d18c0)}}, Status:v1beta2.JobStatus{Conditions:[]v1beta2.JobCondition(nil), ReplicaStatuses:map[v1beta2.ReplicaType]*v1beta2.ReplicaStatus{"Worker":(*v1beta2.ReplicaStatus)(0xc4205456e0)}, StartTime:(*v1.Time)(0xc4203f30a0), CompletionTime:(*v1.Time)(nil), LastReconcileTime:(*v1.Time)(nil)}}' due to: 'selfLink was empty, can't make reference'. Will not report event: 'Warning' 'TFJobRestarting' 'TFJob test-tfjob is restarting because 1 Worker replica(s) failed.'
time="2019-08-16T10:13:11Z" level=info msg="need to create new service: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-08-16T10:13:11Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (524.643µs)" job=default.test-tfjob
--- PASS: TestExitCode (0.00s)
=== RUN   TestAddService
time="2019-08-16T10:13:11Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:13:11Z" level=info msg="Creating Job controller"
time="2019-08-16T10:13:11Z" level=info msg="Starting TFJob controller"
time="2019-08-16T10:13:11Z" level=info msg="Waiting for informer caches to sync"
time="2019-08-16T10:13:11Z" level=info msg="Starting TFJob controller"
time="2019-08-16T10:13:11Z" level=info msg="Waiting for informer caches to sync"
time="2019-08-16T10:13:11Z" level=info msg="Starting 1 workers"
time="2019-08-16T10:13:11Z" level=info msg="Started workers"
time="2019-08-16T10:13:11Z" level=info msg="Shutting down workers"
time="2019-08-16T10:13:12Z" level=info msg="Starting 1 workers"
time="2019-08-16T10:13:12Z" level=info msg="Started workers"
--- PASS: TestAddService (0.10s)
=== RUN   TestFailed
time="2019-08-16T10:13:12Z" level=info msg="Shutting down workers"
time="2019-08-16T10:13:12Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:13:12Z" level=info msg="Creating Job controller"
time="2019-08-16T10:13:12Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=3, running=0, failed=1" job=default.test-tfjob uid=
E0816 10:13:12.050602   10022 event.go:259] Could not construct reference to: '&v1beta2.TFJob{TypeMeta:v1.TypeMeta{Kind:"TFJob", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"test-tfjob", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, Spec:v1beta2.TFJobSpec{ActiveDeadlineSeconds:(*int64)(nil), BackoffLimit:(*int32)(nil), CleanPodPolicy:(*v1beta2.CleanPodPolicy)(nil), TTLSecondsAfterFinished:(*int32)(nil), TFReplicaSpecs:map[v1beta2.TFReplicaType]*v1beta2.ReplicaSpec{"Worker":(*v1beta2.ReplicaSpec)(0xc42002eb00)}}, Status:v1beta2.JobStatus{Conditions:[]v1beta2.JobCondition(nil), ReplicaStatuses:map[v1beta2.ReplicaType]*v1beta2.ReplicaStatus{"Worker":(*v1beta2.ReplicaStatus)(0xc420045ab0)}, StartTime:(*v1.Time)(0xc420414680), CompletionTime:(*v1.Time)(nil), LastReconcileTime:(*v1.Time)(nil)}}' due to: 'selfLink was empty, can't make reference'. Will not report event: 'Normal' 'TFJobFailed' 'TFJob test-tfjob has failed because 1 Worker replica(s) failed.'
--- PASS: TestFailed (0.00s)
=== RUN   TestStatus
time="2019-08-16T10:13:12Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:13:12Z" level=info msg="Creating Job controller"
time="2019-08-16T10:13:12Z" level=info msg="TFJob=test-tfjob, ReplicaType=Chief expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:13:12Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:13:12Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:13:12Z" level=info msg="Creating Job controller"
time="2019-08-16T10:13:12Z" level=info msg="TFJob=test-tfjob, ReplicaType=Chief expected=1, running=1, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:13:12Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=1, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:13:12Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:13:12Z" level=info msg="Creating Job controller"
time="2019-08-16T10:13:12Z" level=info msg="TFJob=test-tfjob, ReplicaType=Chief expected=1, running=0, failed=1" job=default.test-tfjob uid=
time="2019-08-16T10:13:12Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=1, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:13:12Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:13:12Z" level=info msg="Creating Job controller"
time="2019-08-16T10:13:12Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=1, running=0, failed=1" job=default.test-tfjob uid=
time="2019-08-16T10:13:12Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:13:12Z" level=info msg="Creating Job controller"
time="2019-08-16T10:13:12Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:13:12Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:13:12Z" level=info msg="Creating Job controller"
time="2019-08-16T10:13:12Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=1, running=1, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:13:12Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:13:12Z" level=info msg="Creating Job controller"
time="2019-08-16T10:13:12Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:13:12Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:13:12Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:13:12Z" level=info msg="Creating Job controller"
time="2019-08-16T10:13:12Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=2, failed=2" job=default.test-tfjob uid=
time="2019-08-16T10:13:12Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:13:12Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:13:12Z" level=info msg="Creating Job controller"
time="2019-08-16T10:13:12Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=2, running=0, failed=2" job=default.test-tfjob uid=
time="2019-08-16T10:13:12Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:13:12Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:13:12Z" level=info msg="Creating Job controller"
time="2019-08-16T10:13:12Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=3, running=3, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:13:12Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:13:12Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:13:12Z" level=info msg="Creating Job controller"
time="2019-08-16T10:13:12Z" level=info msg="TFJob=test-tfjob, ReplicaType=Chief expected=1, running=1, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:13:12Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=0, failed=4" job=default.test-tfjob uid=
time="2019-08-16T10:13:12Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:13:12Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:13:12Z" level=info msg="Creating Job controller"
time="2019-08-16T10:13:12Z" level=info msg="TFJob=test-tfjob, ReplicaType=Chief expected=1, running=1, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:13:12Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:13:12Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:13:12Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:13:12Z" level=info msg="Creating Job controller"
time="2019-08-16T10:13:12Z" level=info msg="TFJob=test-tfjob, ReplicaType=Chief expected=1, running=1, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:13:12Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:13:12Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=1, failed=1" job=default.test-tfjob uid=
time="2019-08-16T10:13:12Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:13:12Z" level=info msg="Creating Job controller"
time="2019-08-16T10:13:12Z" level=info msg="TFJob=test-tfjob, ReplicaType=Chief expected=1, running=0, failed=1" job=default.test-tfjob uid=
time="2019-08-16T10:13:12Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:13:12Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:13:12Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:13:12Z" level=info msg="Creating Job controller"
time="2019-08-16T10:13:12Z" level=info msg="TFJob=test-tfjob, ReplicaType=Chief expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:13:12Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=0, failed=4" job=default.test-tfjob uid=
time="2019-08-16T10:13:12Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-16T10:13:12Z" level=info msg="Creating TFJob controller"
time="2019-08-16T10:13:12Z" level=info msg="Creating Job controller"
time="2019-08-16T10:13:12Z" level=info msg="TFJob=test-tfjob, ReplicaType=Chief expected=1, running=0, failed=1" job=default.test-tfjob uid=
time="2019-08-16T10:13:12Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=0, failed=4" job=default.test-tfjob uid=
time="2019-08-16T10:13:12Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
--- PASS: TestStatus (0.02s)
=== RUN   TestGenOwnerReference
--- PASS: TestGenOwnerReference (0.00s)
=== RUN   TestGenLabels
--- PASS: TestGenLabels (0.00s)
=== RUN   TestConvertTFJobToUnstructured
--- PASS: TestConvertTFJobToUnstructured (0.00s)
PASS
coverage: 43.8% of statements in github.com/kubeflow/tf-operator/pkg/apis/common/v1, github.com/kubeflow/tf-operator/pkg/apis/common/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/controller.v1beta2/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
ok  	github.com/kubeflow/tf-operator/pkg/controller.v1beta2/tensorflow	4.932s	coverage: 43.8% of statements in github.com/kubeflow/tf-operator/pkg/apis/common/v1, github.com/kubeflow/tf-operator/pkg/apis/common/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/controller.v1beta2/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
?   	github.com/kubeflow/tf-operator/pkg/logger	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/util	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/util/k8sutil	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/util/signals	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/util/train	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/version	[no test files]
ignoring pkg/apis/common/v1/openapi_generated.go
ignoring pkg/apis/common/v1/zz_generated.deepcopy.go
ignoring pkg/apis/common/v1/zz_generated.defaults.go
ignoring pkg/apis/tensorflow/v1/openapi_generated.go
ignoring pkg/apis/tensorflow/v1/zz_generated.deepcopy.go
ignoring pkg/apis/tensorflow/v1/zz_generated.defaults.go
Job #2542.1
https://coveralls.io/jobs/52098959
TravisBuddy Request Identifier: 764b6b00-c00e-11e9-aed2-2b81c1e31580

Copy link
Member

@gaocegege gaocegege left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

Waiting until #1069 is merged.

@ohmystack
Copy link
Contributor Author

@gaocegege #1069 is merged. It's ok to rerun the test now

@gaocegege
Copy link
Member

@ohmystack Can you rebase the master?

@ohmystack
Copy link
Contributor Author

@ohmystack Can you rebase the master?

Done.

@TravisBuddy
Copy link

Travis tests have failed

Hey @ohmystack,
Please read the following log in order to understand the failure reason.
It'll be awesome if you fix what's wrong and commit the changes.

1st Build

View build log

hack/verify-codegen.sh
Generating deepcopy funcs
Generating deepcopy funcs
Generating clientset for tensorflow:v1beta2,v1 at github.com/kubeflow/tf-operator/pkg/client/clientset
Generating listers for tensorflow:v1beta2,v1 at github.com/kubeflow/tf-operator/pkg/client/listers
Generating informers for tensorflow:v1beta2,v1 at github.com/kubeflow/tf-operator/pkg/client/informers
Generating defaulters for tensorflow/v1beta2
Generating OpenAPI specification for common/v1beta2
hack/../hack/update-codegen.sh: line 49: /home/travis/gopath/bin/openapi-gen: No such file or directory
goveralls -service=travis-ci -v -package ./pkg/... -ignore "pkg/client/*/*.go,pkg/client/*/*/*.go,pkg/client/*/*/*/*.go,pkg/client/*/*/*/*/*.go,pkg/client/*/*/*/*/*/*.go,pkg/client/*/*/*/*/*/*/*.go,pkg/util/testutil/*.go,pkg/apis/tensorflow/*/zz_generated.*.go,pkg/apis/tensorflow/*/*_generated.go,pkg/apis/common/*/zz_generated.*.go,pkg/apis/common/*/*_generated.go"
?   	github.com/kubeflow/tf-operator/pkg/apis/common/v1	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/apis/common/v1beta2	[no test files]
=== RUN   TestSetTypeNames
--- PASS: TestSetTypeNames (0.00s)
=== RUN   TestSetDefaultTFJob
--- PASS: TestSetDefaultTFJob (0.00s)
=== RUN   TestIsChieforMaster
--- PASS: TestIsChieforMaster (0.00s)
PASS
coverage: 20.9% of statements in github.com/kubeflow/tf-operator/pkg/apis/common/v1, github.com/kubeflow/tf-operator/pkg/apis/common/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/controller.v1beta2/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
ok  	github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1	0.029s	coverage: 20.9% of statements in github.com/kubeflow/tf-operator/pkg/apis/common/v1, github.com/kubeflow/tf-operator/pkg/apis/common/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/controller.v1beta2/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
=== RUN   TestSetTypeNames
--- PASS: TestSetTypeNames (0.00s)
=== RUN   TestSetDefaultTFJob
--- PASS: TestSetDefaultTFJob (0.00s)
=== RUN   TestIsChieforMaster
--- PASS: TestIsChieforMaster (0.00s)
PASS
coverage: 20.9% of statements in github.com/kubeflow/tf-operator/pkg/apis/common/v1, github.com/kubeflow/tf-operator/pkg/apis/common/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/controller.v1beta2/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
ok  	github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1beta2	0.030s	coverage: 20.9% of statements in github.com/kubeflow/tf-operator/pkg/apis/common/v1, github.com/kubeflow/tf-operator/pkg/apis/common/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/controller.v1beta2/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
=== RUN   TestValidateBetaTwoTFJobSpec
time="2019-08-19T08:29:09Z" level=error msg="TFJobSpec is not valid: Image is undefined in the container of Worker"
time="2019-08-19T08:29:09Z" level=error msg="TFJobSpec is not valid: There is no container named tensorflow in Worker"
--- PASS: TestValidateBetaTwoTFJobSpec (0.00s)
=== RUN   TestValidateV1TFJobSpec
time="2019-08-19T08:29:09Z" level=error msg="TFJobSpec is not valid: Image is undefined in the container of Worker"
time="2019-08-19T08:29:09Z" level=error msg="TFJobSpec is not valid: There is no container named tensorflow in Worker"
--- PASS: TestValidateV1TFJobSpec (0.00s)
PASS
coverage: 14.2% of statements in github.com/kubeflow/tf-operator/pkg/apis/common/v1, github.com/kubeflow/tf-operator/pkg/apis/common/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/controller.v1beta2/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
ok  	github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation	0.030s	coverage: 14.2% of statements in github.com/kubeflow/tf-operator/pkg/apis/common/v1, github.com/kubeflow/tf-operator/pkg/apis/common/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/controller.v1beta2/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
?   	github.com/kubeflow/tf-operator/pkg/client/clientset/versioned	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2/fake	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/informers/externalversions	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1beta2	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1beta2	[no test files]
=== RUN   TestGenGeneralName
--- PASS: TestGenGeneralName (0.00s)
PASS
coverage: 0.4% of statements in github.com/kubeflow/tf-operator/pkg/apis/common/v1, github.com/kubeflow/tf-operator/pkg/apis/common/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/controller.v1beta2/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
ok  	github.com/kubeflow/tf-operator/pkg/common/jobcontroller	0.021s	coverage: 0.4% of statements in github.com/kubeflow/tf-operator/pkg/apis/common/v1, github.com/kubeflow/tf-operator/pkg/apis/common/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/controller.v1beta2/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
?   	github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/testutil	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/unstructured	[no test files]
=== RUN   TestCreatePods
--- PASS: TestCreatePods (0.01s)
=== RUN   TestCreateService
time="2019-08-19T08:29:27Z" level=info msg="Controller test-tfjob created service empty_service"
--- PASS: TestCreateService (0.00s)
=== RUN   TestCreateServicesWithControllerRef
time="2019-08-19T08:29:27Z" level=info msg="Controller test-tfjob created service empty_service"
--- PASS: TestCreateServicesWithControllerRef (0.00s)
=== RUN   TestClaimServices
--- PASS: TestClaimServices (0.00s)
PASS
coverage: 41.1% of statements in github.com/kubeflow/tf-operator/pkg/apis/common/v1, github.com/kubeflow/tf-operator/pkg/apis/common/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/controller.v1beta2/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
ok  	github.com/kubeflow/tf-operator/pkg/control	0.058s	coverage: 41.1% of statements in github.com/kubeflow/tf-operator/pkg/apis/common/v1, github.com/kubeflow/tf-operator/pkg/apis/common/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/controller.v1beta2/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
=== RUN   TestNormalPath
time="2019-08-19T08:29:34Z" level=info msg="Creating TFJob controller"
time="2019-08-19T08:29:34Z" level=info msg="Creating Job controller"
time="2019-08-19T08:29:34Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-19T08:29:34Z" level=info msg="Need to create new pod: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-08-19T08:29:34Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=1, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-19T08:29:34Z" level=info msg="need to create new service: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-08-19T08:29:34Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (2.429393ms)" job=default.test-tfjob
time="2019-08-19T08:29:34Z" level=info msg="Creating TFJob controller"
time="2019-08-19T08:29:34Z" level=info msg="Creating Job controller"
time="2019-08-19T08:29:34Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-19T08:29:34Z" level=info msg="Need to create new pod: ps-0" job=default.test-tfjob replica-type=ps uid=
time="2019-08-19T08:29:34Z" level=info msg="Need to create new pod: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2019-08-19T08:29:34Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-19T08:29:34Z" level=info msg="need to create new service: ps-0" job=default.test-tfjob replica-type=ps uid=
time="2019-08-19T08:29:34Z" level=info msg="need to create new service: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2019-08-19T08:29:34Z" level=info msg="Need to create new pod: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-08-19T08:29:34Z" level=info msg="Need to create new pod: worker-1" job=default.test-tfjob replica-type=worker uid=
time="2019-08-19T08:29:34Z" level=info msg="Need to create new pod: worker-2" job=default.test-tfjob replica-type=worker uid=
time="2019-08-19T08:29:34Z" level=info msg="Need to create new pod: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2019-08-19T08:29:34Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-19T08:29:34Z" level=info msg="need to create new service: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-08-19T08:29:34Z" level=info msg="need to create new service: worker-1" job=default.test-tfjob replica-type=worker uid=
time="2019-08-19T08:29:34Z" level=info msg="need to create new service: worker-2" job=default.test-tfjob replica-type=worker uid=
time="2019-08-19T08:29:34Z" level=info msg="need to create new service: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2019-08-19T08:29:34Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (1.055091ms)" job=default.test-tfjob
time="2019-08-19T08:29:34Z" level=info msg="Creating TFJob controller"
time="2019-08-19T08:29:34Z" level=info msg="Creating Job controller"
time="2019-08-19T08:29:34Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-19T08:29:34Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-19T08:29:34Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-19T08:29:34Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (552.172µs)" job=default.test-tfjob
time="2019-08-19T08:29:34Z" level=info msg="Creating TFJob controller"
time="2019-08-19T08:29:34Z" level=info msg="Creating Job controller"
time="2019-08-19T08:29:34Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-19T08:29:34Z" level=info msg="Need to create new pod: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2019-08-19T08:29:34Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-19T08:29:34Z" level=info msg="need to create new service: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2019-08-19T08:29:34Z" level=info msg="Need to create new pod: worker-2" job=default.test-tfjob replica-type=worker uid=
time="2019-08-19T08:29:34Z" level=info msg="Need to create new pod: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2019-08-19T08:29:34Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-19T08:29:34Z" level=info msg="need to create new service: worker-2" job=default.test-tfjob replica-type=worker uid=
time="2019-08-19T08:29:34Z" level=info msg="need to create new service: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2019-08-19T08:29:34Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (2.256367ms)" job=default.test-tfjob
time="2019-08-19T08:29:34Z" level=info msg="Creating TFJob controller"
time="2019-08-19T08:29:34Z" level=info msg="Creating Job controller"
time="2019-08-19T08:29:34Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-19T08:29:34Z" level=info msg="Ignoring inactive pod default/worker-2 in state Succeeded, deletion time <nil>"
time="2019-08-19T08:29:34Z" level=info msg="Need to create new pod: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2019-08-19T08:29:34Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-19T08:29:34Z" level=info msg="need to create new service: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2019-08-19T08:29:34Z" level=info msg="Need to create new pod: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2019-08-19T08:29:34Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=3, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-19T08:29:34Z" level=info msg="need to create new service: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2019-08-19T08:29:34Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (921.608µs)" job=default.test-tfjob
time="2019-08-19T08:29:34Z" level=info msg="Creating TFJob controller"
time="2019-08-19T08:29:34Z" level=info msg="Creating Job controller"
time="2019-08-19T08:29:34Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-19T08:29:34Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=4, failed=0" job=default.test-tfjob uid=
time="2019-08-19T08:29:34Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-19T08:29:34Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (673.412µs)" job=default.test-tfjob
time="2019-08-19T08:29:34Z" level=info msg="Creating TFJob controller"
time="2019-08-19T08:29:34Z" level=info msg="Creating Job controller"
time="2019-08-19T08:29:34Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-19T08:29:34Z" level=info msg="Need to create new pod: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2019-08-19T08:29:34Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-19T08:29:34Z" level=info msg="need to create new service: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2019-08-19T08:29:34Z" level=info msg="Need to create new pod: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2019-08-19T08:29:34Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=1, failed=0" job=default.test-tfjob uid=
time="2019-08-19T08:29:34Z" level=info msg="need to create new service: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2019-08-19T08:29:34Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (1.214691ms)" job=default.test-tfjob
time="2019-08-19T08:29:34Z" level=info msg="Creating TFJob controller"
time="2019-08-19T08:29:34Z" level=info msg="Creating Job controller"
time="2019-08-19T08:29:34Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-19T08:29:34Z" level=info msg="Ignoring inactive pod default/ps-0 in state Succeeded, deletion time <nil>"
time="2019-08-19T08:29:34Z" level=info msg="Ignoring inactive pod default/ps-1 in state Succeeded, deletion time <nil>"
time="2019-08-19T08:29:34Z" level=info msg="Ignoring inactive pod default/worker-0 in state Succeeded, deletion time <nil>"
time="2019-08-19T08:29:34Z" level=info msg="Ignoring inactive pod default/worker-1 in state Succeeded, deletion time <nil>"
time="2019-08-19T08:29:34Z" level=info msg="Ignoring inactive pod default/worker-2 in state Succeeded, deletion time <nil>"
time="2019-08-19T08:29:34Z" level=info msg="Ignoring inactive pod default/worker-3 in state Succeeded, deletion time <nil>"
time="2019-08-19T08:29:34Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=0, running=0, failed=0" job=default.test-tfjob uid=
E0819 08:29:34.829375    9905 event.go:259] Could not construct reference to: '&v1.TFJob{TypeMeta:v1.TypeMeta{Kind:"TFJob", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"test-tfjob", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, Spec:v1.TFJobSpec{ActiveDeadlineSeconds:(*int64)(nil), BackoffLimit:(*int32)(nil), CleanPodPolicy:(*v1.CleanPodPolicy)(0xc4205bf180), TTLSecondsAfterFinished:(*int32)(nil), TFReplicaSpecs:map[v1.TFReplicaType]*v1.ReplicaSpec{"PS":(*v1.ReplicaSpec)(0xc42040e000), "Worker":(*v1.ReplicaSpec)(0xc42040e2c0)}}, Status:v1.JobStatus{Conditions:[]v1.JobCondition(nil), ReplicaStatuses:map[v1.ReplicaType]*v1.ReplicaStatus{"Worker":(*v1.ReplicaStatus)(0xc420428d70)}, StartTime:(*v1.Time)(0xc420512680), CompletionTime:(*v1.Time)(nil), LastReconcileTime:(*v1.Time)(nil)}}' due to: 'selfLink was empty, can't make reference'. Will not report event: 'Normal' 'TFJobSucceeded' 'TFJob test-tfjob successfully completed.'
time="2019-08-19T08:29:34Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-19T08:29:34Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (1.199716ms)" job=default.test-tfjob
--- PASS: TestNormalPath (0.02s)
=== RUN   TestRun
time="2019-08-19T08:29:34Z" level=info msg="Creating TFJob controller"
time="2019-08-19T08:29:34Z" level=info msg="Creating Job controller"
time="2019-08-19T08:29:34Z" level=info msg="Starting TFJob controller"
time="2019-08-19T08:29:34Z" level=info msg="Waiting for informer caches to sync"
time="2019-08-19T08:29:34Z" level=info msg="Starting 1 workers"
time="2019-08-19T08:29:34Z" level=info msg="Started workers"
time="2019-08-19T08:29:35Z" level=info msg="Shutting down workers"
--- PASS: TestRun (0.50s)
=== RUN   TestSyncPdb
time="2019-08-19T08:29:35Z" level=info msg="Creating TFJob controller"
time="2019-08-19T08:29:35Z" level=info msg="Creating Job controller"
--- PASS: TestSyncPdb (0.00s)
=== RUN   TestAddTFJob
time="2019-08-19T08:29:35Z" level=info msg="Creating TFJob controller"
time="2019-08-19T08:29:35Z" level=info msg="Creating Job controller"
time="2019-08-19T08:29:35Z" level=info msg="TFJob test-tfjob is created." job=default.test-tfjob uid=
time="2019-08-19T08:29:35Z" level=info msg="Starting TFJob controller"
time="2019-08-19T08:29:35Z" level=info msg="Waiting for informer caches to sync"
time="2019-08-19T08:29:35Z" level=info msg="Starting 1 workers"
time="2019-08-19T08:29:35Z" level=info msg="Started workers"
--- PASS: TestAddTFJob (0.10s)
=== RUN   TestCopyLabelsAndAnnotation
time="2019-08-19T08:29:35Z" level=info msg="Shutting down workers"
time="2019-08-19T08:29:35Z" level=info msg="Creating TFJob controller"
time="2019-08-19T08:29:35Z" level=info msg="Creating Job controller"
time="2019-08-19T08:29:35Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-19T08:29:35Z" level=info msg="Need to create new pod: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-08-19T08:29:35Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=1, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-19T08:29:35Z" level=info msg="need to create new service: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-08-19T08:29:35Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (746.466µs)" job=default.test-tfjob
--- PASS: TestCopyLabelsAndAnnotation (0.00s)
=== RUN   TestDeletePodsAndServices
time="2019-08-19T08:29:35Z" level=info msg="Creating TFJob controller"
time="2019-08-19T08:29:35Z" level=info msg="Creating Job controller"
time="2019-08-19T08:29:35Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-19T08:29:35Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (452.498µs)" job=default.test-tfjob
time="2019-08-19T08:29:35Z" level=info msg="Starting TFJob controller"
time="2019-08-19T08:29:35Z" level=info msg="Waiting for informer caches to sync"
time="2019-08-19T08:29:35Z" level=info msg="Starting 1 workers"
time="2019-08-19T08:29:35Z" level=info msg="Started workers"
time="2019-08-19T08:29:35Z" level=info msg="Shutting down workers"
time="2019-08-19T08:29:35Z" level=info msg="Creating TFJob controller"
time="2019-08-19T08:29:35Z" level=info msg="Creating Job controller"
time="2019-08-19T08:29:35Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-19T08:29:35Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (312.456µs)" job=default.test-tfjob
time="2019-08-19T08:29:35Z" level=info msg="Creating TFJob controller"
time="2019-08-19T08:29:35Z" level=info msg="Creating Job controller"
time="2019-08-19T08:29:35Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-19T08:29:35Z" level=info msg="Ignoring inactive pod default/worker-3 in state Succeeded, deletion time <nil>"
time="2019-08-19T08:29:35Z" level=info msg="Ignoring inactive pod default/ps-0 in state Succeeded, deletion time <nil>"
time="2019-08-19T08:29:35Z" level=info msg="Ignoring inactive pod default/ps-1 in state Succeeded, deletion time <nil>"
time="2019-08-19T08:29:35Z" level=info msg="Ignoring inactive pod default/worker-0 in state Succeeded, deletion time <nil>"
time="2019-08-19T08:29:35Z" level=info msg="Ignoring inactive pod default/worker-1 in state Succeeded, deletion time <nil>"
time="2019-08-19T08:29:35Z" level=info msg="Ignoring inactive pod default/worker-2 in state Succeeded, deletion time <nil>"
time="2019-08-19T08:29:35Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (378.701µs)" job=default.test-tfjob
time="2019-08-19T08:29:35Z" level=info msg="Creating TFJob controller"
time="2019-08-19T08:29:35Z" level=info msg="Creating Job controller"
time="2019-08-19T08:29:35Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-19T08:29:35Z" level=info msg="Ignoring inactive pod default/worker-2 in state Succeeded, deletion time <nil>"
time="2019-08-19T08:29:35Z" level=info msg="Ignoring inactive pod default/worker-3 in state Succeeded, deletion time <nil>"
time="2019-08-19T08:29:35Z" level=info msg="Ignoring inactive pod default/ps-0 in state Succeeded, deletion time <nil>"
time="2019-08-19T08:29:35Z" level=info msg="Ignoring inactive pod default/ps-1 in state Succeeded, deletion time <nil>"
time="2019-08-19T08:29:35Z" level=info msg="Ignoring inactive pod default/worker-0 in state Succeeded, deletion time <nil>"
time="2019-08-19T08:29:35Z" level=info msg="Ignoring inactive pod default/worker-1 in state Succeeded, deletion time <nil>"
time="2019-08-19T08:29:35Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (366.863µs)" job=default.test-tfjob
--- PASS: TestDeletePodsAndServices (0.01s)
=== RUN   TestCleanupTFJob
time="2019-08-19T08:29:35Z" level=info msg="Creating TFJob controller"
time="2019-08-19T08:29:35Z" level=info msg="Creating Job controller"
time="2019-08-19T08:29:35Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-19T08:29:35Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (300.78µs)" job=default.test-tfjob
time="2019-08-19T08:29:35Z" level=info msg="Creating TFJob controller"
time="2019-08-19T08:29:35Z" level=info msg="Creating Job controller"
time="2019-08-19T08:29:35Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-19T08:29:35Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (293.377µs)" job=default.test-tfjob
time="2019-08-19T08:29:35Z" level=info msg="Creating TFJob controller"
time="2019-08-19T08:29:35Z" level=info msg="Creating Job controller"
time="2019-08-19T08:29:37Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-19T08:29:37Z" level=info msg="Ignoring inactive pod default/ps-0 in state Succeeded, deletion time <nil>"
time="2019-08-19T08:29:37Z" level=info msg="Ignoring inactive pod default/ps-1 in state Succeeded, deletion time <nil>"
time="2019-08-19T08:29:37Z" level=info msg="Ignoring inactive pod default/worker-0 in state Succeeded, deletion time <nil>"
time="2019-08-19T08:29:37Z" level=info msg="Ignoring inactive pod default/worker-1 in state Succeeded, deletion time <nil>"
time="2019-08-19T08:29:37Z" level=info msg="Ignoring inactive pod default/worker-2 in state Succeeded, deletion time <nil>"
time="2019-08-19T08:29:37Z" level=info msg="Ignoring inactive pod default/worker-3 in state Succeeded, deletion time <nil>"
time="2019-08-19T08:29:37Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (742.982µs)" job=default.test-tfjob
--- PASS: TestCleanupTFJob (2.00s)
=== RUN   TestActiveDeadlineSeconds
time="2019-08-19T08:29:37Z" level=info msg="Creating TFJob controller"
time="2019-08-19T08:29:37Z" level=info msg="Creating Job controller"
time="2019-08-19T08:29:37Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-19T08:29:37Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-19T08:29:37Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=4, failed=0" job=default.test-tfjob uid=
time="2019-08-19T08:29:37Z" level=info msg="Creating TFJob controller"
time="2019-08-19T08:29:37Z" level=info msg="Creating Job controller"
time="2019-08-19T08:29:39Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
--- PASS: TestActiveDeadlineSeconds (2.00s)
=== RUN   TestBackoffForOnFailure
time="2019-08-19T08:29:39Z" level=info msg="Creating TFJob controller"
time="2019-08-19T08:29:39Z" level=info msg="Creating Job controller"
time="2019-08-19T08:29:39Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-19T08:29:39Z" level=warning msg="The restart policy of replica PS of the job test-tfjob is not OnFailure or Always. Not counted in backoff limit." job=default.test-tfjob uid=
time="2019-08-19T08:29:39Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (428.396µs)" job=default.test-tfjob
--- PASS: TestBackoffForOnFailure (0.00s)
=== RUN   TestAddPod
time="2019-08-19T08:29:39Z" level=info msg="Creating TFJob controller"
time="2019-08-19T08:29:39Z" level=info msg="Creating Job controller"
time="2019-08-19T08:29:39Z" level=info msg="Starting TFJob controller"
time="2019-08-19T08:29:39Z" level=info msg="Waiting for informer caches to sync"
time="2019-08-19T08:29:39Z" level=info msg="Starting 1 workers"
time="2019-08-19T08:29:39Z" level=info msg="Started workers"
--- PASS: TestAddPod (0.10s)
=== RUN   TestClusterSpec
time="2019-08-19T08:29:39Z" level=info msg="Shutting down workers"
--- PASS: TestClusterSpec (0.00s)
=== RUN   TestRestartPolicy
--- PASS: TestRestartPolicy (0.00s)
=== RUN   TestExitCode
time="2019-08-19T08:29:39Z" level=info msg="Creating TFJob controller"
time="2019-08-19T08:29:39Z" level=info msg="Creating Job controller"
time="2019-08-19T08:29:39Z" level=info msg="Starting TFJob controller"
time="2019-08-19T08:29:39Z" level=info msg="Waiting for informer caches to sync"
time="2019-08-19T08:29:39Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-19T08:29:39Z" level=info msg="Ignoring inactive pod default/worker-0 in state Failed, deletion time <nil>"
time="2019-08-19T08:29:39Z" level=info msg="Pod: default.worker-0 exited with code 130" job=default.test-tfjob replica-type=worker uid=
E0819 08:29:39.559921    9905 event.go:259] Could not construct reference to: '&v1.TFJob{TypeMeta:v1.TypeMeta{Kind:"TFJob", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"test-tfjob", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, Spec:v1.TFJobSpec{ActiveDeadlineSeconds:(*int64)(nil), BackoffLimit:(*int32)(nil), CleanPodPolicy:(*v1.CleanPodPolicy)(0xc420a45580), TTLSecondsAfterFinished:(*int32)(nil), TFReplicaSpecs:map[v1.TFReplicaType]*v1.ReplicaSpec{"Worker":(*v1.ReplicaSpec)(0xc420a32b00)}}, Status:v1.JobStatus{Conditions:[]v1.JobCondition(nil), ReplicaStatuses:map[v1.ReplicaType]*v1.ReplicaStatus{"Worker":(*v1.ReplicaStatus)(0xc420a3acb0)}, StartTime:(*v1.Time)(nil), CompletionTime:(*v1.Time)(nil), LastReconcileTime:(*v1.Time)(nil)}}' due to: 'selfLink was empty, can't make reference'. Will not report event: 'Normal' 'ExitedWithCode' 'Pod: default.worker-0 exited with code 130'
time="2019-08-19T08:29:39Z" level=info msg="Need to restart the pod: default.worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-08-19T08:29:39Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=1, running=0, failed=1" job=default.test-tfjob uid=
E0819 08:29:39.560048    9905 event.go:259] Could not construct reference to: '&v1.TFJob{TypeMeta:v1.TypeMeta{Kind:"TFJob", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"test-tfjob", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, Spec:v1.TFJobSpec{ActiveDeadlineSeconds:(*int64)(nil), BackoffLimit:(*int32)(nil), CleanPodPolicy:(*v1.CleanPodPolicy)(0xc420a45580), TTLSecondsAfterFinished:(*int32)(nil), TFReplicaSpecs:map[v1.TFReplicaType]*v1.ReplicaSpec{"Worker":(*v1.ReplicaSpec)(0xc420a32b00)}}, Status:v1.JobStatus{Conditions:[]v1.JobCondition(nil), ReplicaStatuses:map[v1.ReplicaType]*v1.ReplicaStatus{"Worker":(*v1.ReplicaStatus)(0xc420a3acb0)}, StartTime:(*v1.Time)(0xc420a4d020), CompletionTime:(*v1.Time)(nil), LastReconcileTime:(*v1.Time)(nil)}}' due to: 'selfLink was empty, can't make reference'. Will not report event: 'Warning' 'TFJobRestarting' 'TFJob test-tfjob is restarting because 1 Worker replica(s) failed.'
time="2019-08-19T08:29:39Z" level=info msg="need to create new service: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-08-19T08:29:39Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (652.972µs)" job=default.test-tfjob
--- PASS: TestExitCode (0.00s)
=== RUN   TestAddService
time="2019-08-19T08:29:39Z" level=info msg="Starting 1 workers"
time="2019-08-19T08:29:39Z" level=info msg="Started workers"
time="2019-08-19T08:29:39Z" level=info msg="Shutting down workers"
time="2019-08-19T08:29:39Z" level=info msg="Creating TFJob controller"
time="2019-08-19T08:29:39Z" level=info msg="Creating Job controller"
time="2019-08-19T08:29:39Z" level=info msg="Starting TFJob controller"
time="2019-08-19T08:29:39Z" level=info msg="Waiting for informer caches to sync"
time="2019-08-19T08:29:39Z" level=info msg="Starting 1 workers"
time="2019-08-19T08:29:39Z" level=info msg="Started workers"
--- PASS: TestAddService (0.10s)
=== RUN   TestFailed
time="2019-08-19T08:29:39Z" level=info msg="Shutting down workers"
time="2019-08-19T08:29:39Z" level=info msg="Creating TFJob controller"
time="2019-08-19T08:29:39Z" level=info msg="Creating Job controller"
time="2019-08-19T08:29:39Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=3, running=0, failed=1" job=default.test-tfjob uid=
E0819 08:29:39.663034    9905 event.go:259] Could not construct reference to: '&v1.TFJob{TypeMeta:v1.TypeMeta{Kind:"TFJob", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"test-tfjob", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, Spec:v1.TFJobSpec{ActiveDeadlineSeconds:(*int64)(nil), BackoffLimit:(*int32)(nil), CleanPodPolicy:(*v1.CleanPodPolicy)(0xc420acab10), TTLSecondsAfterFinished:(*int32)(nil), TFReplicaSpecs:map[v1.TFReplicaType]*v1.ReplicaSpec{"Worker":(*v1.ReplicaSpec)(0xc420aee000)}}, Status:v1.JobStatus{Conditions:[]v1.JobCondition(nil), ReplicaStatuses:map[v1.ReplicaType]*v1.ReplicaStatus{"Worker":(*v1.ReplicaStatus)(0xc4209f51b0)}, StartTime:(*v1.Time)(0xc420ab7cc0), CompletionTime:(*v1.Time)(nil), LastReconcileTime:(*v1.Time)(nil)}}' due to: 'selfLink was empty, can't make reference'. Will not report event: 'Normal' 'TFJobFailed' 'TFJob test-tfjob has failed because 1 Worker replica(s) failed.'
--- PASS: TestFailed (0.00s)
=== RUN   TestStatus
time="2019-08-19T08:29:39Z" level=info msg="Creating TFJob controller"
time="2019-08-19T08:29:39Z" level=info msg="Creating Job controller"
time="2019-08-19T08:29:39Z" level=info msg="TFJob=test-tfjob, ReplicaType=Chief expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-19T08:29:39Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-19T08:29:39Z" level=info msg="Creating TFJob controller"
time="2019-08-19T08:29:39Z" level=info msg="Creating Job controller"
time="2019-08-19T08:29:39Z" level=info msg="TFJob=test-tfjob, ReplicaType=Chief expected=1, running=1, failed=0" job=default.test-tfjob uid=
time="2019-08-19T08:29:39Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=1, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-19T08:29:39Z" level=info msg="Creating TFJob controller"
time="2019-08-19T08:29:39Z" level=info msg="Creating Job controller"
time="2019-08-19T08:29:39Z" level=info msg="TFJob=test-tfjob, ReplicaType=Chief expected=1, running=0, failed=1" job=default.test-tfjob uid=
time="2019-08-19T08:29:39Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=1, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-19T08:29:39Z" level=info msg="Creating TFJob controller"
time="2019-08-19T08:29:39Z" level=info msg="Creating Job controller"
time="2019-08-19T08:29:39Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=1, running=0, failed=1" job=default.test-tfjob uid=
time="2019-08-19T08:29:39Z" level=info msg="Creating TFJob controller"
time="2019-08-19T08:29:39Z" level=info msg="Creating Job controller"
time="2019-08-19T08:29:39Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-19T08:29:39Z" level=info msg="Creating TFJob controller"
time="2019-08-19T08:29:39Z" level=info msg="Creating Job controller"
time="2019-08-19T08:29:39Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=1, running=1, failed=0" job=default.test-tfjob uid=
time="2019-08-19T08:29:39Z" level=info msg="Creating TFJob controller"
time="2019-08-19T08:29:39Z" level=info msg="Creating Job controller"
time="2019-08-19T08:29:39Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-19T08:29:39Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-19T08:29:39Z" level=info msg="Creating TFJob controller"
time="2019-08-19T08:29:39Z" level=info msg="Creating Job controller"
time="2019-08-19T08:29:39Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=2, failed=2" job=default.test-tfjob uid=
time="2019-08-19T08:29:39Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-19T08:29:39Z" level=info msg="Creating TFJob controller"
time="2019-08-19T08:29:39Z" level=info msg="Creating Job controller"
time="2019-08-19T08:29:39Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=2, running=0, failed=2" job=default.test-tfjob uid=
time="2019-08-19T08:29:39Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-19T08:29:39Z" level=info msg="Creating TFJob controller"
time="2019-08-19T08:29:39Z" level=info msg="Creating Job controller"
time="2019-08-19T08:29:39Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=3, running=3, failed=0" job=default.test-tfjob uid=
time="2019-08-19T08:29:39Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-19T08:29:39Z" level=info msg="Creating TFJob controller"
time="2019-08-19T08:29:39Z" level=info msg="Creating Job controller"
time="2019-08-19T08:29:39Z" level=info msg="TFJob=test-tfjob, ReplicaType=Chief expected=1, running=1, failed=0" job=default.test-tfjob uid=
time="2019-08-19T08:29:39Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=0, failed=4" job=default.test-tfjob uid=
time="2019-08-19T08:29:39Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-19T08:29:39Z" level=info msg="Creating TFJob controller"
time="2019-08-19T08:29:39Z" level=info msg="Creating Job controller"
time="2019-08-19T08:29:39Z" level=info msg="TFJob=test-tfjob, ReplicaType=Chief expected=1, running=1, failed=0" job=default.test-tfjob uid=
time="2019-08-19T08:29:39Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-19T08:29:39Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-19T08:29:39Z" level=info msg="Creating TFJob controller"
time="2019-08-19T08:29:39Z" level=info msg="Creating Job controller"
time="2019-08-19T08:29:39Z" level=info msg="TFJob=test-tfjob, ReplicaType=Chief expected=1, running=1, failed=0" job=default.test-tfjob uid=
time="2019-08-19T08:29:39Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-19T08:29:39Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=1, failed=1" job=default.test-tfjob uid=
time="2019-08-19T08:29:39Z" level=info msg="Creating TFJob controller"
time="2019-08-19T08:29:39Z" level=info msg="Creating Job controller"
time="2019-08-19T08:29:39Z" level=info msg="TFJob=test-tfjob, ReplicaType=Chief expected=1, running=0, failed=1" job=default.test-tfjob uid=
time="2019-08-19T08:29:39Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-19T08:29:39Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-19T08:29:39Z" level=info msg="Creating TFJob controller"
time="2019-08-19T08:29:39Z" level=info msg="Creating Job controller"
time="2019-08-19T08:29:39Z" level=info msg="TFJob=test-tfjob, ReplicaType=Chief expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-19T08:29:39Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=0, failed=4" job=default.test-tfjob uid=
time="2019-08-19T08:29:39Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-19T08:29:39Z" level=info msg="Creating TFJob controller"
time="2019-08-19T08:29:39Z" level=info msg="Creating Job controller"
time="2019-08-19T08:29:39Z" level=info msg="TFJob=test-tfjob, ReplicaType=Chief expected=1, running=0, failed=1" job=default.test-tfjob uid=
time="2019-08-19T08:29:39Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=0, failed=4" job=default.test-tfjob uid=
time="2019-08-19T08:29:39Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
--- PASS: TestStatus (0.02s)
=== RUN   TestGenOwnerReference
--- PASS: TestGenOwnerReference (0.00s)
=== RUN   TestGenLabels
--- PASS: TestGenLabels (0.00s)
=== RUN   TestConvertTFJobToUnstructured
--- PASS: TestConvertTFJobToUnstructured (0.00s)
PASS
coverage: 43.9% of statements in github.com/kubeflow/tf-operator/pkg/apis/common/v1, github.com/kubeflow/tf-operator/pkg/apis/common/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/controller.v1beta2/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
ok  	github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow	4.926s	coverage: 43.9% of statements in github.com/kubeflow/tf-operator/pkg/apis/common/v1, github.com/kubeflow/tf-operator/pkg/apis/common/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/controller.v1beta2/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
=== RUN   TestNormalPath
time="2019-08-19T08:29:45Z" level=info msg="Creating TFJob controller"
time="2019-08-19T08:29:45Z" level=info msg="Creating Job controller"
time="2019-08-19T08:29:45Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-19T08:29:45Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-19T08:29:45Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=4, failed=0" job=default.test-tfjob uid=
time="2019-08-19T08:29:45Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (1.707445ms)" job=default.test-tfjob
time="2019-08-19T08:29:45Z" level=info msg="Creating TFJob controller"
time="2019-08-19T08:29:45Z" level=info msg="Creating Job controller"
time="2019-08-19T08:29:45Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-19T08:29:45Z" level=info msg="Need to create new pod: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2019-08-19T08:29:45Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-19T08:29:45Z" level=info msg="need to create new service: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2019-08-19T08:29:45Z" level=info msg="Need to create new pod: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2019-08-19T08:29:45Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=1, failed=0" job=default.test-tfjob uid=
time="2019-08-19T08:29:45Z" level=info msg="need to create new service: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2019-08-19T08:29:45Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (815.253µs)" job=default.test-tfjob
time="2019-08-19T08:29:45Z" level=info msg="Creating TFJob controller"
time="2019-08-19T08:29:45Z" level=info msg="Creating Job controller"
time="2019-08-19T08:29:45Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-19T08:29:45Z" level=info msg="Ignoring inactive pod default/ps-1 in state Succeeded, deletion time <nil>"
time="2019-08-19T08:29:45Z" level=info msg="Ignoring inactive pod default/worker-0 in state Succeeded, deletion time <nil>"
time="2019-08-19T08:29:45Z" level=info msg="Ignoring inactive pod default/worker-1 in state Succeeded, deletion time <nil>"
time="2019-08-19T08:29:45Z" level=info msg="Ignoring inactive pod default/worker-2 in state Succeeded, deletion time <nil>"
time="2019-08-19T08:29:45Z" level=info msg="Ignoring inactive pod default/worker-3 in state Succeeded, deletion time <nil>"
time="2019-08-19T08:29:45Z" level=info msg="Ignoring inactive pod default/ps-0 in state Succeeded, deletion time <nil>"
time="2019-08-19T08:29:45Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-19T08:29:45Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=0, running=0, failed=0" job=default.test-tfjob uid=
E0819 08:29:45.783455   10018 event.go:259] Could not construct reference to: '&v1beta2.TFJob{TypeMeta:v1.TypeMeta{Kind:"TFJob", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"test-tfjob", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, Spec:v1beta2.TFJobSpec{ActiveDeadlineSeconds:(*int64)(nil), BackoffLimit:(*int32)(nil), CleanPodPolicy:(*v1beta2.CleanPodPolicy)(0xc4203c98f0), TTLSecondsAfterFinished:(*int32)(nil), TFReplicaSpecs:map[v1beta2.TFReplicaType]*v1beta2.ReplicaSpec{"PS":(*v1beta2.ReplicaSpec)(0xc42012b8c0), "Worker":(*v1beta2.ReplicaSpec)(0xc42012bb80)}}, Status:v1beta2.JobStatus{Conditions:[]v1beta2.JobCondition(nil), ReplicaStatuses:map[v1beta2.ReplicaType]*v1beta2.ReplicaStatus{"PS":(*v1beta2.ReplicaStatus)(0xc4204fbf50), "Worker":(*v1beta2.ReplicaStatus)(0xc4204fbf80)}, StartTime:(*v1.Time)(0xc4203e7500), CompletionTime:(*v1.Time)(nil), LastReconcileTime:(*v1.Time)(nil)}}' due to: 'selfLink was empty, can't make reference'. Will not report event: 'Normal' 'TFJobSucceeded' 'TFJob test-tfjob successfully completed.'
time="2019-08-19T08:29:45Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (1.903949ms)" job=default.test-tfjob
time="2019-08-19T08:29:45Z" level=info msg="Creating TFJob controller"
time="2019-08-19T08:29:45Z" level=info msg="Creating Job controller"
time="2019-08-19T08:29:45Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-19T08:29:45Z" level=info msg="Need to create new pod: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-08-19T08:29:45Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=1, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-19T08:29:45Z" level=info msg="need to create new service: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-08-19T08:29:45Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (579.193µs)" job=default.test-tfjob
time="2019-08-19T08:29:45Z" level=info msg="Creating TFJob controller"
time="2019-08-19T08:29:45Z" level=info msg="Creating Job controller"
time="2019-08-19T08:29:45Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-19T08:29:45Z" level=info msg="Need to create new pod: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-08-19T08:29:45Z" level=info msg="Need to create new pod: worker-1" job=default.test-tfjob replica-type=worker uid=
time="2019-08-19T08:29:45Z" level=info msg="Need to create new pod: worker-2" job=default.test-tfjob replica-type=worker uid=
time="2019-08-19T08:29:45Z" level=info msg="Need to create new pod: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2019-08-19T08:29:45Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-19T08:29:45Z" level=info msg="need to create new service: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-08-19T08:29:45Z" level=info msg="need to create new service: worker-1" job=default.test-tfjob replica-type=worker uid=
time="2019-08-19T08:29:45Z" level=info msg="need to create new service: worker-2" job=default.test-tfjob replica-type=worker uid=
time="2019-08-19T08:29:45Z" level=info msg="need to create new service: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2019-08-19T08:29:45Z" level=info msg="Need to create new pod: ps-0" job=default.test-tfjob replica-type=ps uid=
time="2019-08-19T08:29:45Z" level=info msg="Need to create new pod: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2019-08-19T08:29:45Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-19T08:29:45Z" level=info msg="need to create new service: ps-0" job=default.test-tfjob replica-type=ps uid=
time="2019-08-19T08:29:45Z" level=info msg="need to create new service: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2019-08-19T08:29:45Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (2.439896ms)" job=default.test-tfjob
time="2019-08-19T08:29:45Z" level=info msg="Creating TFJob controller"
time="2019-08-19T08:29:45Z" level=info msg="Creating Job controller"
time="2019-08-19T08:29:45Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-19T08:29:45Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-19T08:29:45Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-19T08:29:45Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (649.019µs)" job=default.test-tfjob
time="2019-08-19T08:29:45Z" level=info msg="Creating TFJob controller"
time="2019-08-19T08:29:45Z" level=info msg="Creating Job controller"
time="2019-08-19T08:29:45Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-19T08:29:45Z" level=info msg="Need to create new pod: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2019-08-19T08:29:45Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-19T08:29:45Z" level=info msg="need to create new service: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2019-08-19T08:29:45Z" level=info msg="Need to create new pod: worker-2" job=default.test-tfjob replica-type=worker uid=
time="2019-08-19T08:29:45Z" level=info msg="Need to create new pod: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2019-08-19T08:29:45Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-19T08:29:45Z" level=info msg="need to create new service: worker-2" job=default.test-tfjob replica-type=worker uid=
time="2019-08-19T08:29:45Z" level=info msg="need to create new service: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2019-08-19T08:29:45Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (1.462233ms)" job=default.test-tfjob
time="2019-08-19T08:29:45Z" level=info msg="Creating TFJob controller"
time="2019-08-19T08:29:45Z" level=info msg="Creating Job controller"
time="2019-08-19T08:29:45Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-19T08:29:45Z" level=info msg="Ignoring inactive pod default/worker-2 in state Succeeded, deletion time <nil>"
time="2019-08-19T08:29:45Z" level=info msg="Need to create new pod: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2019-08-19T08:29:45Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-19T08:29:45Z" level=info msg="need to create new service: ps-1" job=default.test-tfjob replica-type=ps uid=
time="2019-08-19T08:29:45Z" level=info msg="Need to create new pod: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2019-08-19T08:29:45Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=3, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-19T08:29:45Z" level=info msg="need to create new service: worker-3" job=default.test-tfjob replica-type=worker uid=
time="2019-08-19T08:29:45Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (557.453µs)" job=default.test-tfjob
--- PASS: TestNormalPath (0.03s)
=== RUN   TestRun
time="2019-08-19T08:29:45Z" level=info msg="Creating TFJob controller"
time="2019-08-19T08:29:45Z" level=info msg="Creating Job controller"
time="2019-08-19T08:29:45Z" level=info msg="Starting TFJob controller"
time="2019-08-19T08:29:45Z" level=info msg="Waiting for informer caches to sync"
time="2019-08-19T08:29:45Z" level=info msg="Starting 1 workers"
time="2019-08-19T08:29:45Z" level=info msg="Started workers"
time="2019-08-19T08:29:46Z" level=info msg="Shutting down workers"
--- PASS: TestRun (0.50s)
=== RUN   TestSyncPdb
time="2019-08-19T08:29:46Z" level=info msg="Creating TFJob controller"
time="2019-08-19T08:29:46Z" level=info msg="Creating Job controller"
--- PASS: TestSyncPdb (0.00s)
=== RUN   TestAddTFJob
time="2019-08-19T08:29:46Z" level=info msg="Creating TFJob controller"
time="2019-08-19T08:29:46Z" level=info msg="Creating Job controller"
time="2019-08-19T08:29:46Z" level=info msg="Starting TFJob controller"
time="2019-08-19T08:29:46Z" level=info msg="Waiting for informer caches to sync"
time="2019-08-19T08:29:46Z" level=info msg="TFJob test-tfjob is created." job=default.test-tfjob uid=
time="2019-08-19T08:29:46Z" level=info msg="Starting 1 workers"
time="2019-08-19T08:29:46Z" level=info msg="Started workers"
--- PASS: TestAddTFJob (0.10s)
time="2019-08-19T08:29:46Z" level=info msg="Shutting down workers"
=== RUN   TestCopyLabelsAndAnnotation
time="2019-08-19T08:29:46Z" level=info msg="Creating TFJob controller"
time="2019-08-19T08:29:46Z" level=info msg="Creating Job controller"
time="2019-08-19T08:29:46Z" level=info msg="Starting TFJob controller"
time="2019-08-19T08:29:46Z" level=info msg="Waiting for informer caches to sync"
time="2019-08-19T08:29:46Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-19T08:29:46Z" level=info msg="Need to create new pod: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-08-19T08:29:46Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=1, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-19T08:29:46Z" level=info msg="need to create new service: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-08-19T08:29:46Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (1.084154ms)" job=default.test-tfjob
--- PASS: TestCopyLabelsAndAnnotation (0.00s)
=== RUN   TestDeletePodsAndServices
time="2019-08-19T08:29:46Z" level=info msg="Starting 1 workers"
time="2019-08-19T08:29:46Z" level=info msg="Started workers"
time="2019-08-19T08:29:46Z" level=info msg="Shutting down workers"
time="2019-08-19T08:29:46Z" level=info msg="Creating TFJob controller"
time="2019-08-19T08:29:46Z" level=info msg="Creating Job controller"
time="2019-08-19T08:29:46Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-19T08:29:46Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-19T08:29:46Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=4, failed=0" job=default.test-tfjob uid=
time="2019-08-19T08:29:46Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (545.176µs)" job=default.test-tfjob
time="2019-08-19T08:29:46Z" level=info msg="Creating TFJob controller"
time="2019-08-19T08:29:46Z" level=info msg="Creating Job controller"
time="2019-08-19T08:29:46Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-19T08:29:46Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-19T08:29:46Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=4, failed=0" job=default.test-tfjob uid=
time="2019-08-19T08:29:46Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (491.983µs)" job=default.test-tfjob
time="2019-08-19T08:29:46Z" level=info msg="Creating TFJob controller"
time="2019-08-19T08:29:46Z" level=info msg="Creating Job controller"
time="2019-08-19T08:29:46Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-19T08:29:46Z" level=info msg="Ignoring inactive pod default/worker-0 in state Succeeded, deletion time <nil>"
time="2019-08-19T08:29:46Z" level=info msg="Ignoring inactive pod default/worker-1 in state Succeeded, deletion time <nil>"
time="2019-08-19T08:29:46Z" level=info msg="Ignoring inactive pod default/worker-2 in state Succeeded, deletion time <nil>"
time="2019-08-19T08:29:46Z" level=info msg="Ignoring inactive pod default/worker-3 in state Succeeded, deletion time <nil>"
time="2019-08-19T08:29:46Z" level=info msg="Ignoring inactive pod default/ps-0 in state Succeeded, deletion time <nil>"
time="2019-08-19T08:29:46Z" level=info msg="Ignoring inactive pod default/ps-1 in state Succeeded, deletion time <nil>"
time="2019-08-19T08:29:46Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-19T08:29:46Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-19T08:29:46Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (1.035292ms)" job=default.test-tfjob
time="2019-08-19T08:29:46Z" level=info msg="Creating TFJob controller"
time="2019-08-19T08:29:46Z" level=info msg="Creating Job controller"
time="2019-08-19T08:29:46Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-19T08:29:46Z" level=info msg="Ignoring inactive pod default/worker-1 in state Succeeded, deletion time <nil>"
time="2019-08-19T08:29:46Z" level=info msg="Ignoring inactive pod default/worker-2 in state Succeeded, deletion time <nil>"
time="2019-08-19T08:29:46Z" level=info msg="Ignoring inactive pod default/worker-3 in state Succeeded, deletion time <nil>"
time="2019-08-19T08:29:46Z" level=info msg="Ignoring inactive pod default/ps-0 in state Succeeded, deletion time <nil>"
time="2019-08-19T08:29:46Z" level=info msg="Ignoring inactive pod default/ps-1 in state Succeeded, deletion time <nil>"
time="2019-08-19T08:29:46Z" level=info msg="Ignoring inactive pod default/worker-0 in state Succeeded, deletion time <nil>"
time="2019-08-19T08:29:46Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-19T08:29:46Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-19T08:29:46Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (1.094003ms)" job=default.test-tfjob
--- PASS: TestDeletePodsAndServices (0.01s)
=== RUN   TestCleanupTFJob
time="2019-08-19T08:29:46Z" level=info msg="Creating TFJob controller"
time="2019-08-19T08:29:46Z" level=info msg="Creating Job controller"
time="2019-08-19T08:29:46Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-19T08:29:46Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-19T08:29:46Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=4, failed=0" job=default.test-tfjob uid=
time="2019-08-19T08:29:46Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (647.495µs)" job=default.test-tfjob
time="2019-08-19T08:29:46Z" level=info msg="Creating TFJob controller"
time="2019-08-19T08:29:46Z" level=info msg="Creating Job controller"
time="2019-08-19T08:29:46Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-19T08:29:46Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=4, failed=0" job=default.test-tfjob uid=
time="2019-08-19T08:29:46Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-19T08:29:46Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (1.142099ms)" job=default.test-tfjob
time="2019-08-19T08:29:46Z" level=info msg="Creating TFJob controller"
time="2019-08-19T08:29:46Z" level=info msg="Creating Job controller"
time="2019-08-19T08:29:48Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-19T08:29:48Z" level=info msg="Ignoring inactive pod default/worker-3 in state Succeeded, deletion time <nil>"
time="2019-08-19T08:29:48Z" level=info msg="Ignoring inactive pod default/ps-0 in state Succeeded, deletion time <nil>"
time="2019-08-19T08:29:48Z" level=info msg="Ignoring inactive pod default/ps-1 in state Succeeded, deletion time <nil>"
time="2019-08-19T08:29:48Z" level=info msg="Ignoring inactive pod default/worker-0 in state Succeeded, deletion time <nil>"
time="2019-08-19T08:29:48Z" level=info msg="Ignoring inactive pod default/worker-1 in state Succeeded, deletion time <nil>"
time="2019-08-19T08:29:48Z" level=info msg="Ignoring inactive pod default/worker-2 in state Succeeded, deletion time <nil>"
time="2019-08-19T08:29:48Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-19T08:29:48Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-19T08:29:48Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (948.138µs)" job=default.test-tfjob
--- PASS: TestCleanupTFJob (2.01s)
=== RUN   TestActiveDeadlineSeconds
time="2019-08-19T08:29:48Z" level=info msg="Creating TFJob controller"
time="2019-08-19T08:29:48Z" level=info msg="Creating Job controller"
time="2019-08-19T08:29:48Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-19T08:29:48Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=4, failed=0" job=default.test-tfjob uid=
time="2019-08-19T08:29:48Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-19T08:29:48Z" level=info msg="Creating TFJob controller"
time="2019-08-19T08:29:48Z" level=info msg="Creating Job controller"
time="2019-08-19T08:29:50Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
--- PASS: TestActiveDeadlineSeconds (2.00s)
=== RUN   TestBackoffForOnFailure
time="2019-08-19T08:29:50Z" level=info msg="Creating TFJob controller"
time="2019-08-19T08:29:50Z" level=info msg="Creating Job controller"
time="2019-08-19T08:29:50Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-19T08:29:50Z" level=warning msg="The restart policy of replica PS of the job test-tfjob is not OnFailure or Always. Not counted in backoff limit." job=default.test-tfjob uid=
time="2019-08-19T08:29:50Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (439µs)" job=default.test-tfjob
--- PASS: TestBackoffForOnFailure (0.00s)
=== RUN   TestAddPod
time="2019-08-19T08:29:50Z" level=info msg="Creating TFJob controller"
time="2019-08-19T08:29:50Z" level=info msg="Creating Job controller"
time="2019-08-19T08:29:50Z" level=info msg="Starting TFJob controller"
time="2019-08-19T08:29:50Z" level=info msg="Waiting for informer caches to sync"
time="2019-08-19T08:29:50Z" level=info msg="Starting 1 workers"
time="2019-08-19T08:29:50Z" level=info msg="Started workers"
--- PASS: TestAddPod (0.10s)
=== RUN   TestClusterSpec
time="2019-08-19T08:29:50Z" level=info msg="Shutting down workers"
--- PASS: TestClusterSpec (0.00s)
=== RUN   TestRestartPolicy
--- PASS: TestRestartPolicy (0.00s)
=== RUN   TestExitCode
time="2019-08-19T08:29:50Z" level=info msg="Creating TFJob controller"
time="2019-08-19T08:29:50Z" level=info msg="Creating Job controller"
time="2019-08-19T08:29:50Z" level=info msg="Reconcile TFJobs test-tfjob" job=default.test-tfjob uid=
time="2019-08-19T08:29:50Z" level=info msg="Ignoring inactive pod default/worker-0 in state Failed, deletion time <nil>"
time="2019-08-19T08:29:50Z" level=info msg="Pod: default.worker-0 exited with code 130" job=default.test-tfjob replica-type=worker uid=
E0819 08:29:50.532832   10018 event.go:259] Could not construct reference to: '&v1beta2.TFJob{TypeMeta:v1.TypeMeta{Kind:"TFJob", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"test-tfjob", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, Spec:v1beta2.TFJobSpec{ActiveDeadlineSeconds:(*int64)(nil), BackoffLimit:(*int32)(nil), CleanPodPolicy:(*v1beta2.CleanPodPolicy)(0xc4204ed2f0), TTLSecondsAfterFinished:(*int32)(nil), TFReplicaSpecs:map[v1beta2.TFReplicaType]*v1beta2.ReplicaSpec{"Worker":(*v1beta2.ReplicaSpec)(0xc420a5fb80)}}, Status:v1beta2.JobStatus{Conditions:[]v1beta2.JobCondition(nil), ReplicaStatuses:map[v1beta2.ReplicaType]*v1beta2.ReplicaStatus{"Worker":(*v1beta2.ReplicaStatus)(0xc4204c0cc0)}, StartTime:(*v1.Time)(nil), CompletionTime:(*v1.Time)(nil), LastReconcileTime:(*v1.Time)(nil)}}' due to: 'selfLink was empty, can't make reference'. Will not report event: 'Normal' 'ExitedWithCode' 'Pod: default.worker-0 exited with code 130'
time="2019-08-19T08:29:50Z" level=info msg="Need to restart the pod: default.worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-08-19T08:29:50Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=1, running=0, failed=1" job=default.test-tfjob uid=
E0819 08:29:50.532957   10018 event.go:259] Could not construct reference to: '&v1beta2.TFJob{TypeMeta:v1.TypeMeta{Kind:"TFJob", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"test-tfjob", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, Spec:v1beta2.TFJobSpec{ActiveDeadlineSeconds:(*int64)(nil), BackoffLimit:(*int32)(nil), CleanPodPolicy:(*v1beta2.CleanPodPolicy)(0xc4204ed2f0), TTLSecondsAfterFinished:(*int32)(nil), TFReplicaSpecs:map[v1beta2.TFReplicaType]*v1beta2.ReplicaSpec{"Worker":(*v1beta2.ReplicaSpec)(0xc420a5fb80)}}, Status:v1beta2.JobStatus{Conditions:[]v1beta2.JobCondition(nil), ReplicaStatuses:map[v1beta2.ReplicaType]*v1beta2.ReplicaStatus{"Worker":(*v1beta2.ReplicaStatus)(0xc4204c0cc0)}, StartTime:(*v1.Time)(0xc42052aa40), CompletionTime:(*v1.Time)(nil), LastReconcileTime:(*v1.Time)(nil)}}' due to: 'selfLink was empty, can't make reference'. Will not report event: 'Warning' 'TFJobRestarting' 'TFJob test-tfjob is restarting because 1 Worker replica(s) failed.'
time="2019-08-19T08:29:50Z" level=info msg="need to create new service: worker-0" job=default.test-tfjob replica-type=worker uid=
time="2019-08-19T08:29:50Z" level=info msg="Finished syncing tfjob \"default/test-tfjob\" (505.365µs)" job=default.test-tfjob
--- PASS: TestExitCode (0.00s)
=== RUN   TestAddService
time="2019-08-19T08:29:50Z" level=info msg="Creating TFJob controller"
time="2019-08-19T08:29:50Z" level=info msg="Creating Job controller"
time="2019-08-19T08:29:50Z" level=info msg="Starting TFJob controller"
time="2019-08-19T08:29:50Z" level=info msg="Waiting for informer caches to sync"
time="2019-08-19T08:29:50Z" level=info msg="Starting TFJob controller"
time="2019-08-19T08:29:50Z" level=info msg="Waiting for informer caches to sync"
time="2019-08-19T08:29:50Z" level=info msg="Starting 1 workers"
time="2019-08-19T08:29:50Z" level=info msg="Started workers"
time="2019-08-19T08:29:50Z" level=info msg="Shutting down workers"
time="2019-08-19T08:29:50Z" level=info msg="Starting 1 workers"
time="2019-08-19T08:29:50Z" level=info msg="Started workers"
--- PASS: TestAddService (0.10s)
=== RUN   TestFailed
time="2019-08-19T08:29:50Z" level=info msg="Shutting down workers"
time="2019-08-19T08:29:50Z" level=info msg="Creating TFJob controller"
time="2019-08-19T08:29:50Z" level=info msg="Creating Job controller"
time="2019-08-19T08:29:50Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=3, running=0, failed=1" job=default.test-tfjob uid=
E0819 08:29:50.635760   10018 event.go:259] Could not construct reference to: '&v1beta2.TFJob{TypeMeta:v1.TypeMeta{Kind:"TFJob", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"test-tfjob", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, Spec:v1beta2.TFJobSpec{ActiveDeadlineSeconds:(*int64)(nil), BackoffLimit:(*int32)(nil), CleanPodPolicy:(*v1beta2.CleanPodPolicy)(0xc42085e1d0), TTLSecondsAfterFinished:(*int32)(nil), TFReplicaSpecs:map[v1beta2.TFReplicaType]*v1beta2.ReplicaSpec{"Worker":(*v1beta2.ReplicaSpec)(0xc420816840)}}, Status:v1beta2.JobStatus{Conditions:[]v1beta2.JobCondition(nil), ReplicaStatuses:map[v1beta2.ReplicaType]*v1beta2.ReplicaStatus{"Worker":(*v1beta2.ReplicaStatus)(0xc420858750)}, StartTime:(*v1.Time)(0xc420888060), CompletionTime:(*v1.Time)(nil), LastReconcileTime:(*v1.Time)(nil)}}' due to: 'selfLink was empty, can't make reference'. Will not report event: 'Normal' 'TFJobFailed' 'TFJob test-tfjob has failed because 1 Worker replica(s) failed.'
--- PASS: TestFailed (0.00s)
=== RUN   TestStatus
time="2019-08-19T08:29:50Z" level=info msg="Creating TFJob controller"
time="2019-08-19T08:29:50Z" level=info msg="Creating Job controller"
time="2019-08-19T08:29:50Z" level=info msg="TFJob=test-tfjob, ReplicaType=Chief expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-19T08:29:50Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-19T08:29:50Z" level=info msg="Creating TFJob controller"
time="2019-08-19T08:29:50Z" level=info msg="Creating Job controller"
time="2019-08-19T08:29:50Z" level=info msg="TFJob=test-tfjob, ReplicaType=Chief expected=1, running=1, failed=0" job=default.test-tfjob uid=
time="2019-08-19T08:29:50Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=1, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-19T08:29:50Z" level=info msg="Creating TFJob controller"
time="2019-08-19T08:29:50Z" level=info msg="Creating Job controller"
time="2019-08-19T08:29:50Z" level=info msg="TFJob=test-tfjob, ReplicaType=Chief expected=1, running=0, failed=1" job=default.test-tfjob uid=
time="2019-08-19T08:29:50Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=1, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-19T08:29:50Z" level=info msg="Creating TFJob controller"
time="2019-08-19T08:29:50Z" level=info msg="Creating Job controller"
time="2019-08-19T08:29:50Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=1, running=0, failed=1" job=default.test-tfjob uid=
time="2019-08-19T08:29:50Z" level=info msg="Creating TFJob controller"
time="2019-08-19T08:29:50Z" level=info msg="Creating Job controller"
time="2019-08-19T08:29:50Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-19T08:29:50Z" level=info msg="Creating TFJob controller"
time="2019-08-19T08:29:50Z" level=info msg="Creating Job controller"
time="2019-08-19T08:29:50Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=1, running=1, failed=0" job=default.test-tfjob uid=
time="2019-08-19T08:29:50Z" level=info msg="Creating TFJob controller"
time="2019-08-19T08:29:50Z" level=info msg="Creating Job controller"
time="2019-08-19T08:29:50Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-19T08:29:50Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-19T08:29:50Z" level=info msg="Creating TFJob controller"
time="2019-08-19T08:29:50Z" level=info msg="Creating Job controller"
time="2019-08-19T08:29:50Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=2, failed=2" job=default.test-tfjob uid=
time="2019-08-19T08:29:50Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-19T08:29:50Z" level=info msg="Creating TFJob controller"
time="2019-08-19T08:29:50Z" level=info msg="Creating Job controller"
time="2019-08-19T08:29:50Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=2, running=0, failed=2" job=default.test-tfjob uid=
time="2019-08-19T08:29:50Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-19T08:29:50Z" level=info msg="Creating TFJob controller"
time="2019-08-19T08:29:50Z" level=info msg="Creating Job controller"
time="2019-08-19T08:29:50Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=3, running=3, failed=0" job=default.test-tfjob uid=
time="2019-08-19T08:29:50Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-19T08:29:50Z" level=info msg="Creating TFJob controller"
time="2019-08-19T08:29:50Z" level=info msg="Creating Job controller"
time="2019-08-19T08:29:50Z" level=info msg="TFJob=test-tfjob, ReplicaType=Chief expected=1, running=1, failed=0" job=default.test-tfjob uid=
time="2019-08-19T08:29:50Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=0, failed=4" job=default.test-tfjob uid=
time="2019-08-19T08:29:50Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-19T08:29:50Z" level=info msg="Creating TFJob controller"
time="2019-08-19T08:29:50Z" level=info msg="Creating Job controller"
time="2019-08-19T08:29:50Z" level=info msg="TFJob=test-tfjob, ReplicaType=Chief expected=1, running=1, failed=0" job=default.test-tfjob uid=
time="2019-08-19T08:29:50Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-19T08:29:50Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-19T08:29:50Z" level=info msg="Creating TFJob controller"
time="2019-08-19T08:29:50Z" level=info msg="Creating Job controller"
time="2019-08-19T08:29:50Z" level=info msg="TFJob=test-tfjob, ReplicaType=Chief expected=1, running=1, failed=0" job=default.test-tfjob uid=
time="2019-08-19T08:29:50Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-19T08:29:50Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=1, failed=1" job=default.test-tfjob uid=
time="2019-08-19T08:29:50Z" level=info msg="Creating TFJob controller"
time="2019-08-19T08:29:50Z" level=info msg="Creating Job controller"
time="2019-08-19T08:29:50Z" level=info msg="TFJob=test-tfjob, ReplicaType=Chief expected=1, running=0, failed=1" job=default.test-tfjob uid=
time="2019-08-19T08:29:50Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-19T08:29:50Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-19T08:29:50Z" level=info msg="Creating TFJob controller"
time="2019-08-19T08:29:50Z" level=info msg="Creating Job controller"
time="2019-08-19T08:29:50Z" level=info msg="TFJob=test-tfjob, ReplicaType=Chief expected=0, running=0, failed=0" job=default.test-tfjob uid=
time="2019-08-19T08:29:50Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=0, failed=4" job=default.test-tfjob uid=
time="2019-08-19T08:29:50Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
time="2019-08-19T08:29:50Z" level=info msg="Creating TFJob controller"
time="2019-08-19T08:29:50Z" level=info msg="Creating Job controller"
time="2019-08-19T08:29:50Z" level=info msg="TFJob=test-tfjob, ReplicaType=Chief expected=1, running=0, failed=1" job=default.test-tfjob uid=
time="2019-08-19T08:29:50Z" level=info msg="TFJob=test-tfjob, ReplicaType=Worker expected=4, running=0, failed=4" job=default.test-tfjob uid=
time="2019-08-19T08:29:50Z" level=info msg="TFJob=test-tfjob, ReplicaType=PS expected=2, running=2, failed=0" job=default.test-tfjob uid=
--- PASS: TestStatus (0.02s)
=== RUN   TestGenOwnerReference
--- PASS: TestGenOwnerReference (0.00s)
=== RUN   TestGenLabels
--- PASS: TestGenLabels (0.00s)
=== RUN   TestConvertTFJobToUnstructured
--- PASS: TestConvertTFJobToUnstructured (0.00s)
PASS
coverage: 43.8% of statements in github.com/kubeflow/tf-operator/pkg/apis/common/v1, github.com/kubeflow/tf-operator/pkg/apis/common/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/controller.v1beta2/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
ok  	github.com/kubeflow/tf-operator/pkg/controller.v1beta2/tensorflow	4.930s	coverage: 43.8% of statements in github.com/kubeflow/tf-operator/pkg/apis/common/v1, github.com/kubeflow/tf-operator/pkg/apis/common/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/apis/tensorflow/validation, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/scheme, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1/fake, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/clientset/versioned/typed/tensorflow/v1beta2/fake, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/internalinterfaces, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/informers/externalversions/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1, github.com/kubeflow/tf-operator/pkg/client/listers/tensorflow/v1beta2, github.com/kubeflow/tf-operator/pkg/common/jobcontroller, github.com/kubeflow/tf-operator/pkg/common/util/v1/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1/unstructured, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/testutil, github.com/kubeflow/tf-operator/pkg/common/util/v1beta2/unstructured, github.com/kubeflow/tf-operator/pkg/control, github.com/kubeflow/tf-operator/pkg/controller.v1/tensorflow, github.com/kubeflow/tf-operator/pkg/controller.v1beta2/tensorflow, github.com/kubeflow/tf-operator/pkg/logger, github.com/kubeflow/tf-operator/pkg/util, github.com/kubeflow/tf-operator/pkg/util/k8sutil, github.com/kubeflow/tf-operator/pkg/util/signals, github.com/kubeflow/tf-operator/pkg/util/train, github.com/kubeflow/tf-operator/pkg/version
?   	github.com/kubeflow/tf-operator/pkg/logger	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/util	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/util/k8sutil	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/util/signals	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/util/train	[no test files]
?   	github.com/kubeflow/tf-operator/pkg/version	[no test files]
ignoring pkg/apis/common/v1/openapi_generated.go
ignoring pkg/apis/common/v1/zz_generated.deepcopy.go
ignoring pkg/apis/common/v1/zz_generated.defaults.go
ignoring pkg/apis/tensorflow/v1/openapi_generated.go
ignoring pkg/apis/tensorflow/v1/zz_generated.deepcopy.go
ignoring pkg/apis/tensorflow/v1/zz_generated.defaults.go
Job #2549.1
https://coveralls.io/jobs/52178458
TravisBuddy Request Identifier: 8562d4d0-c25b-11e9-86a4-77838e0d0b7a

@gaocegege
Copy link
Member

hack/../hack/update-codegen.sh: line 49: /home/travis/gopath/bin/openapi-gen: No such file or directory

Maybe the error is caused by the path hack/../hack/update-codegen.sh

@ohmystack
Copy link
Contributor Author

@gaocegege Thanks for helping debuging.
It turns out that vendor/k8s.io/code-generator/generate-groups.sh will install "defaulter-gen,client-gen,lister-gen,informer-gen,deepcopy-gen", leaving openapi-gen not installed because "generate-groups.sh" doesn't need it.
I add a command in .travis.yml to manually install it.

@gaocegege
Copy link
Member

/lgtm
/approve

Though this is a workaround, I do not have a better idea, too. Thus LGTM

Thanks for your contribution! 🎉 👍

@k8s-ci-robot
Copy link

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: gaocegege

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot merged commit bcd7e70 into kubeflow:master Aug 19, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants