Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Flaky test] ci-kubernetes-unit #107708

Closed
Nivedita-coder opened this issue Jan 24, 2022 · 9 comments
Closed

[Flaky test] ci-kubernetes-unit #107708

Nivedita-coder opened this issue Jan 24, 2022 · 9 comments
Assignees
@aojea
Labels
kind/flake Categorizes issue or PR as related to a flaky test. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. triage/accepted Indicates an issue or PR is ready to be actively worked on.

Comments

@Nivedita-coder
Copy link

Nivedita-coder commented Jan 24, 2022 •
edited
Loading

Which jobs are flaking?

  • ci-kubernetes-unit

Which tests are flaking?

  • ci-kubernetes-unit.Overall

  • k8s.io/kubernetes/pkg/kubelet/eviction.TestStart

  • k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.TestTimeoutHeaders

  • k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/httpstream/spdy. TestRoundTripSocks5AndNewConnection

  • k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/httpstream/spdy. TestRoundTripSocks5AndNewConnection/redirect_=_true

  • k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/httpstream/spdy. TestRoundTripSocks5AndNewConnection/redirect_=true/proxied_with_valid_auth->https(valid_hostname_+_RootCAs)

  • k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/httpstream/spdy. TestRoundTripSocks5AndNewConnection/redirect_=true/proxied_with_valid_auth->_http

  • k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/httpstream/spdy. TestRoundTripSocks5AndNewConnection/redirect_=_false

  • k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/httpstream/spdy. TestRoundTripSocks5AndNewConnection/redirect_=false/proxied_with_valid_auth->https(invalid_hostname_+_InsecureSkipVerify)

Since when has it been flaking?

History goes back to 2021-12-30

Testgrid link

https://testgrid.k8s.io/sig-release-master-blocking#ci-kubernetes-unit

Reason for failure (if possible)

No response

Anything else we need to know?

No response

Relevant SIG(s)

/sig testing (I guess)
cc @kubernetes/ci-signal
@Nivedita-coder Nivedita-coder added the kind/flake Categorizes issue or PR as related to a flaky test. label Jan 24, 2022
@k8s-ci-robot k8s-ci-robot added the sig/testing Categorizes an issue or PR as relevant to SIG Testing. label Jan 24, 2022
@k8s-ci-robot
Copy link
Contributor

@Nivedita-coder: The label(s) sig/(i, sig/guess) cannot be applied, because the repository doesn't have them.

In response to this:

Which jobs are flaking?

  • ci-kubernetes-unit

Which tests are flaking?

  • ci-kubernetes-unit.Overall

  • k8s.io/kubernetes/pkg/kubelet/eviction.TestStart

  • k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/filters.TestTimeoutHeaders

  • k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/httpstream/spdy. TestRoundTripSocks5AndNewConnection

  • k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/httpstream/spdy. TestRoundTripSocks5AndNewConnection/redirect_=_true

  • k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/httpstream/spdy: TestRoundTripSocks5AndNewConnection/redirect_=true/proxied_with_valid_auth->https(valid_hostname_+_RootCAs)

  • k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/httpstream/spdy: TestRoundTripSocks5AndNewConnection/redirect_=true/proxied_with_valid_auth->_http

  • k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/httpstream/spdy: TestRoundTripSocks5AndNewConnection/redirect_=_false

  • k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/httpstream/spdy: TestRoundTripSocks5AndNewConnection/redirect_=false/proxied_with_valid_auth->https(invalid_hostname_+_InsecureSkipVerify)

Since when has it been flaking?

History goes back to 2021-12-30

Testgrid link

https://testgrid.k8s.io/sig-release-master-blocking#ci-kubernetes-unit

Reason for failure (if possible)

No response

Anything else we need to know?

No response

Relevant SIG(s)

/sig testing (I guess)
cc @kubernetes/ci-signal

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@k8s-ci-robot k8s-ci-robot added the needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. label Jan 24, 2022
@Nivedita-coder Nivedita-coder changed the title ci-kubernetes-unit [Flaky test] ci-kubernetes-unit Jan 24, 2022
@leonardpahlke
Copy link
Member

/remove-sig testing

/sig api-machinery

@k8s-ci-robot k8s-ci-robot added sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. and removed sig/testing Categorizes an issue or PR as relevant to SIG Testing. labels Jan 24, 2022
@aojea
Copy link
Member

aojea commented Jan 24, 2022

errors related to connection failures, snippet from 2 different jobs

=== FAIL: vendor/k8s.io/apimachinery/pkg/util/httpstream/spdy TestRoundTripSocks5AndNewConnection/redirect_=false/proxied_with_valid_auth->http (0.07s)
2022/01/24 05:09:20 [ERR] socks: Failed to handle request: readfrom tcp 127.0.0.1:40351->127.0.0.1:33160: splice: connection reset by peer
roundtripper_test.go:590: ServeConn error: Failed to handle request: readfrom tcp 127.0.0.1:40351->127.0.0.1:33160: splice: connection reset by peer
--- FAIL: TestRoundTripSocks5AndNewConnection/redirect=false/proxied_with_valid_auth->_http (0.07s)

and

=== Failed
=== FAIL: vendor/k8s.io/apimachinery/pkg/util/httpstream/spdy TestRoundTripSocks5AndNewConnection/redirect_=false/proxied_with_valid_auth->https(invalid_hostname_+InsecureSkipVerify) (0.10s)
2022/01/23 17:04:15 [ERR] socks: Failed to handle request: readfrom tcp 127.0.0.1:32846->127.0.0.1:45495: splice: broken pipe
roundtripper_test.go:590: ServeConn error: Failed to handle request: readfrom tcp 127.0.0.1:32846->127.0.0.1:45495: splice: broken pipe
--- FAIL: TestRoundTripSocks5AndNewConnection/redirect=false/proxied_with_valid_auth->https(invalid_hostname_+_InsecureSkipVerify) (0.10s)

@aojea
Copy link
Member

aojea commented Jan 24, 2022

started to flake recently based on

https://storage.googleapis.com/k8s-triage/index.html?ci=0&pr=1&test=TestRoundTripSocks5AndNewConnection

The test TestRoundTripSocks5AndNewConnection was added in

0a98875

similar error on the socks5 dependency

armon/go-socks5#34

The error seems to come from

https://github.com/armon/go-socks5/blob/e75332964ef517daa070d7c38a9466a0d687e0a5/socks5.go#L161-L166

I'll keep investigating once I have more time, my main suspicion now is a race in the go-socks5 module similar to the one fixed here golang/go#40888

/cc @liggitt

@aojea aojea mentioned this issue Jan 24, 2022
Closed
@aojea
Copy link
Member

aojea commented Jan 24, 2022

/assign

@liggitt liggitt mentioned this issue Jan 24, 2022
Merged
@liggitt
Copy link
Member

liggitt commented Jan 24, 2022

cc @xens

@liggitt liggitt added the priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. label Jan 24, 2022
@alexzielenski alexzielenski mentioned this issue Jan 24, 2022
Merged
@alculquicondor alculquicondor mentioned this issue Jan 25, 2022
Merged
@aojea aojea mentioned this issue Jan 25, 2022
Merged
@fedebongio
Copy link
Contributor

/triage accepted

@k8s-ci-robot k8s-ci-robot added triage/accepted Indicates an issue or PR is ready to be actively worked on. and removed needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Jan 25, 2022
@aojea aojea mentioned this issue Jan 25, 2022
Closed
@aojea
Copy link
Member

aojea commented Jan 28, 2022

this has been closed with #107758

The test fix will be tracked in #107766
/close

@k8s-ci-robot
Copy link
Contributor

@aojea: Closing this issue.

In response to this:

this has been closed with #107758

The test fix will be tracked in #107766
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@endocrimes endocrimes mentioned this issue Feb 8, 2022
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@aojea aojea

Labels
kind/flake Categorizes issue or PR as related to a flaky test. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. triage/accepted Indicates an issue or PR is ready to be actively worked on.
Projects
CI Signal (SIG Release / Release Team)
Status: RESOLVED
Milestone
No milestone
Development

No branches or pull requests
6 participants
@liggitt @fedebongio @aojea @k8s-ci-robot @Nivedita-coder @leonardpahlke