Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Timeouts in crypto tests on Raspberry PIs #42379

Closed
mhdawson opened this issue Mar 17, 2022 · 4 comments
Closed

Timeouts in crypto tests on Raspberry PIs #42379

mhdawson opened this issue Mar 17, 2022 · 4 comments
Labels
flaky-test Issues and PRs related to the tests with unstable failures on the CI.

Comments

@mhdawson
Copy link
Member

Test

parallel/test-crypto-binary-default paralllel/test-crypto-dh-odd-key (3) parallel/test-crypto-binary-default (2) parallel/test-crypto-dh (2) parallel/test-crypto-dh-constructor (2) parallel/test-crypto-dh-leak (2)

Platform

ARM

Console output

22:20:01   ---
22:20:01   duration_ms: 240.146
22:20:01   severity: fail
22:20:01   exitcode: -15
22:20:01   stack: |-
22:20:01     timeout
22:20:01   ...

Build links

Additional information

These have been flaky for quite a long time

I can see failures in parallel/test-crypto-dh all the way back to October 14 2021 in the reliability reportl - nodejs/reliability#86
@mhdawson mhdawson added the flaky-test Issues and PRs related to the tests with unstable failures on the CI. label Mar 17, 2022
@mhdawson
Copy link
Member Author

The update to OpenSSL 3 was around that time

commit 66da32c045035cf2710a48773dc6f55f00e20c40
Author: Daniel Bevenius <daniel.bevenius@gmail.com>
Date:   Wed Apr 14 11:19:54 2021 +0200

    deps,test,src,doc,tools: update to OpenSSL 3.0

    This pull request updates the OpenSSL version that is statically
    linked with Node.js from OpenSSl 1.1.1 to quictls OpenSSL 3.0.0+quic.

    This pull request will replace the OpenSSL version that is currently
    in the deps directory and when performing a normal build
    OpenSSL 3.0+quic will be statically linked to the Node.js executable.
    We will still be able to dynamically link to OpenSSL 1.1.1 and we have
    a CI job which dynamically links to OpenSSL 1.1.1 which is run for
    every pull request to make sure that we maintain backward compatibility.

    PR-URL: https://github.com/nodejs/node/pull/38512
    Reviewed-By: Michael Dawson <midawson@redhat.com>
    Reviewed-By: Tobias Nießen <tniessen@tnie.de>

and it actually landed on the 11th.

My guess is that the update made things a bit slower which is why we are seeing the intermittent timeouts on the PIs

@mhdawson mhdawson mentioned this issue Mar 17, 2022
Closed
mhdawson added a commit to mhdawson/io.js that referenced this issue Mar 17, 2022
@mhdawson
test: give slow tests more time on Rasberry PIs
646437c 
Refs: nodejs#42379

A number of tests have been failing internmittently
on the Rasberry PIs with timeouts since the update
to OpenSSL 3.0. Give them more time.

Signed-off-by: Michael Dawson <mdawson@devrus.com>
@Trott
Copy link
Member

Trott commented Mar 17, 2022

My guess is that the update made things a bit slower

My guess is that a bug was introduced or at least an OS bug or limitation is now being exposed.

One thing that could help would be to modify the tests to either break them up into many smaller test files and/or to add some console.log() statements so that when the test times out, we have some idea how far along it is and how long it was stuck in that state. Right now, all we know is that something locks up somewhere in these (often very long) test files but we have no idea where.

mhdawson added a commit that referenced this issue Mar 18, 2022
@mhdawson
test: give slow tests more time on Rasberry PIs
6374f6b 
Refs: #42379

A number of tests have been failing internmittently
on the Rasberry PIs with timeouts since the update
to OpenSSL 3.0. Give them more time.

Signed-off-by: Michael Dawson <mdawson@devrus.com>

PR-URL: #42380
Reviewed-By: Rich Trott <rtrott@gmail.com>
Reviewed-By: Tobias Nießen <tniessen@tnie.de>
Reviewed-By: Luigi Pinca <luigipinca@gmail.com>
bengl pushed a commit that referenced this issue Mar 21, 2022
@mhdawson @bengl
test: give slow tests more time on Rasberry PIs
d9a5c2b 
Refs: #42379

A number of tests have been failing internmittently
on the Rasberry PIs with timeouts since the update
to OpenSSL 3.0. Give them more time.

Signed-off-by: Michael Dawson <mdawson@devrus.com>

PR-URL: #42380
Reviewed-By: Rich Trott <rtrott@gmail.com>
Reviewed-By: Tobias Nießen <tniessen@tnie.de>
Reviewed-By: Luigi Pinca <luigipinca@gmail.com>
danielleadams pushed a commit to danielleadams/node that referenced this issue Apr 21, 2022
@mhdawson @danielleadams
test: give slow tests more time on Rasberry PIs
bd71773 
Refs: nodejs#42379

A number of tests have been failing internmittently
on the Rasberry PIs with timeouts since the update
to OpenSSL 3.0. Give them more time.

Signed-off-by: Michael Dawson <mdawson@devrus.com>

PR-URL: nodejs#42380
Reviewed-By: Rich Trott <rtrott@gmail.com>
Reviewed-By: Tobias Nießen <tniessen@tnie.de>
Reviewed-By: Luigi Pinca <luigipinca@gmail.com>
danielleadams pushed a commit that referenced this issue Apr 24, 2022
@mhdawson @danielleadams
test: give slow tests more time on Rasberry PIs
44bd1b2 
Refs: #42379

A number of tests have been failing internmittently
on the Rasberry PIs with timeouts since the update
to OpenSSL 3.0. Give them more time.

Signed-off-by: Michael Dawson <mdawson@devrus.com>

PR-URL: #42380
Reviewed-By: Rich Trott <rtrott@gmail.com>
Reviewed-By: Tobias Nießen <tniessen@tnie.de>
Reviewed-By: Luigi Pinca <luigipinca@gmail.com>
danielleadams pushed a commit that referenced this issue Apr 24, 2022
@mhdawson @danielleadams
test: give slow tests more time on Rasberry PIs
3636b2a 
Refs: #42379

A number of tests have been failing internmittently
on the Rasberry PIs with timeouts since the update
to OpenSSL 3.0. Give them more time.

Signed-off-by: Michael Dawson <mdawson@devrus.com>

PR-URL: #42380
Reviewed-By: Rich Trott <rtrott@gmail.com>
Reviewed-By: Tobias Nießen <tniessen@tnie.de>
Reviewed-By: Luigi Pinca <luigipinca@gmail.com>
danielleadams pushed a commit that referenced this issue Apr 24, 2022
@mhdawson @danielleadams
test: give slow tests more time on Rasberry PIs
e3bced1 
Refs: #42379

A number of tests have been failing internmittently
on the Rasberry PIs with timeouts since the update
to OpenSSL 3.0. Give them more time.

Signed-off-by: Michael Dawson <mdawson@devrus.com>

PR-URL: #42380
Reviewed-By: Rich Trott <rtrott@gmail.com>
Reviewed-By: Tobias Nießen <tniessen@tnie.de>
Reviewed-By: Luigi Pinca <luigipinca@gmail.com>
xtx1130 pushed a commit to xtx1130/node that referenced this issue Apr 25, 2022
@mhdawson @xtx1130
test: give slow tests more time on Rasberry PIs
eab299e 
Refs: nodejs#42379

A number of tests have been failing internmittently
on the Rasberry PIs with timeouts since the update
to OpenSSL 3.0. Give them more time.

Signed-off-by: Michael Dawson <mdawson@devrus.com>

PR-URL: nodejs#42380
Reviewed-By: Rich Trott <rtrott@gmail.com>
Reviewed-By: Tobias Nießen <tniessen@tnie.de>
Reviewed-By: Luigi Pinca <luigipinca@gmail.com>
@richardlau
Copy link
Member

Since we no longer have Raspberry Pi's in our CI cluster, should we close this?

@mhdawson
Copy link
Member Author

agreed, closing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
flaky-test Issues and PRs related to the tests with unstable failures on the CI.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@Trott @richardlau @mhdawson