Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Intermittent failure of etc/ci/manifest_changed.sh #11859

Closed
asajeffrey opened this issue Jun 24, 2016 · 2 comments
Closed

Intermittent failure of etc/ci/manifest_changed.sh #11859

asajeffrey opened this issue Jun 24, 2016 · 2 comments

Comments

@asajeffrey
Copy link
Member

@asajeffrey asajeffrey commented Jun 24, 2016

http://build.servo.org/builders/android/builds/1563

bash ./etc/ci/manifest_changed.sh
 in dir /home/servo/buildbot/slave/android/build (timeout 1200 secs)
 watching logfiles {}
 argv: ['bash', './etc/ci/manifest_changed.sh']
 environment:
  ANDROID_NDK=/home/servo/android/ndk/current/
  ANDROID_SDK=/home/servo/android/sdk/current/
  ANDROID_TOOLCHAIN=/home/servo/android/toolchain/current/
  CARGO_HOME=/home/servo/.cargo
  CCACHE=/usr/bin/ccache
  DISPLAY=:0
  HOME=/home/servo
  PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin:/home/servo/android/sdk/current/platform-tools:/home/servo/android/toolchain/current/bin
  PWD=/home/servo/buildbot/slave/android/build
  RUST_BACKTRACE=1
  SERVO_CACHE_DIR=/home/servo/.servo
  SHELL=/bin/bash
  TERM=linux
  UPSTART_INSTANCE=
  UPSTART_JOB=buildbot-slave
 using PTY: False
program finished with exit code 1
@jdm jdm added the I-intermittent label Jun 24, 2016
cbrewster added a commit to cbrewster/servo that referenced this issue Jun 27, 2016
cbrewster added a commit to cbrewster/servo that referenced this issue Jun 27, 2016
bors-servo added a commit that referenced this issue Jun 27, 2016
Investigating #11859 (do not merge!)

<!-- Please describe your changes on the following line: -->
This makes sure that manifest_changed.sh fails. Just checking to see if the diff is printed on the builders.

---
<!-- Thank you for contributing to Servo! Please replace each `[ ]` by `[X]` when the step is complete, and replace `__` with appropriate data: -->
- [ ] `./mach build -d` does not report any errors
- [ ] `./mach test-tidy` does not report any errors
- [ ] These changes fix #__ (github issue number if applicable).

<!-- Either: -->
- [ ] There are tests for these changes OR
- [ ] These changes do not require tests because _____

<!-- Pull requests that do not address these steps are welcome, but they will require additional verification as part of the review process. -->
@jdm
Copy link
Member

@jdm jdm commented Jun 27, 2016

Looks like this is actually a duplicate of #11858, based on my on-builder debugging:

servo@ip-172-31-34-111:~/buildbot/slave/arm32/build$ ./etc/ci/manifest_changed.sh ; echo $?
starting manifest update
Error running mach:

    ['test-wpt', '--manifest-update', '--binary=', 'SKIP_TESTS']

The error occurred in code that was called by the mach command. This is either
a bug in the called code itself or in the way that mach is calling it.

You should consider filing a bug for this issue.

If filing a bug, please include the full output of mach, including this error
message.

The details of the failure are as follows:

CalledProcessError: Command '[u'/usr/bin/openssl', 'ca', '-config', '/tmp/tmpoIirbA/openssl.cfg', '-batch', '-create_serial', '-keyfile', '/tmp/tmpoIirbA/cakey.pem', '-passin', 'pass:web-platform-tests', '-selfsign', '-extensions', 'v3_ca', '-in', '/tmp/tmpoIirbA/careq.pem', '-out', '/tmp/tmpoIirbA/cacert.pem']' returned non-zero exit status 1

  File "/home/servo/buildbot/slave/arm32/build/python/servo/testing_commands.py", line 328, in test_wpt
    return self.wptrunner(run_file, **kwargs)
  File "/home/servo/buildbot/slave/arm32/build/python/servo/testing_commands.py", line 348, in wptrunner
    return run_globals["run_tests"](**kwargs)
  File "/home/servo/buildbot/slave/arm32/build/tests/wpt/run_wpt.py", line 13, in run_tests
    return run.run_tests(paths=paths, **kwargs)
  File "/home/servo/buildbot/slave/arm32/build/tests/wpt/run.py", line 36, in run_tests
    success = wptrunner.run_tests(**kwargs)
  File "/home/servo/buildbot/slave/arm32/build/tests/wpt/harness/wptrunner/wptrunner.py", line 155, in run_tests
    env_options) as test_environment:
  File "/home/servo/buildbot/slave/arm32/build/tests/wpt/harness/wptrunner/environment.py", line 104, in __enter__
    self.config = self.load_config()
  File "/home/servo/buildbot/slave/arm32/build/tests/wpt/harness/wptrunner/environment.py", line 152, in load_config
    key_file, certificate = self.ssl_env.host_cert_path(hosts)
  File "/home/servo/buildbot/slave/arm32/build/tests/wpt/web-platform-tests/tools/sslutils/openssl.py", line 351, in host_cert_path
    key, cert = self._generate_host_cert(hosts)
  File "/home/servo/buildbot/slave/arm32/build/tests/wpt/web-platform-tests/tools/sslutils/openssl.py", line 371, in _generate_host_cert
    self._generate_ca()
  File "/home/servo/buildbot/slave/arm32/build/tests/wpt/web-platform-tests/tools/sslutils/openssl.py", line 332, in _generate_ca
    "-out", cert_path)
  File "/home/servo/buildbot/slave/arm32/build/tests/wpt/web-platform-tests/tools/sslutils/openssl.py", line 70, in __call__
    output=stdout)
1
@asajeffrey
Copy link
Member Author

@asajeffrey asajeffrey commented Jun 27, 2016

@jdm: we did start seeing them both at the same time, so it wouldn't be too surprising if it's the same root cause.

bors-servo added a commit that referenced this issue Jun 27, 2016
Fix common OpenSSL intermittent failure on automated builders

OpenSSL requires that the digits of the serial number for the generated certs contain an even number of digits. The recent change in sslutils (w3c/wpt-tools@b388d6b) does not fulfill this requirement. I'll be making the same PR upstream to address that, but this will get us out of the woods locally.

---
<!-- Thank you for contributing to Servo! Please replace each `[ ]` by `[X]` when the step is complete, and replace `__` with appropriate data: -->
- [X] `./mach build -d` does not report any errors
- [X] `./mach test-tidy` does not report any errors
- [X] These changes fix #11858 and fix #11859
- [X] These changes do not require tests because there's no automated testing for the SSL cert generation

<!-- Reviewable:start -->
---
This change is [<img src="https://reviewable.io/review_button.svg" height="35" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/servo/servo/11885)
<!-- Reviewable:end -->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

2 participants
You can’t perform that action at this time.