Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Chrome builds interrupted by module compilation failure #649

Open
jugglinmike opened this issue Feb 14, 2019 · 12 comments

Comments

@jugglinmike
Copy link
Collaborator

commented Feb 14, 2019

Yesterday, 2 Chrome stable builds and 1 Chrome experimental builds failed as the WPT CLI installed Python dependencies. Specifically, the workers were unable to compile the native extensions provided by the cffi module (error log included below).

I haven't been able to find a conclusive explanation for this failure, but I suspect it was a transitory packaging issue. Here's why:

  • all three builds failed within a small time window (less than 90 seconds)
  • a new version of cffi was released at approximately that time](https://groups.google.com/forum/#!topic/python-cffi/AuYExAf8LSY)
  • no changes were made to this project at that time
  • no subsequent build has exhibited this problem (over 350 in total)

I'll keep my eyes peeled for future occurrences, but I'm hopeful that it was a passing glitch. I've manually re-triggered the failed builds; the corresponding data should be available on wpt.fyi shortly.

Partial error log
2019-02-13 13:39:57,247 INFO validate-wpt-results     python ./wpt run --log-raw /tmp/tmpPTpDNe/log-raw.txt --log-wptreport /tmp/tmpPTpDNe/report.json --log-mach - --this-chunk 8 --total-chunks 20 --binary /usr/bin/google-chrome --webdriver-binary /home/jade/worker/GNU_Linux_Chunked_Runner/build/chromedriver --install-fonts --no-headless --binary-arg=--use-fake-ui-for-media-stream --binary-arg=--use-fake-device-for-media-stream --binary-arg=--enable-experimental-web-platform-features chrome
2019-02-13 13:40:00,123 INFO validate-wpt-results wpt-run:stderr DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.
2019-02-13 13:40:01,010 INFO validate-wpt-results wpt-run:stderr DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.
2019-02-13 13:40:03,706 INFO validate-wpt-results wpt-run:stderr   Failed building wheel for cffi
2019-02-13 13:40:04,626 INFO validate-wpt-results wpt-run:stderr Command "/home/jade/worker/GNU_Linux_Chunked_Runner/build/_venv/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-FhQJPN/cffi/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-record-LhHMGn/install-record.txt --single-version-externally-managed --compile --install-headers /home/jade/worker/GNU_Linux_Chunked_Runner/build/_venv/include/site/python2.7/cffi" failed with error code 1 in /tmp/pip-install-FhQJPN/cffi/
2019-02-13 13:40:04,676 INFO validate-wpt-results wpt-run:stderr CRITICAL:tools.wpt.utils:('/home/jade/worker/GNU_Linux_Chunked_Runner/build/_venv/bin/pip', 'install', '-r', u'/home/jade/worker/GNU_Linux_Chunked_Runner/build/tools/wpt/../wptrunner/requirements.txt') exited with return code 1
2019-02-13 13:40:04,677 INFO validate-wpt-results wpt-run:stderr CRITICAL:tools.wpt.utils:Collecting html5lib==1.0.1 (from -r /home/jade/worker/GNU_Linux_Chunked_Runner/build/tools/wpt/../wptrunner/requirements.txt (line 1))
2019-02-13 13:40:04,677 INFO validate-wpt-results wpt-run:stderr   Using cached https://files.pythonhosted.org/packages/a5/62/bbd2be0e7943ec8504b517e62bab011b4946e1258842bc159e5dfde15b96/html5lib-1.0.1-py2.py3-none-any.whl
2019-02-13 13:40:04,677 INFO validate-wpt-results wpt-run:stderr Collecting mozinfo==0.10 (from -r /home/jade/worker/GNU_Linux_Chunked_Runner/build/tools/wpt/../wptrunner/requirements.txt (line 2))
2019-02-13 13:40:04,677 INFO validate-wpt-results wpt-run:stderr Collecting mozlog==4.0 (from -r /home/jade/worker/GNU_Linux_Chunked_Runner/build/tools/wpt/../wptrunner/requirements.txt (line 3))
2019-02-13 13:40:04,677 INFO validate-wpt-results wpt-run:stderr   Using cached https://files.pythonhosted.org/packages/18/7b/37831f9ae29ea950dd2dd332ae49a684ffda3aaf30a72c7e6b275b080315/mozlog-4.0-py2.py3-none-any.whl
2019-02-13 13:40:04,677 INFO validate-wpt-results wpt-run:stderr Collecting mozdebug==0.1.1 (from -r /home/jade/worker/GNU_Linux_Chunked_Runner/build/tools/wpt/../wptrunner/requirements.txt (line 4))
2019-02-13 13:40:04,677 INFO validate-wpt-results wpt-run:stderr   Using cached https://files.pythonhosted.org/packages/cb/fb/cd673c7e9f7d4911d93c7aacd4d464ec8465d213c1ca63fcd4f59cc24ef7/mozdebug-0.1.1-py2.py3-none-any.whl
2019-02-13 13:40:04,677 INFO validate-wpt-results wpt-run:stderr Requirement already satisfied: urllib3[secure]==1.24.1 in ./_venv/lib/python2.7/site-packages (from -r /home/jade/worker/GNU_Linux_Chunked_Runner/build/tools/wpt/../wptrunner/requirements.txt (line 5)) (1.24.1)
2019-02-13 13:40:04,678 INFO validate-wpt-results wpt-run:stderr Collecting six>=1.9 (from html5lib==1.0.1->-r /home/jade/worker/GNU_Linux_Chunked_Runner/build/tools/wpt/../wptrunner/requirements.txt (line 1))
2019-02-13 13:40:04,678 INFO validate-wpt-results wpt-run:stderr   Using cached https://files.pythonhosted.org/packages/73/fb/00a976f728d0d1fecfe898238ce23f502a721c0ac0ecfedb80e0d88c64e9/six-1.12.0-py2.py3-none-any.whl
2019-02-13 13:40:04,678 INFO validate-wpt-results wpt-run:stderr Collecting webencodings (from html5lib==1.0.1->-r /home/jade/worker/GNU_Linux_Chunked_Runner/build/tools/wpt/../wptrunner/requirements.txt (line 1))
2019-02-13 13:40:04,678 INFO validate-wpt-results wpt-run:stderr   Using cached https://files.pythonhosted.org/packages/f4/24/2a3e3df732393fed8b3ebf2ec078f05546de641fe1b667ee316ec1dcf3b7/webencodings-0.5.1-py2.py3-none-any.whl
2019-02-13 13:40:04,678 INFO validate-wpt-results wpt-run:stderr Collecting mozfile>=0.12 (from mozinfo==0.10->-r /home/jade/worker/GNU_Linux_Chunked_Runner/build/tools/wpt/../wptrunner/requirements.txt (line 2))
2019-02-13 13:40:04,678 INFO validate-wpt-results wpt-run:stderr   Using cached https://files.pythonhosted.org/packages/85/ed/4f8bec5ac04373ef0c79428bdd50ce039e91e4b53660d08e9a45988b74dc/mozfile-2.0.0-py2.py3-none-any.whl
2019-02-13 13:40:04,678 INFO validate-wpt-results wpt-run:stderr Collecting mozterm (from mozlog==4.0->-r /home/jade/worker/GNU_Linux_Chunked_Runner/build/tools/wpt/../wptrunner/requirements.txt (line 3))
2019-02-13 13:40:04,678 INFO validate-wpt-results wpt-run:stderr   Using cached https://files.pythonhosted.org/packages/ab/eb/0c53684f5821d666372c6ea03d8c64558c8f74fda0fd5b24ae3dd2ab6a95/mozterm-1.0.0-py2.py3-none-any.whl
2019-02-13 13:40:04,679 INFO validate-wpt-results wpt-run:stderr Collecting blessings>=1.3 (from mozlog==4.0->-r /home/jade/worker/GNU_Linux_Chunked_Runner/build/tools/wpt/../wptrunner/requirements.txt (line 3))
2019-02-13 13:40:04,679 INFO validate-wpt-results wpt-run:stderr   Using cached https://files.pythonhosted.org/packages/8d/b1/a3fe6fd8a012e6d019bafd671c2fee0597ea97ff2e76c25aadfa4545fc32/blessings-1.7-py2-none-any.whl
2019-02-13 13:40:04,679 INFO validate-wpt-results wpt-run:stderr Collecting pyOpenSSL>=0.14; extra == "secure" (from urllib3[secure]==1.24.1->-r /home/jade/worker/GNU_Linux_Chunked_Runner/build/tools/wpt/../wptrunner/requirements.txt (line 5))
2019-02-13 13:40:04,679 INFO validate-wpt-results wpt-run:stderr   Using cached https://files.pythonhosted.org/packages/01/c8/ceb170d81bd3941cbeb9940fc6cc2ef2ca4288d0ca8929ea4db5905d904d/pyOpenSSL-19.0.0-py2.py3-none-any.whl
2019-02-13 13:40:04,679 INFO validate-wpt-results wpt-run:stderr Requirement already satisfied: idna>=2.0.0; extra == "secure" in ./_venv/lib/python2.7/site-packages (from urllib3[secure]==1.24.1->-r /home/jade/worker/GNU_Linux_Chunked_Runner/build/tools/wpt/../wptrunner/requirements.txt (line 5)) (2.8)
2019-02-13 13:40:04,679 INFO validate-wpt-results wpt-run:stderr Collecting cryptography>=1.3.4; extra == "secure" (from urllib3[secure]==1.24.1->-r /home/jade/worker/GNU_Linux_Chunked_Runner/build/tools/wpt/../wptrunner/requirements.txt (line 5))
2019-02-13 13:40:04,679 INFO validate-wpt-results wpt-run:stderr   Using cached https://files.pythonhosted.org/packages/17/fd/4c2c8953a9dfe38fbe0c3adafb6355540bd98cef70cc82734acb0a4c0e2f/cryptography-2.5-cp27-cp27mu-manylinux1_x86_64.whl
2019-02-13 13:40:04,679 INFO validate-wpt-results wpt-run:stderr Requirement already satisfied: certifi; extra == "secure" in ./_venv/lib/python2.7/site-packages (from urllib3[secure]==1.24.1->-r /home/jade/worker/GNU_Linux_Chunked_Runner/build/tools/wpt/../wptrunner/requirements.txt (line 5)) (2018.11.29)
2019-02-13 13:40:04,680 INFO validate-wpt-results wpt-run:stderr Collecting ipaddress; extra == "secure" (from urllib3[secure]==1.24.1->-r /home/jade/worker/GNU_Linux_Chunked_Runner/build/tools/wpt/../wptrunner/requirements.txt (line 5))
2019-02-13 13:40:04,680 INFO validate-wpt-results wpt-run:stderr   Using cached https://files.pythonhosted.org/packages/fc/d0/7fc3a811e011d4b388be48a0e381db8d990042df54aa4ef4599a31d39853/ipaddress-1.0.22-py2.py3-none-any.whl
2019-02-13 13:40:04,680 INFO validate-wpt-results wpt-run:stderr Collecting asn1crypto>=0.21.0 (from cryptography>=1.3.4; extra == "secure"->urllib3[secure]==1.24.1->-r /home/jade/worker/GNU_Linux_Chunked_Runner/build/tools/wpt/../wptrunner/requirements.txt (line 5))
2019-02-13 13:40:04,680 INFO validate-wpt-results wpt-run:stderr   Using cached https://files.pythonhosted.org/packages/ea/cd/35485615f45f30a510576f1a56d1e0a7ad7bd8ab5ed7cdc600ef7cd06222/asn1crypto-0.24.0-py2.py3-none-any.whl
2019-02-13 13:40:04,680 INFO validate-wpt-results wpt-run:stderr Collecting enum34; python_version < "3" (from cryptography>=1.3.4; extra == "secure"->urllib3[secure]==1.24.1->-r /home/jade/worker/GNU_Linux_Chunked_Runner/build/tools/wpt/../wptrunner/requirements.txt (line 5))
2019-02-13 13:40:04,683 INFO validate-wpt-results wpt-run:stderr   Using cached https://files.pythonhosted.org/packages/c5/db/e56e6b4bbac7c4a06de1c50de6fe1ef3810018ae11732a50f15f62c7d050/enum34-1.1.6-py2-none-any.whl
2019-02-13 13:40:04,684 INFO validate-wpt-results wpt-run:stderr Collecting cffi!=1.11.3,>=1.8 (from cryptography>=1.3.4; extra == "secure"->urllib3[secure]==1.24.1->-r /home/jade/worker/GNU_Linux_Chunked_Runner/build/tools/wpt/../wptrunner/requirements.txt (line 5))
2019-02-13 13:40:04,684 INFO validate-wpt-results wpt-run:stderr   Downloading https://files.pythonhosted.org/packages/10/fe/b6362c613a70ac29cf7cac36307d85f08ebe4a96d9d54b895b10a807e39b/cffi-1.12.0.tar.gz (453kB)
2019-02-13 13:40:04,684 INFO validate-wpt-results wpt-run:stderr Collecting pycparser (from cffi!=1.11.3,>=1.8->cryptography>=1.3.4; extra == "secure"->urllib3[secure]==1.24.1->-r /home/jade/worker/GNU_Linux_Chunked_Runner/build/tools/wpt/../wptrunner/requirements.txt (line 5))
2019-02-13 13:40:04,684 INFO validate-wpt-results wpt-run:stderr Building wheels for collected packages: cffi
2019-02-13 13:40:04,684 INFO validate-wpt-results wpt-run:stderr   Building wheel for cffi (setup.py): started
2019-02-13 13:40:04,685 INFO validate-wpt-results wpt-run:stderr   Building wheel for cffi (setup.py): finished with status 'error'
2019-02-13 13:40:04,685 INFO validate-wpt-results wpt-run:stderr   Complete output from command /home/jade/worker/GNU_Linux_Chunked_Runner/build/_venv/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-FhQJPN/cffi/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/pip-wheel-9U0S4d --python-tag cp27:
2019-02-13 13:40:04,685 INFO validate-wpt-results wpt-run:stderr   running bdist_wheel
2019-02-13 13:40:04,685 INFO validate-wpt-results wpt-run:stderr   running build
2019-02-13 13:40:04,685 INFO validate-wpt-results wpt-run:stderr   running build_py
2019-02-13 13:40:04,685 INFO validate-wpt-results wpt-run:stderr   creating build
2019-02-13 13:40:04,685 INFO validate-wpt-results wpt-run:stderr   creating build/lib.linux-x86_64-2.7
2019-02-13 13:40:04,686 INFO validate-wpt-results wpt-run:stderr   creating build/lib.linux-x86_64-2.7/cffi
2019-02-13 13:40:04,686 INFO validate-wpt-results wpt-run:stderr   copying cffi/ffiplatform.py -> build/lib.linux-x86_64-2.7/cffi
2019-02-13 13:40:04,686 INFO validate-wpt-results wpt-run:stderr   copying cffi/error.py -> build/lib.linux-x86_64-2.7/cffi
2019-02-13 13:40:04,686 INFO validate-wpt-results wpt-run:stderr   copying cffi/lock.py -> build/lib.linux-x86_64-2.7/cffi
2019-02-13 13:40:04,686 INFO validate-wpt-results wpt-run:stderr   copying cffi/__init__.py -> build/lib.linux-x86_64-2.7/cffi
2019-02-13 13:40:04,686 INFO validate-wpt-results wpt-run:stderr   copying cffi/commontypes.py -> build/lib.linux-x86_64-2.7/cffi
2019-02-13 13:40:04,686 INFO validate-wpt-results wpt-run:stderr   copying cffi/setuptools_ext.py -> build/lib.linux-x86_64-2.7/cffi
2019-02-13 13:40:04,686 INFO validate-wpt-results wpt-run:stderr   copying cffi/vengine_gen.py -> build/lib.linux-x86_64-2.7/cffi
2019-02-13 13:40:04,687 INFO validate-wpt-results wpt-run:stderr   copying cffi/verifier.py -> build/lib.linux-x86_64-2.7/cffi
2019-02-13 13:40:04,687 INFO validate-wpt-results wpt-run:stderr   copying cffi/vengine_cpy.py -> build/lib.linux-x86_64-2.7/cffi
2019-02-13 13:40:04,687 INFO validate-wpt-results wpt-run:stderr   copying cffi/backend_ctypes.py -> build/lib.linux-x86_64-2.7/cffi
2019-02-13 13:40:04,687 INFO validate-wpt-results wpt-run:stderr   copying cffi/recompiler.py -> build/lib.linux-x86_64-2.7/cffi
2019-02-13 13:40:04,687 INFO validate-wpt-results wpt-run:stderr   copying cffi/cffi_opcode.py -> build/lib.linux-x86_64-2.7/cffi
2019-02-13 13:40:04,687 INFO validate-wpt-results wpt-run:stderr   copying cffi/api.py -> build/lib.linux-x86_64-2.7/cffi
2019-02-13 13:40:04,687 INFO validate-wpt-results wpt-run:stderr   copying cffi/pkgconfig.py -> build/lib.linux-x86_64-2.7/cffi
2019-02-13 13:40:04,688 INFO validate-wpt-results wpt-run:stderr   copying cffi/model.py -> build/lib.linux-x86_64-2.7/cffi
2019-02-13 13:40:04,688 INFO validate-wpt-results wpt-run:stderr   copying cffi/cparser.py -> build/lib.linux-x86_64-2.7/cffi
2019-02-13 13:40:04,688 INFO validate-wpt-results wpt-run:stderr   copying cffi/_cffi_include.h -> build/lib.linux-x86_64-2.7/cffi
2019-02-13 13:40:04,688 INFO validate-wpt-results wpt-run:stderr   copying cffi/parse_c_type.h -> build/lib.linux-x86_64-2.7/cffi
2019-02-13 13:40:04,688 INFO validate-wpt-results wpt-run:stderr   copying cffi/_embedding.h -> build/lib.linux-x86_64-2.7/cffi
2019-02-13 13:40:04,688 INFO validate-wpt-results wpt-run:stderr   copying cffi/_cffi_errors.h -> build/lib.linux-x86_64-2.7/cffi
2019-02-13 13:40:04,688 INFO validate-wpt-results wpt-run:stderr   running build_ext
2019-02-13 13:40:04,688 INFO validate-wpt-results wpt-run:stderr   building '_cffi_backend' extension
2019-02-13 13:40:04,689 INFO validate-wpt-results wpt-run:stderr   creating build/temp.linux-x86_64-2.7
2019-02-13 13:40:04,689 INFO validate-wpt-results wpt-run:stderr   creating build/temp.linux-x86_64-2.7/c
2019-02-13 13:40:04,689 INFO validate-wpt-results wpt-run:stderr   x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fdebug-prefix-map=/build/python2.7-3hk45v/python2.7-2.7.15~rc1=. -fstack-protector-strong -Wformat -Werror=format-security -fPIC -DUSE__THREAD -DHAVE_SYNC_SYNCHRONIZE -I/usr/include/ffi -I/usr/include/libffi -I/usr/include/python2.7 -c c/_cffi_backend.c -o build/temp.linux-x86_64-2.7/c/_cffi_backend.o
2019-02-13 13:40:04,689 INFO validate-wpt-results wpt-run:stderr   c/_cffi_backend.c:15:10: fatal error: ffi.h: No such file or directory
2019-02-13 13:40:04,689 INFO validate-wpt-results wpt-run:stderr    #include <ffi.h>
2019-02-13 13:40:04,689 INFO validate-wpt-results wpt-run:stderr             ^~~~~~~
2019-02-13 13:40:04,689 INFO validate-wpt-results wpt-run:stderr   compilation terminated.
2019-02-13 13:40:04,690 INFO validate-wpt-results wpt-run:stderr   error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
2019-02-13 13:40:04,690 INFO validate-wpt-results wpt-run:stderr 
2019-02-13 13:40:04,690 INFO validate-wpt-results wpt-run:stderr   ----------------------------------------
2019-02-13 13:40:04,690 INFO validate-wpt-results wpt-run:stderr   Running setup.py clean for cffi
2019-02-13 13:40:04,690 INFO validate-wpt-results wpt-run:stderr Failed to build cffi
2019-02-13 13:40:04,690 INFO validate-wpt-results wpt-run:stderr Installing collected packages: six, webencodings, html5lib, mozfile, mozinfo, mozterm, blessings, mozlog, mozdebug, asn1crypto, enum34, ipaddress, pycparser, cffi, cryptography, pyOpenSSL
2019-02-13 13:40:04,690 INFO validate-wpt-results wpt-run:stderr   Running setup.py install for cffi: started
2019-02-13 13:40:04,690 INFO validate-wpt-results wpt-run:stderr     Running setup.py install for cffi: finished with status 'error'
2019-02-13 13:40:04,691 INFO validate-wpt-results wpt-run:stderr     Complete output from command /home/jade/worker/GNU_Linux_Chunked_Runner/build/_venv/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-FhQJPN/cffi/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-record-LhHMGn/install-record.txt --single-version-externally-managed --compile --install-headers /home/jade/worker/GNU_Linux_Chunked_Runner/build/_venv/include/site/python2.7/cffi:
2019-02-13 13:40:04,691 INFO validate-wpt-results wpt-run:stderr     running install
2019-02-13 13:40:04,691 INFO validate-wpt-results wpt-run:stderr     running build
2019-02-13 13:40:04,691 INFO validate-wpt-results wpt-run:stderr     running build_py
2019-02-13 13:40:04,691 INFO validate-wpt-results wpt-run:stderr     creating build
2019-02-13 13:40:04,691 INFO validate-wpt-results wpt-run:stderr     creating build/lib.linux-x86_64-2.7
2019-02-13 13:40:04,692 INFO validate-wpt-results wpt-run:stderr     creating build/lib.linux-x86_64-2.7/cffi
2019-02-13 13:40:04,692 INFO validate-wpt-results wpt-run:stderr     copying cffi/ffiplatform.py -> build/lib.linux-x86_64-2.7/cffi
2019-02-13 13:40:04,692 INFO validate-wpt-results wpt-run:stderr     copying cffi/error.py -> build/lib.linux-x86_64-2.7/cffi
2019-02-13 13:40:04,692 INFO validate-wpt-results wpt-run:stderr     copying cffi/lock.py -> build/lib.linux-x86_64-2.7/cffi
2019-02-13 13:40:04,692 INFO validate-wpt-results wpt-run:stderr     copying cffi/__init__.py -> build/lib.linux-x86_64-2.7/cffi
2019-02-13 13:40:04,692 INFO validate-wpt-results wpt-run:stderr     copying cffi/commontypes.py -> build/lib.linux-x86_64-2.7/cffi
2019-02-13 13:40:04,692 INFO validate-wpt-results wpt-run:stderr     copying cffi/setuptools_ext.py -> build/lib.linux-x86_64-2.7/cffi
2019-02-13 13:40:04,692 INFO validate-wpt-results wpt-run:stderr     copying cffi/vengine_gen.py -> build/lib.linux-x86_64-2.7/cffi
2019-02-13 13:40:04,693 INFO validate-wpt-results wpt-run:stderr     copying cffi/verifier.py -> build/lib.linux-x86_64-2.7/cffi
2019-02-13 13:40:04,693 INFO validate-wpt-results wpt-run:stderr     copying cffi/vengine_cpy.py -> build/lib.linux-x86_64-2.7/cffi
2019-02-13 13:40:04,693 INFO validate-wpt-results wpt-run:stderr     copying cffi/backend_ctypes.py -> build/lib.linux-x86_64-2.7/cffi
2019-02-13 13:40:04,693 INFO validate-wpt-results wpt-run:stderr     copying cffi/recompiler.py -> build/lib.linux-x86_64-2.7/cffi
2019-02-13 13:40:04,693 INFO validate-wpt-results wpt-run:stderr     copying cffi/cffi_opcode.py -> build/lib.linux-x86_64-2.7/cffi
2019-02-13 13:40:04,693 INFO validate-wpt-results wpt-run:stderr     copying cffi/api.py -> build/lib.linux-x86_64-2.7/cffi
2019-02-13 13:40:04,693 INFO validate-wpt-results wpt-run:stderr     copying cffi/pkgconfig.py -> build/lib.linux-x86_64-2.7/cffi
2019-02-13 13:40:04,694 INFO validate-wpt-results wpt-run:stderr     copying cffi/model.py -> build/lib.linux-x86_64-2.7/cffi
2019-02-13 13:40:04,694 INFO validate-wpt-results wpt-run:stderr     copying cffi/cparser.py -> build/lib.linux-x86_64-2.7/cffi
2019-02-13 13:40:04,694 INFO validate-wpt-results wpt-run:stderr     copying cffi/_cffi_include.h -> build/lib.linux-x86_64-2.7/cffi
2019-02-13 13:40:04,694 INFO validate-wpt-results wpt-run:stderr     copying cffi/parse_c_type.h -> build/lib.linux-x86_64-2.7/cffi
2019-02-13 13:40:04,694 INFO validate-wpt-results wpt-run:stderr     copying cffi/_embedding.h -> build/lib.linux-x86_64-2.7/cffi
2019-02-13 13:40:04,694 INFO validate-wpt-results wpt-run:stderr     copying cffi/_cffi_errors.h -> build/lib.linux-x86_64-2.7/cffi
2019-02-13 13:40:04,694 INFO validate-wpt-results wpt-run:stderr     running build_ext
2019-02-13 13:40:04,694 INFO validate-wpt-results wpt-run:stderr     building '_cffi_backend' extension
2019-02-13 13:40:04,694 INFO validate-wpt-results wpt-run:stderr     creating build/temp.linux-x86_64-2.7
2019-02-13 13:40:04,695 INFO validate-wpt-results wpt-run:stderr     creating build/temp.linux-x86_64-2.7/c
2019-02-13 13:40:04,695 INFO validate-wpt-results wpt-run:stderr     x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fdebug-prefix-map=/build/python2.7-3hk45v/python2.7-2.7.15~rc1=. -fstack-protector-strong -Wformat -Werror=format-security -fPIC -DUSE__THREAD -DHAVE_SYNC_SYNCHRONIZE -I/usr/include/ffi -I/usr/include/libffi -I/usr/include/python2.7 -c c/_cffi_backend.c -o build/temp.linux-x86_64-2.7/c/_cffi_backend.o
2019-02-13 13:40:04,695 INFO validate-wpt-results wpt-run:stderr     c/_cffi_backend.c:15:10: fatal error: ffi.h: No such file or directory
2019-02-13 13:40:04,695 INFO validate-wpt-results wpt-run:stderr      #include <ffi.h>
2019-02-13 13:40:04,695 INFO validate-wpt-results wpt-run:stderr               ^~~~~~~
2019-02-13 13:40:04,695 INFO validate-wpt-results wpt-run:stderr     compilation terminated.
2019-02-13 13:40:04,696 INFO validate-wpt-results wpt-run:stderr     error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
2019-02-13 13:40:04,696 INFO validate-wpt-results wpt-run:stderr 
2019-02-13 13:40:04,696 INFO validate-wpt-results wpt-run:stderr     ----------------------------------------
2019-02-13 13:40:04,696 INFO validate-wpt-results wpt-run:stderr 
2019-02-13 13:40:04,696 INFO validate-wpt-results wpt-run:stderr Traceback (most recent call last):
2019-02-13 13:40:04,696 INFO validate-wpt-results wpt-run:stderr   File "./wpt", line 5, in <module>
2019-02-13 13:40:04,696 INFO validate-wpt-results wpt-run:stderr     wpt.main()
2019-02-13 13:40:04,696 INFO validate-wpt-results wpt-run:stderr   File "/home/jade/worker/GNU_Linux_Chunked_Runner/build/tools/wpt/wpt.py", line 115, in main
2019-02-13 13:40:04,697 INFO validate-wpt-results wpt-run:stderr     venv = setup_virtualenv(main_args.venv, main_args.skip_venv_setup, props)
2019-02-13 13:40:04,697 INFO validate-wpt-results wpt-run:stderr   File "/home/jade/worker/GNU_Linux_Chunked_Runner/build/tools/wpt/wpt.py", line 95, in setup_virtualenv
2019-02-13 13:40:04,697 INFO validate-wpt-results wpt-run:stderr     venv.install_requirements(path)
2019-02-13 13:40:04,697 INFO validate-wpt-results wpt-run:stderr   File "/home/jade/worker/GNU_Linux_Chunked_Runner/build/tools/wpt/virtualenv.py", line 55, in install_requirements
2019-02-13 13:40:04,697 INFO validate-wpt-results wpt-run:stderr     call(self.pip_path, "install", "-r", requirements_path)
2019-02-13 13:40:04,697 INFO validate-wpt-results wpt-run:stderr   File "/home/jade/worker/GNU_Linux_Chunked_Runner/build/tools/wpt/utils.py", line 43, in call
2019-02-13 13:40:04,697 INFO validate-wpt-results wpt-run:stderr     return subprocess.check_output(args)
2019-02-13 13:40:04,697 INFO validate-wpt-results wpt-run:stderr   File "/usr/lib/python2.7/subprocess.py", line 223, in check_output
2019-02-13 13:40:04,698 INFO validate-wpt-results wpt-run:stderr     raise CalledProcessError(retcode, cmd, output=output)
2019-02-13 13:40:04,698 INFO validate-wpt-results wpt-run:stderr subprocess.CalledProcessError: Command '('/home/jade/worker/GNU_Linux_Chunked_Runner/build/_venv/bin/pip', 'install', '-r', u'/home/jade/worker/GNU_Linux_Chunked_Runner/build/tools/wpt/../wptrunner/requirements.txt')' returned non-zero exit status 1
2019-02-13 13:40:04,698 INFO validate-wpt-results WPT CLI exited with return code None
@gsnedders

This comment has been minimized.

Copy link
Collaborator

commented Feb 18, 2019

My guess is we don't have the libffi-dev package installed hence we couldn't build cffi from source, and the appropriate cffi wheel hadn't been uploaded yet?

@jugglinmike

This comment has been minimized.

Copy link
Collaborator Author

commented Feb 18, 2019

That sounds plausible and more satisfying than my feeble "transitory packaging issue."

@jugglinmike

This comment has been minimized.

Copy link
Collaborator Author

commented Feb 26, 2019

Surprisingly, two more builds just failed with the very same error (8 days later). This happened about 3 hours ago, and that directly correlates with the release of cffi 1.12.2.

@gsnedders @jgraham I know next to nothing about Python packaging, but this seems like something we ought to be able to avoid. Is there anything we can change in WPT to guard against invalid installations?

@jgraham

This comment has been minimized.

Copy link
Collaborator

commented Feb 26, 2019

Hah.

cffi seems to be a dep of pyOpenSSL, which depends on cffi [required: >=1.8,!=1.11.3, installed: 1.11.5]. I don't think that pinning transitive dependencies is supported in any good way, but what could work is this:

Adjust the requirements.txt file to specify a specific version of cffi. Assume that pip will notice that the pinned version is a match for the requirement and not try to install a newer version. Then rely on pyup.io to install newer versions which will come with enough delay to assure we don't end up with races for the wheel to be uploaded.

@jugglinmike

This comment has been minimized.

Copy link
Collaborator Author

commented Feb 26, 2019

Thanks, James! That looks like a fairly module-specific solution. Is that an indication of some bad practice in pyOpenSSL or cffi? Or is this a problem that Python developers experience commonly?

@jgraham

This comment has been minimized.

Copy link
Collaborator

commented Feb 27, 2019

Ooh, pip seems to have a --prefer-binary option. I think that might help here.

@jugglinmike

This comment has been minimized.

Copy link
Collaborator Author

commented Feb 27, 2019

Six more builds (3 Firefox, 2 Chrome, and 1 Edge) were interrupted today at 14:05 UTC. There's no new release of cffi to blame this time, though. @jgraham do you know how this could happen again? Could it occur when some transitive dependency transitions to the new release?

@jgraham

This comment has been minimized.

Copy link
Collaborator

commented Feb 27, 2019

Do you have logs?

@jugglinmike

This comment has been minimized.

Copy link
Collaborator Author

commented Feb 27, 2019

@jgraham

This comment has been minimized.

Copy link
Collaborator

commented Feb 27, 2019

I'm not sure why it's trying to build it there; maybe something happened to the wheel package. In any case passing --prefer-binary or --only-binary=:all: to the pip command seems like it should avoid this failure mode.

@gsnedders

This comment has been minimized.

Copy link
Collaborator

commented Feb 28, 2019

At least the ones I looked at there aren't cffi, they're cryptography, which had two releases yesterday.

To expand on my previous comment: note that uploads aren't atomic on PyPI.

The common flow is:

  1. Create a new release.
  2. Upload an artifact.
  3. If you have another artifact, goto 2; else, complete.

Each of these items happens sequentially, and if a source release artifact is uploaded before a (suitable) binary release artifact is anything looking for the latest release will download the source release and build it. As noted, either of the options @jgraham suggested will passing this will change this (preferring an older release with a binary release).

@jugglinmike

This comment has been minimized.

Copy link
Collaborator Author

commented Mar 1, 2019

We got hit by both releases of the "cryptography" package (the second release interfered with six builds for Firefox stable). I'm going to look in to patching WPT as you folks recommend. I'm curious, though: have either of you seen this occur in Taskcluster?

jgraham added a commit to web-platform-tests/wpt that referenced this issue Mar 1, 2019

moz-v2v-gh pushed a commit to mozilla/gecko-dev that referenced this issue Apr 1, 2019

Bug 1531941 [wpt PR 15614] - [wpt] Avoid race condition during module…
… install, a=testonly

Automatic update from web-platform-tests
[wpt] Avoid race condition during module install (#15614)

See web-platform-tests/results-collection#649
--

wpt-commits: cc4f2960306867e50d5beba4e86749aa75961fe8
wpt-pr: 15614

mykmelez pushed a commit to mykmelez/gecko that referenced this issue Apr 2, 2019

Bug 1531941 [wpt PR 15614] - [wpt] Avoid race condition during module…
… install, a=testonly

Automatic update from web-platform-tests
[wpt] Avoid race condition during module install (#15614)

See web-platform-tests/results-collection#649
--

wpt-commits: cc4f2960306867e50d5beba4e86749aa75961fe8
wpt-pr: 15614

gecko-dev-updater pushed a commit to marco-c/gecko-dev-comments-removed that referenced this issue Jul 30, 2019

Bug 1531941 [wpt PR 15614] - [wpt] Avoid race condition during module…
… install, a=testonly

Automatic update from web-platform-tests
[wpt] Avoid race condition during module install (#15614)

See web-platform-tests/results-collection#649
--

wpt-commits: cc4f2960306867e50d5beba4e86749aa75961fe8
wpt-pr: 15614

UltraBlame original commit: 63a641c33b4d6264c2226911fb61d7d59f9aacff

gecko-dev-updater pushed a commit to marco-c/gecko-dev-wordified-and-comments-removed that referenced this issue Jul 31, 2019

Bug 1531941 [wpt PR 15614] - [wpt] Avoid race condition during module…
… install, a=testonly

Automatic update from web-platform-tests
[wpt] Avoid race condition during module install (#15614)

See web-platform-tests/results-collection#649
--

wpt-commits: cc4f2960306867e50d5beba4e86749aa75961fe8
wpt-pr: 15614

UltraBlame original commit: 63a641c33b4d6264c2226911fb61d7d59f9aacff
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.