QA's syncstorage-loadtest env broken

[pjenvey]

Rebecca's seeing the following error with a $ sudo apt-get install python3-dev not solving it

(0125) ubuntu@ip-172-31-50-232:~/Library/github/mozilla-services/syncstorage-loadtest$ pip install -r requirements.txt 

Collecting git+git://github.com/loads/molotov.git (from -r requirements.txt (line 1))

  Cloning git://github.com/loads/molotov.git to /tmp/pip-req-build-d7314wjx

Collecting pexpect (from -r requirements.txt (line 2))

  Using cached https://files.pythonhosted.org/packages/39/7b/88dbb785881c28a102619d46423cb853b46dbccc70d3ac362d99773a78ce/pexpect-4.8.0-py2.py3-none-any.whl

Collecting configparser==3.5.0b2 (from -r requirements.txt (line 3))

  Using cached https://files.pythonhosted.org/packages/61/a7/48f627413200d5dfd0a5da8b494d34ccac03f8a87a6b2d765b7d81e175e3/configparser-3.5.0b2.tar.gz

Collecting PyBrowserID (from -r requirements.txt (line 4))

  Using cached https://files.pythonhosted.org/packages/e4/2e/e9bb9e24e600da08ff6a4d003362434eed717151f58413d8f73427e7e315/PyBrowserID-0.14.0.tar.gz

Collecting hawkauthlib (from -r requirements.txt (line 5))

  Using cached https://files.pythonhosted.org/packages/53/ec/23dd5cbd5e950543fdd30d91ddac4f56e395d14316677aa4cb78a029f8e2/hawkauthlib-2.0.0-py2.py3-none-any.whl

Collecting tokenlib (from -r requirements.txt (line 6))

  Using cached https://files.pythonhosted.org/packages/f6/d0/dd970f811a64391355a387f0ba2a129f135c841b5ffb7d2d30a03ba6b4eb/tokenlib-2.0.0-py2.py3-none-any.whl

Collecting aiomeasures (from -r requirements.txt (line 7))

  Using cached https://files.pythonhosted.org/packages/0f/aa/40f89fc915669b8492b5f060597aff25d9b11ad10f28fb4a97cba6896658/aiomeasures-0.5.14-py2.py3-none-any.whl

Collecting aiohttp (from molotov==2.3->-r requirements.txt (line 1))

  Using cached https://files.pythonhosted.org/packages/68/96/40a765d7d68028c5a6d169b2747ea3f4828ec91a358a63818d468380521c/aiohttp-3.7.3.tar.gz

  Installing build dependencies ... done

Collecting aiodogstatsd (from molotov==2.3->-r requirements.txt (line 1))

  Using cached https://files.pythonhosted.org/packages/b9/30/6b34d1d89d01ef47b6f29e7dddff73a61e1fd7a646698d61de8e26b50f1a/aiodogstatsd-0.15.0-py3-none-any.whl

Collecting multiprocessing_on_dill (from molotov==2.3->-r requirements.txt (line 1))

  Using cached https://files.pythonhosted.org/packages/86/4d/4b135e2e5cd0194eb29f2ed36e9a77a07596787a9a8ac2279bd4445398f2/multiprocessing_on_dill-3.5.0a4.tar.gz

Collecting ptyprocess>=0.5 (from pexpect->-r requirements.txt (line 2))

  Using cached https://files.pythonhosted.org/packages/22/a6/858897256d0deac81a172289110f31629fc4cee19b6f01283303e18c8db3/ptyprocess-0.7.0-py2.py3-none-any.whl

Collecting requests (from PyBrowserID->-r requirements.txt (line 4))

  Using cached https://files.pythonhosted.org/packages/29/c1/24814557f1d22c56d50280771a17307e6bf87b70727d975fd6b2ce6b014a/requests-2.25.1-py2.py3-none-any.whl

Collecting webob (from hawkauthlib->-r requirements.txt (line 5))

  Using cached https://files.pythonhosted.org/packages/18/3c/de37900faff3c95c7d55dd557aa71bd77477950048983dcd4b53f96fde40/WebOb-1.8.6-py2.py3-none-any.whl

Collecting attrs>=17.3.0 (from aiohttp->molotov==2.3->-r requirements.txt (line 1))

  Using cached https://files.pythonhosted.org/packages/c3/aa/cb45262569fcc047bf070b5de61813724d6726db83259222cd7b4c79821a/attrs-20.3.0-py2.py3-none-any.whl

Collecting chardet<4.0,>=2.0 (from aiohttp->molotov==2.3->-r requirements.txt (line 1))

  Using cached https://files.pythonhosted.org/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl

Collecting multidict<7.0,>=4.5 (from aiohttp->molotov==2.3->-r requirements.txt (line 1))

  Using cached https://files.pythonhosted.org/packages/1c/74/e8b46156f37ca56d10d895d4e8595aa2b344cff3c1fb3629ec97a8656ccb/multidict-5.1.0.tar.gz

  Installing build dependencies ... done

Collecting async_timeout<4.0,>=3.0 (from aiohttp->molotov==2.3->-r requirements.txt (line 1))

  Using cached https://files.pythonhosted.org/packages/e1/1e/5a4441be21b0726c4464f3f23c8b19628372f606755a9d2e46c187e65ec4/async_timeout-3.0.1-py3-none-any.whl

Collecting yarl<2.0,>=1.0 (from aiohttp->molotov==2.3->-r requirements.txt (line 1))

  Using cached https://files.pythonhosted.org/packages/97/e7/af7219a0fe240e8ef6bb555341a63c43045c21ab0392b4435e754b716fa1/yarl-1.6.3.tar.gz

  Installing build dependencies ... done

Collecting typing_extensions>=3.6.5 (from aiohttp->molotov==2.3->-r requirements.txt (line 1))

  Using cached https://files.pythonhosted.org/packages/60/7a/e881b5abb54db0e6e671ab088d079c57ce54e8a01a3ca443f561ccadb37e/typing_extensions-3.7.4.3-py3-none-any.whl

Collecting idna-ssl>=1.0 (from aiohttp->molotov==2.3->-r requirements.txt (line 1))

  Using cached https://files.pythonhosted.org/packages/46/03/07c4894aae38b0de52b52586b24bf189bb83e4ddabfe2e2c8f2419eec6f4/idna-ssl-1.1.0.tar.gz

Collecting dill (from multiprocessing_on_dill->molotov==2.3->-r requirements.txt (line 1))

  Using cached https://files.pythonhosted.org/packages/52/d6/79f40d230895fa1ce3b6af0d22e0ac79c65175dc069c194b79cc8e05a033/dill-0.3.3-py2.py3-none-any.whl

Collecting idna<3,>=2.5 (from requests->PyBrowserID->-r requirements.txt (line 4))

  Using cached https://files.pythonhosted.org/packages/a2/38/928ddce2273eaa564f6f50de919327bf3a00f091b5baba8dfa9460f3a8a8/idna-2.10-py2.py3-none-any.whl

Collecting certifi>=2017.4.17 (from requests->PyBrowserID->-r requirements.txt (line 4))

  Using cached https://files.pythonhosted.org/packages/5e/a0/5f06e1e1d463903cf0c0eebeb751791119ed7a4b3737fdc9a77f1cdfb51f/certifi-2020.12.5-py2.py3-none-any.whl

Collecting urllib3<1.27,>=1.21.1 (from requests->PyBrowserID->-r requirements.txt (line 4))

  Using cached https://files.pythonhosted.org/packages/f5/71/45d36a8df68f3ebb098d6861b2c017f3d094538c0fb98fa61d4dc43e69b9/urllib3-1.26.2-py2.py3-none-any.whl

Installing collected packages: ptyprocess, pexpect, configparser, idna, certifi, urllib3, chardet, requests, PyBrowserID, webob, hawkauthlib, tokenlib, aiomeasures, attrs, multidict, async-timeout, typing-extensions, yarl, idna-ssl, aiohttp, aiodogstatsd, dill, multiprocessing-on-dill, molotov

  Running setup.py install for configparser ... done

  Running setup.py install for PyBrowserID ... done

  Running setup.py install for multidict ... error

    Complete output from command /home/ubuntu/Library/github/mozilla-services/syncstorage-loadtest/0125/bin/python3.6 -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-wrejk_mi/multidict/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-record-uqscyh3y/install-record.txt --single-version-externally-managed --compile --install-headers /home/ubuntu/Library/github/mozilla-services/syncstorage-loadtest/0125/include/site/python3.6/multidict:

    *********************

    * Accelerated build *

    *********************

    running install

    running build

    running build_py

    creating build

    creating build/lib.linux-x86_64-3.6

    creating build/lib.linux-x86_64-3.6/multidict

    copying multidict/_compat.py -> build/lib.linux-x86_64-3.6/multidict

    copying multidict/_multidict_py.py -> build/lib.linux-x86_64-3.6/multidict

    copying multidict/__init__.py -> build/lib.linux-x86_64-3.6/multidict

    copying multidict/_abc.py -> build/lib.linux-x86_64-3.6/multidict

    copying multidict/_multidict_base.py -> build/lib.linux-x86_64-3.6/multidict

    running egg_info

    writing multidict.egg-info/PKG-INFO

    writing dependency_links to multidict.egg-info/dependency_links.txt

    writing top-level names to multidict.egg-info/top_level.txt

    reading manifest file 'multidict.egg-info/SOURCES.txt'

    reading manifest template 'MANIFEST.in'

    warning: no previously-included files matching '*.pyc' found anywhere in distribution

    warning: no previously-included files found matching 'multidict/_multidict.html'

    warning: no previously-included files found matching 'multidict/*.so'

    warning: no previously-included files found matching 'multidict/*.pyd'

    warning: no previously-included files found matching 'multidict/*.pyd'

    no previously-included directories found matching 'docs/_build'

    writing manifest file 'multidict.egg-info/SOURCES.txt'

    copying multidict/__init__.pyi -> build/lib.linux-x86_64-3.6/multidict

    copying multidict/_multidict.c -> build/lib.linux-x86_64-3.6/multidict

    copying multidict/py.typed -> build/lib.linux-x86_64-3.6/multidict

    creating build/lib.linux-x86_64-3.6/multidict/_multilib

    copying multidict/_multilib/defs.h -> build/lib.linux-x86_64-3.6/multidict/_multilib

    copying multidict/_multilib/dict.h -> build/lib.linux-x86_64-3.6/multidict/_multilib

    copying multidict/_multilib/istr.h -> build/lib.linux-x86_64-3.6/multidict/_multilib

    copying multidict/_multilib/iter.h -> build/lib.linux-x86_64-3.6/multidict/_multilib

    copying multidict/_multilib/pair_list.h -> build/lib.linux-x86_64-3.6/multidict/_multilib

    copying multidict/_multilib/views.h -> build/lib.linux-x86_64-3.6/multidict/_multilib

    running build_ext

    building 'multidict._multidict' extension

    creating build/temp.linux-x86_64-3.6

    creating build/temp.linux-x86_64-3.6/multidict

    x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/home/ubuntu/Library/github/mozilla-services/syncstorage-loadtest/0125/include -I/usr/include/python3.6m -c multidict/_multidict.c -o build/temp.linux-x86_64-3.6/multidict/_multidict.o -O2 -std=c99 -Wall -Wsign-compare -Wconversion -fno-strict-aliasing -pedantic

    multidict/_multidict.c:1:20: fatal error: Python.h: No such file or directory

    compilation terminated.

    error: command 'x86_64-linux-gnu-gcc' failed with exit status 1

    

    ----------------------------------------

Command "/home/ubuntu/Library/github/mozilla-services/syncstorage-loadtest/0125/bin/python3.6 -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-wrejk_mi/multidict/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-record-uqscyh3y/install-record.txt --single-version-externally-managed --compile --install-headers /home/ubuntu/Library/github/mozilla-services/syncstorage-loadtest/0125/include/site/python3.6/multidict" failed with error code 1 in /tmp/pip-install-wrejk_mi/multidict/

[jrconlin]

Having talked with QA, this is possibly OBE. They were able to find a second server instance where the tests could be run without errors.

In addition mozilla-services/syncstorage-loadtest#34 may also address the problem.