"Too many open files" while using BeakerExecutor

[dirkgr]

🐛 Describe the bug

Currently, the number of concurrent requests that BeakerExecutor makes to the Beaker service is capped at 3. Beaker fixed their service limitations, so I set this number to 9. Now I am getting on my MacBook, saying "too many open files".

Steps to reproduce:

• find a large tango config and run it
• load up the URL that it prints (because the Tango server will make a ton of requests against Beaker)
• wait 5 minutes or so

This SO post has ideas about how to fix it: https://stackoverflow.com/questions/55230576/too-many-open-files-when-using-requests-package-python

Versions

Python 3.9.12
absl-py==1.0.0
-e git+https://github.com/allenai/tango.git@903569c820e3cd72f598c02da46f9170cce4fa17#egg=ai2_tango
aiohttp==3.8.1
aiosignal==1.2.0
alabaster==0.7.12
appnope @ file:///opt/concourse/worker/volumes/live/6ca6f098-d773-4461-5c91-a24a17435bda/volume/appnope_1606859448531/work
asttokens @ file:///opt/conda/conda-bld/asttokens_1646925590279/work
astunparse==1.6.3
async-timeout==4.0.2
attrs==21.4.0
Babel==2.10.1
backcall @ file:///home/ktietz/src/ci/backcall_1611930011877/work
base58==2.1.1
beaker-py==1.4.2
beautifulsoup4==4.11.1
black==22.3.0
bleach==5.0.0
BLEURT @ https://github.com/google-research/bleurt/archive/b610120347ef22b494b6d69b4316e303f5932516.zip
blis==0.7.7
boto3==1.23.3
botocore==1.26.3
brotlipy==0.7.0
cached-path==1.1.2
cachetools==5.1.0
catalogue==2.0.7
catwalk @ git+https://github.com/allenai/catwalk.git@738e19253f9ea2fddd54a3e6b1265743a7566a11
certifi==2021.10.8
cffi @ file:///private/var/folders/sy/f16zz6x50xz3113nwtb9bvq00000gp/T/croot-2p6q7e2r/cffi_1642701115063/work
chardet==4.0.0
charset-normalizer @ file:///tmp/build/80754af9/charset-normalizer_1630003229654/work
click==8.1.2
click-help-colors==0.9.1
colorama==0.4.4
commonmark==0.9.1
coverage==6.3.3
cryptography @ file:///private/var/folders/sy/f16zz6x50xz3113nwtb9bvq00000gp/T/abs_b470f7cb-c8f1-42c9-b84f-23789ea77e7c9difcqh4/croots/recipe/cryptography_1652101134392/work
cymem==2.0.6
Cython==0.29.30
DataProperty==0.55.0
datasets==2.2.1
decorator @ file:///opt/conda/conda-bld/decorator_1643638310831/work
dill==0.3.4
docker==5.0.3
docker-pycreds==0.4.0
docutils==0.17.1
dyNET==2.1.2
executing @ file:///opt/conda/conda-bld/executing_1646925071911/work
fairscale==0.4.6
filelock==3.6.0
flake8==4.0.1
flatbuffers==1.12
frozenlist==1.3.0
fsspec==2022.3.0
furo==2022.4.7
gast==0.4.0
gitdb==4.0.9
GitPython==3.1.27
glob2==0.7
google-api-core==2.8.0
google-auth==2.6.6
google-auth-oauthlib==0.4.6
google-cloud-core==2.3.0
google-cloud-storage==2.3.0
google-crc32c==1.3.0
google-pasta==0.2.0
google-resumable-media==2.3.2
googleapis-common-protos==1.56.1
grpcio==1.46.1
h5py==3.6.0
huggingface-hub==0.5.1
idna @ file:///tmp/build/80754af9/idna_1637925883363/work
imagesize==1.3.0
importlib-metadata==4.11.3
iniconfig==1.1.1
ipython @ file:///private/var/folders/sy/f16zz6x50xz3113nwtb9bvq00000gp/T/abs_2ff86d21-7133-4fee-b3b2-6349ba52809eua87k1km/croots/recipe/ipython_1651600160735/work
isort==5.10.1
jedi @ file:///opt/concourse/worker/volumes/live/89be3eb0-a85c-453b-67dd-f706f2fa4c43/volume/jedi_1644315269953/work
jieba==0.42.1
Jinja2==3.1.2
jmespath==1.0.0
joblib==1.1.0
jsonlines==2.0.0
jsonnet==0.18.0
keras==2.9.0
Keras-Preprocessing==1.1.2
keyring==23.5.0
langcodes==3.3.0
libclang==14.0.1
livereload==2.6.3
lm-dataformat==0.0.20
lm-eval @ git+https://github.com/dirkgr/lm-evaluation-harness.git@aeb7e846628da1e2dbecb0340edeaef433b487d4
Markdown==3.3.7
markdown-it-py==2.1.0
MarkupSafe==2.1.1
matplotlib-inline @ file:///tmp/build/80754af9/matplotlib-inline_1628242447089/work
mbstrdecoder==1.1.0
mccabe==0.6.1
mdit-py-plugins==0.3.0
mdurl==0.1.1
mkl-fft==1.3.1
mkl-random @ file:///opt/concourse/worker/volumes/live/0cda23d8-7460-44b2-7e5d-3c76a8a0ca7e/volume/mkl_random_1626186083266/work
mkl-service==2.4.0
mock==4.0.3
more-itertools==8.13.0
msgfy==0.2.0
multidict==6.0.2
multiprocess==0.70.12.2
murmurhash==1.0.7
mypy==0.950
mypy-extensions==0.4.3
myst-parser==0.17.2
nagisa==0.2.7
nltk==3.7
numexpr==2.7.2
numpy @ file:///private/var/folders/sy/f16zz6x50xz3113nwtb9bvq00000gp/T/abs_e3f85f54-0572-4c0e-b190-4bc4766fc3fenewxarhq/croots/recipe/numpy_and_numpy_base_1652801682879/work
oauthlib==3.2.0
openai==0.6.4
opt-einsum==3.3.0
packaging==21.3
pandas==1.4.2
parso @ file:///opt/conda/conda-bld/parso_1641458642106/work
pathspec==0.9.0
pathtools==0.1.2
pathvalidate==2.5.0
pathy==0.6.1
petname==2.6
pexpect @ file:///tmp/build/80754af9/pexpect_1605563209008/work
pickleshare @ file:///tmp/build/80754af9/pickleshare_1606932040724/work
Pillow==9.0.1
pkginfo==1.8.2
platformdirs==2.5.2
pluggy==1.0.0
portalocker==2.4.0
preshed==3.0.6
promise==2.3
prompt-toolkit @ file:///tmp/build/80754af9/prompt-toolkit_1633440160888/work
protobuf==3.20.0
psutil==5.9.0
ptyprocess @ file:///tmp/build/80754af9/ptyprocess_1609355006118/work/dist/ptyprocess-0.7.0-py2.py3-none-any.whl
pure-eval @ file:///opt/conda/conda-bld/pure_eval_1646925070566/work
py==1.11.0
pyarrow==8.0.0
pyasn1==0.4.8
pyasn1-modules==0.2.8
pybind11==2.6.2
pycodestyle==2.8.0
pycountry==20.7.3
pycparser @ file:///tmp/build/80754af9/pycparser_1636541352034/work
pydantic==1.9.1
pyDeprecate==0.3.2
pyflakes==2.4.0
Pygments==2.12.0
pyOpenSSL @ file:///opt/conda/conda-bld/pyopenssl_1643788558760/work
pyparsing==3.0.9
PySocks @ file:///opt/concourse/worker/volumes/live/112288ac-9cb0-4e73-768b-13baf4ca6419/volume/pysocks_1605305820043/work
pytablewriter==0.58.0
pytest==7.1.2
pytest-cov==3.0.0
pytest-sphinx==0.4.0
python-dateutil==2.8.2
pytorch-lightning==1.6.3
pytz==2022.1
PyYAML==6.0
readme-renderer==35.0
regex==2022.4.24
requests @ file:///opt/conda/conda-bld/requests_1641824580448/work
requests-oauthlib==1.3.1
requests-toolbelt==0.9.1
responses==0.18.0
rfc3986==2.0.0
rich==12.4.1
rouge-score==0.0.4
rsa==4.8
s3transfer==0.5.2
sacrebleu==1.5.0
sacremoses==0.0.53
scikit-learn==1.1.0
scipy==1.8.1
sentencepiece==0.1.96
sentry-sdk==1.5.12
setproctitle==1.2.3
shortuuid==1.0.9
six @ file:///tmp/build/80754af9/six_1644875935023/work
smart-open==5.2.1
smmap==5.0.0
snowballstemmer==2.2.0
soupsieve==2.3.2.post1
spacy==3.3.0
spacy-legacy==3.0.9
spacy-loggers==1.0.2
Sphinx==4.5.0
sphinx-autobuild==2021.3.14
sphinx-autodoc-typehints==1.18.1
sphinx-copybutton==0.5.0
sphinxcontrib-applehelp==1.0.2
sphinxcontrib-devhelp==1.0.2
sphinxcontrib-htmlhelp==2.0.0
sphinxcontrib-jsmath==1.0.1
sphinxcontrib-qthelp==1.0.3
sphinxcontrib-serializinghtml==1.1.5
sqlitedict==1.6.0
srsly==2.4.3
stack-data @ file:///opt/conda/conda-bld/stack_data_1646927590127/work
tabledata==1.3.0
-e git+https://github.com/allenai/task-complexity.git@2eb8a415b953ea3d137d1be7938fdf129be184bc#egg=tc
tcolorpy==0.1.2
tensorboard==2.9.0
tensorboard-data-server==0.6.1
tensorboard-plugin-wit==1.8.1
tensorflow==2.9.0
tensorflow-estimator==2.9.0
tensorflow-io-gcs-filesystem==0.26.0
termcolor==1.1.0
tf-slim==1.1.0
thinc==8.0.15
threadpoolctl==3.1.0
tokenizers==0.12.1
toml @ file:///tmp/build/80754af9/toml_1616166611790/work
tomli==2.0.1
torch==1.11.0
torchaudio==0.11.0
torchmetrics==0.9.1
torchvision==0.12.0
tornado==6.1
tqdm==4.64.0
tqdm-multiprocess==0.0.11
traitlets @ file:///tmp/build/80754af9/traitlets_1636710298902/work
transformers==4.19.2
twine==4.0.0
typepy==1.3.0
typer==0.4.1
typing_extensions @ file:///opt/conda/conda-bld/typing_extensions_1647553014482/work
ujson==5.3.0
urllib3 @ file:///private/var/folders/sy/f16zz6x50xz3113nwtb9bvq00000gp/T/abs_92f3a881-8cf4-469b-8319-736ffa3c6d61ordl4zjw/croots/recipe/urllib3_1650639998913/work
wandb==0.12.16
wasabi==0.9.1
wcwidth @ file:///Users/ktietz/demo/mc3/conda-bld/wcwidth_1629357192024/work
webencodings==0.5.1
websocket-client==1.3.2
Werkzeug==2.1.2
wget==3.2
wrapt==1.14.1
xxhash==3.0.0
yarl==1.7.2
zipp==3.8.0
zstandard==0.15.2