Stuck when running webshop evaluation #43
Description
Hi, when I use the perbuilt docker and run the webshop with llama2, I run the following command:
python create_assignment.py --assignment configs/assignments/example-our.yaml
bash .assigments/***.sh
here is my assignment yaml file:
default:
agent: configs/agents/api_agents/llama2-7B.yaml
task:
parameters:
workers: 15
assignments:
from: "configs/tasks/webshop/dev.yaml"
parameters:
workers: 6
When I execute it, no error is reported, but it blocks on the last sample with the following output:
bash: /home/haivlab/anaconda3/lib/libtinfo.so.6: no version information available (required by bash)
Evaluating in docker localhost/task:webshop, Parameters: --task outputs/2023-09-14-21-47-35/llama2_7b_chat_hf/WebShop-dev/task.yaml --agent outputs/2023-09-14-21-47-35/llama2_7b_chat_hf/WebShop-dev/agent.yaml --output outputs/2023-09-14-21-47-35/llama2_7b_chat_hf/WebShop-dev
{'module': 'src.tasks.WebShop', 'parameters': {'end': 280, 'max_tokens': 4096, 'name': 'WebShop-dev', 'num_envs': 3, 'start': 200, 'worker_limit': 3, 'workers': 6}}
{'module': 'src.agents.HTTPAgent', 'parameters': {'body': {'Key2': 'Value2', 'model': 'llama2_7b_chat_hf'}, 'headers': {'Content-Type': 'application/json'}, 'max_tokens': 4096, 'name': 'llama2_7b_chat_hf', 'prompter': {'args': {'agent_role': 'assistant'}, 'name': 'role_content_dict'}, 'url': 'http://localhost:8000/v1/chat/completions'}}
[Evaluation] Loading Agent ...
[Evaluation] Successfully loaded Agent.
[Evaluation] Loading Task ...
> [Warning] OSInteraction task not available
> [Warning] ALFWorld task not available
> [Warning] DBBench task not available
Warning: Gym version v0.24.0 has a number of critical issues with `gym.make` such that the `reset` and `step` functions are called before returning the environment. It is recommend to downgrading to v0.23.1 or upgrading to v0.25.1
/root/miniconda3/envs/webshop/lib/python3.8/site-packages/pkg_resources/__init__.py:121: DeprecationWarning: pkg_resources is deprecated as an API
warnings.warn("pkg_resources is deprecated as an API", DeprecationWarning)
/root/miniconda3/envs/webshop/lib/python3.8/site-packages/pkg_resources/__init__.py:2870: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('mpl_toolkits')`.
Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
declare_namespace(pkg)
/root/miniconda3/envs/webshop/lib/python3.8/site-packages/faiss/loader.py:28: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
if LooseVersion(numpy.__version__) >= "1.19":
/root/miniconda3/envs/webshop/lib/python3.8/site-packages/setuptools/_distutils/version.py:345: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
other = LooseVersion(other)
/root/miniconda3/envs/webshop/lib/python3.8/site-packages/thefuzz/fuzz.py:11: UserWarning: Using slow pure-python SequenceMatcher. Install python-Levenshtein to remove this warning
warnings.warn('Using slow pure-python SequenceMatcher. Install python-Levenshtein to remove this warning')
/root/miniconda3/envs/webshop/lib/python3.8/site-packages/gym/envs/registration.py:516: UserWarning: WARN: Overriding environment WebAgentSiteEnv-v0
logger.warn(f"Overriding environment {spec.id}")
/root/miniconda3/envs/webshop/lib/python3.8/site-packages/gym/envs/registration.py:516: UserWarning: WARN: Overriding environment WebAgentTextEnv-v0
logger.warn(f"Overriding environment {spec.id}")
> [Warning] Mind2Web task not available
> [Warning] KnowledgeGraph task not available
[Evaluation] Successfully loaded Task.
Evaluating task 'WebShop-dev' ...
Start Predicting All ...
0%| | 0/80 [00:00<?, ?it/s]> [Warning] OSInteraction task not available
> [Warning] ALFWorld task not available
> [Warning] DBBench task not available
> [Warning] OSInteraction task not available
> [Warning] ALFWorld task not available
> [Warning] DBBench task not available
> [Warning] OSInteraction task not available
Warning: Gym version v0.24.0 has a number of critical issues with `gym.make` such that the `reset` and `step` functions are called before returning the environment. It is recommend to downgrading to v0.23.1 or upgrading to v0.25.1
> [Warning] ALFWorld task not available
> [Warning] DBBench task not available
Warning: Gym version v0.24.0 has a number of critical issues with `gym.make` such that the `reset` and `step` functions are called before returning the environment. It is recommend to downgrading to v0.23.1 or upgrading to v0.25.1
Warning: Gym version v0.24.0 has a number of critical issues with `gym.make` such that the `reset` and `step` functions are called before returning the environment. It is recommend to downgrading to v0.23.1 or upgrading to v0.25.1
/root/miniconda3/envs/webshop/lib/python3.8/site-packages/pkg_resources/__init__.py:121: DeprecationWarning: pkg_resources is deprecated as an API
warnings.warn("pkg_resources is deprecated as an API", DeprecationWarning)
/root/miniconda3/envs/webshop/lib/python3.8/site-packages/pkg_resources/__init__.py:2870: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('mpl_toolkits')`.
Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
declare_namespace(pkg)
/root/miniconda3/envs/webshop/lib/python3.8/site-packages/pkg_resources/__init__.py:121: DeprecationWarning: pkg_resources is deprecated as an API
warnings.warn("pkg_resources is deprecated as an API", DeprecationWarning)
/root/miniconda3/envs/webshop/lib/python3.8/site-packages/pkg_resources/__init__.py:2870: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('mpl_toolkits')`.
Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
declare_namespace(pkg)
/root/miniconda3/envs/webshop/lib/python3.8/site-packages/pkg_resources/__init__.py:121: DeprecationWarning: pkg_resources is deprecated as an API
warnings.warn("pkg_resources is deprecated as an API", DeprecationWarning)
/root/miniconda3/envs/webshop/lib/python3.8/site-packages/pkg_resources/__init__.py:2870: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('mpl_toolkits')`.
Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
declare_namespace(pkg)
/root/miniconda3/envs/webshop/lib/python3.8/site-packages/faiss/loader.py:28: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
if LooseVersion(numpy.__version__) >= "1.19":
/root/miniconda3/envs/webshop/lib/python3.8/site-packages/setuptools/_distutils/version.py:345: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
other = LooseVersion(other)
/root/miniconda3/envs/webshop/lib/python3.8/site-packages/faiss/loader.py:28: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
if LooseVersion(numpy.__version__) >= "1.19":
/root/miniconda3/envs/webshop/lib/python3.8/site-packages/setuptools/_distutils/version.py:345: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
other = LooseVersion(other)
/root/miniconda3/envs/webshop/lib/python3.8/site-packages/faiss/loader.py:28: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
if LooseVersion(numpy.__version__) >= "1.19":
/root/miniconda3/envs/webshop/lib/python3.8/site-packages/setuptools/_distutils/version.py:345: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
other = LooseVersion(other)
/root/miniconda3/envs/webshop/lib/python3.8/site-packages/thefuzz/fuzz.py:11: UserWarning: Using slow pure-python SequenceMatcher. Install python-Levenshtein to remove this warning
warnings.warn('Using slow pure-python SequenceMatcher. Install python-Levenshtein to remove this warning')
/root/miniconda3/envs/webshop/lib/python3.8/site-packages/thefuzz/fuzz.py:11: UserWarning: Using slow pure-python SequenceMatcher. Install python-Levenshtein to remove this warning
warnings.warn('Using slow pure-python SequenceMatcher. Install python-Levenshtein to remove this warning')
/root/miniconda3/envs/webshop/lib/python3.8/site-packages/thefuzz/fuzz.py:11: UserWarning: Using slow pure-python SequenceMatcher. Install python-Levenshtein to remove this warning
warnings.warn('Using slow pure-python SequenceMatcher. Install python-Levenshtein to remove this warning')
/root/miniconda3/envs/webshop/lib/python3.8/site-packages/gym/envs/registration.py:516: UserWarning: WARN: Overriding environment WebAgentSiteEnv-v0
logger.warn(f"Overriding environment {spec.id}")
/root/miniconda3/envs/webshop/lib/python3.8/site-packages/gym/envs/registration.py:516: UserWarning: WARN: Overriding environment WebAgentTextEnv-v0
logger.warn(f"Overriding environment {spec.id}")
> [Warning] Mind2Web task not available
> [Warning] KnowledgeGraph task not available
/root/miniconda3/envs/webshop/lib/python3.8/site-packages/gym/envs/registration.py:516: UserWarning: WARN: Overriding environment WebAgentSiteEnv-v0
logger.warn(f"Overriding environment {spec.id}")
/root/miniconda3/envs/webshop/lib/python3.8/site-packages/gym/envs/registration.py:516: UserWarning: WARN: Overriding environment WebAgentTextEnv-v0
logger.warn(f"Overriding environment {spec.id}")
> [Warning] Mind2Web task not available
> [Warning] KnowledgeGraph task not available
/root/miniconda3/envs/webshop/lib/python3.8/site-packages/gym/envs/registration.py:516: UserWarning: WARN: Overriding environment WebAgentSiteEnv-v0
logger.warn(f"Overriding environment {spec.id}")
/root/miniconda3/envs/webshop/lib/python3.8/site-packages/gym/envs/registration.py:516: UserWarning: WARN: Overriding environment WebAgentTextEnv-v0
logger.warn(f"Overriding environment {spec.id}")
> [Warning] Mind2Web task not available
> [Warning] KnowledgeGraph task not available
Products loaded.
Keys cleaned.
Attributes loaded.
9%|████████████▉ | 107308/1181436 [00:01<00:13, 79208.07it/s]Products loaded.
Keys cleaned.
66%|█████████████████████████████████████████████████████████████████████████████████████████████▋ | 779549/1181436 [00:17<00:05, 67288.15it/s]Attributes loaded.
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1181436/1181436 [00:30<00:00, 38834.25it/s]
66%|██████████████████████████████████████████████████████████████████████████████████████████████ | 782247/1181436 [00:17<00:06, 61574.56it/s]164 skipped
Loaded 12087 goals.
/root/miniconda3/envs/webshop/lib/python3.8/site-packages/flask/testing.py:71: DeprecationWarning: 'werkzeug.urls.url_parse' is deprecated and will be removed in Werkzeug 3.0. Use 'urllib.parse.urlsplit' instead.
url = url_parse(path)
/root/miniconda3/envs/webshop/lib/python3.8/site-packages/werkzeug/urls.py:545: DeprecationWarning: 'werkzeug.urls.URL' is deprecated and will be removed in Werkzeug 3.0. Use the 'urllib.parse' library instead.
return result_type(scheme, netloc, url, query, fragment)
/root/miniconda3/envs/webshop/lib/python3.8/site-packages/bs4/element.py:784: DeprecationWarning: The 'text' argument to find()-type methods is deprecated. Use 'string' instead.
warnings.warn(
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1181436/1181436 [00:30<00:00, 38764.74it/s]
164 skipped
Loaded 12087 goals.
/root/miniconda3/envs/webshop/lib/python3.8/site-packages/flask/testing.py:71: DeprecationWarning: 'werkzeug.urls.url_parse' is deprecated and will be removed in Werkzeug 3.0. Use 'urllib.parse.urlsplit' instead.
url = url_parse(path)
/root/miniconda3/envs/webshop/lib/python3.8/site-packages/werkzeug/urls.py:545: DeprecationWarning: 'werkzeug.urls.URL' is deprecated and will be removed in Werkzeug 3.0. Use the 'urllib.parse' library instead.
return result_type(scheme, netloc, url, query, fragment)
/root/miniconda3/envs/webshop/lib/python3.8/site-packages/bs4/element.py:784: DeprecationWarning: The 'text' argument to find()-type methods is deprecated. Use 'string' instead.
warnings.warn(
99%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████ | 79/80 [11:11<00:04, 4.05s/it]
Can you please help me see what's wrong, I'd appreciate it if you can replay.