-
Notifications
You must be signed in to change notification settings - Fork 150
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
无法正常启动,访问task会报错 #53
Comments
试试使用python3.9?报错是 |
十分感谢您的回答 会一直在这个状态,没有继续执行,一个下午都没有后续内容了 我是在wsl2+ubuntu22.04上运行的,这个影响吗? |
试试直接 build 能不能成功? docker build -f data/os_interaction/res/dockerfiles/default data/os_interaction/res/dockerfiles
docker build -f data/os_interaction/res/dockerfiles/packages data/os_interaction/res/dockerfiles
docker build -f data/os_interaction/res/dockerfiles/ubuntu data/os_interaction/res/dockerfiles |
十分感谢您的回复,有空我试一下,给您回复,最近比较忙
…---原始邮件---
发件人: "Hao ***@***.***>
发送时间: 2023年10月18日(周三) 中午1:15
收件人: ***@***.***>;
抄送: ***@***.******@***.***>;
主题: Re: [THUDM/AgentBench] 无法正常启动,访问task会报错 (Issue #53)
试试直接 build 能不能成功?
docker build -f data/os_interaction/res/dockerfiles/default data/os_interaction/res/dockerfiles docker build -f data/os_interaction/res/dockerfiles/packages data/os_interaction/res/dockerfiles docker build -f data/os_interaction/res/dockerfiles/ubuntu data/os_interaction/res/dockerfiles
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you authored the thread.Message ID: ***@***.***>
|
可以build成功 |
我发现并没有alfworld的和docker进行build的文件 |
只有os需要提前build几个image,如果想跑alfworld的话,可以先docker pull longinyu/agentbench-alfworld,然后按照教程操作。 |
好的,谢谢您 docker: permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Post "http://%2Fvar%2Frun%2Fdocker.sock/v1.24/containers/create": dial unix /var/run/docker.sock: connect: permission denied. 貌似是我的docker容器的权限问题 |
是的,需要用一个有运行docker的权限的账户 |
我已经解决了,在stark_task.py文件下加入sudo即可,以下是我改的内容: |
👍 |
您能帮我解释一下, alfworld 的 json 文件吗?{ |
我已经查看到了,是在里面,谢谢~ |
在alfworld-std任务中,在task.py文件 出现很多重复的内容 Here is one example. \nHere is one example. \nHere is one example. \nHere is one example. \nHere is one example. \nHere is one example. \nHere is one example. \nHere is one example. \nHere is one example. \nHere is one example. \nHere is one example. \nHere is one example. 应该是做的few react shot吧? |
感谢提醒,现已修复。 |
在data中,dev和std的区别是什么呢?dev是原本数据的测试集吗? |
dev 和 std(或者叫 test)的主要区别是数据数量不同(部分任务分布也会有些差异)。 |
也就是说,repo的论文中的测试结果,都是从std里面的数据进行测试出来的,dev适合短时间内进行测试性能提升,然后两者的数据内容不是包含与被包含的关系。感谢,有机会请大佬恰饭~ |
是的。 感谢对我们工作的兴趣,欢迎持续关注!:sparkles: |
当我准备测试os的数据的适合,出现了以下内容: Traceback (most recent call last): 但是HH的测试是正常可以运行了 |
感谢提醒,config 中 os-dev 漏了 name 字段,现在已经修复 |
TaskClient created: os-std (http://localhost:5000/api) python -m src.start_task -a |
问题应该是这样的:assigner试图去找os-std task,没有找到,也许你刚刚启动的是os-dev的服务而不是os-std的服务?可以检查一下configs/start_task.yaml里定义的是否是默认的os-std? |
是的,我都检查过,是我设置的os-std ,都是对应的 |
我会将他的config打印出来,一一检查的。比如我刚才又测了一下: 这是python -m src.assigner Warning: 7 agent(s) and 15 task(s) are defined but not used, they will be ignored. |
是否可以将你 configs/start_task.yaml 中的内容复制粘贴看一下? |
definition: start: |
这个文件是 server 端的配置,start 字段下需要包含 os-std,例如: definition:
import: tasks/task_assembly.yaml
start:
alfworld-dev: 2
os-std: 5 上述配置文件将会启动共计 7 个 task worker(2 个 alfworld-dev 和 5 个 os-std) |
是的,这个我理解,我也是这么设置的。但是却无法找到该task,意外的是alfworld-std 是正常可以测评的 |
重新启动 task server 试试(就是使用 |
是的,我重复做了几次,都是如此,依旧检测不到alfworld-dev这个task INFO: 127.0.0.1:57618 - "GET /api/get_indices?name=alfworld-dev HTTP/1.1" 400 Bad Request |
目前os-dev和os-std是正常可以进行评测了 |
问题找到了,一般如果我开比较大的下载,会导致,部分进程无法绑定ip地址,所以就没有启动成功,给下载数据的东西关掉就好了,主要还是网络拥堵的问题。。。。 |
|
这是 one-shot prompt,每条数据前面均有一条示例,succeed 之后的才是正式的问题,任务的具体定义详见论文。 |
明白了,谢谢您~ |
我们最新的版本应该修复了这个问题。如果只是偶尔出现的话可以忽略。 |
我应该用的是最新的版本V2,并不是偶尔出现,目前只是在cg-std出现了(启动三次,出现三次),dbench,HH,OS都还没有出现过。 |
要不您尝试git pull一下main上最新的修改?我们在a069c7里尝试了修复这一问题。 |
INTERACT_FAILED 一般是什么原因导致的呢? 交互失败,是不是,output无法识别出action 和thought? |
一般来讲INTERACT_FAILED是在交互过程中发生了不可恢复的错误,这通常是意料之外的,例如网络中断,worker断连等。如果是output无法识别出action和thought一般task会有自己的处理方式,不会导致FAILED。 |
谢谢您的回复,一般出现的问题都已经可以解决了。你们是否也对agentlm-13B、6B(huggingface上 上传的模型)进行测评了呢?我目前测了一下50个里面只对了5个,不知道我是什么地方配置错误了吗?同样条件下,gpt4可以跑到84% |
Hi,AgentLM的测试数据应该可以在AgentLM的repo中找到,我们没有重复对其进行测试,也许您可以在他们的repo里提一个issue。 |
我发现用70b模型进行推理十分缓慢,想着可以将需要测试的数据集分成4份,然后启动4个model-work来进行推理,这样可以增加推理的效率。期待可以将这个功能加上去 |
您好,我也遇到了这个问题,但我没理解您这个是怎么解决的 |
我这里发现如果测试的任务需要启动docker镜像就会出现找不到task的情况,例如 |
自动启动,需要先下载docker |
docker的镜像是有拉取的。 docker image ls |
是的,没有错误,还需要更改 configs/start_task.yaml start: |
嗯嗯,我也是这样搞的。。 definition: start: |
Hi @Dhaizei ,目前我们推荐的做法是设一个model worker的转发服务器(如Fastchat中的controller),然后将agent的concurrency设置成4 |
请问您解决了吗?我遇到了和您一样的问题 |
INFO: 127.0.0.1:45654 - "GET /api/get_indices?name=dbbench-std HTTP/1.1" 200 OK
INFO: 127.0.0.1:45656 - "GET /api/get_indices?name=os-std HTTP/1.1" 400 Bad Request
在python -m src.start_task -a 后(未进行任何改动配置)
<class 'src.server.tasks.os_interaction.task.OSInteraction'>
Traceback (most recent call last):
File "/root/anaconda3/envs/py38/lib/python3.8/runpy.py", line 192, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/root/anaconda3/envs/py38/lib/python3.8/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/root/work/AgentBenchV0.2/src/server/task_worker.py", line 256, in
asyncio_task = InstanceFactory.parse_obj(conf[args.name]).create()
File "/root/work/AgentBenchV0.2/src/typings/general.py", line 37, in create
return getattr(mod, self.module.split(".")[-1])(**self.parameters)
File "/root/work/AgentBenchV0.2/src/server/tasks/os_interaction/task.py", line 275, in init
+ os.path.basename(file)
AttributeError: 'str' object has no attribute 'removesuffix'
/root/anaconda3/envs/py38/lib/python3.8/site-packages/requests/init.py:109: RequestsDependencyWarning: urllib3 (2.0.5) or chardet (3.0.4)/charset_normalizer (3.2.0) doesn't match a supported version!
warnings.warn(
/root/anaconda3/envs/py38/lib/python3.8/site-packages/requests/init.py:109: RequestsDependencyWarning: urllib3 (2.0.5) or chardet (3.0.4)/charset_normalizer (3.2.0) doesn't match a supported version!
warnings.warn(
/root/anaconda3/envs/py38/lib/python3.8/site-packages/requests/init.py:109: RequestsDependencyWarning: urllib3 (2.0.5) or chardet (3.0.4)/charset_normalizer (3.2.0) doesn't match a supported version!
warnings.warn(
/root/anaconda3/envs/py38/lib/python3.8/site-packages/requests/init.py:109: RequestsDependencyWarning: urllib3 (2.0.5) or chardet (3.0.4)/charset_normalizer (3.2.0) doesn't match a supported version!
warnings.warn(
<module 'src.server.tasks.os_interaction' from '/root/work/AgentBenchV0.2/src/server/tasks/os_interaction/init.py'> src.server.tasks.os_interaction.OSInteraction
<class 'src.server.tasks.os_interaction.task.OSInteraction'>
Traceback (most recent call last):
File "/root/anaconda3/envs/py38/lib/python3.8/runpy.py", line 192, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/root/anaconda3/envs/py38/lib/python3.8/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/root/work/AgentBenchV0.2/src/server/task_worker.py", line 256, in
asyncio_task = InstanceFactory.parse_obj(conf[args.name]).create()
File "/root/work/AgentBenchV0.2/src/typings/general.py", line 37, in create
return getattr(mod, self.module.split(".")[-1])(**self.parameters)
File "/root/work/AgentBenchV0.2/src/server/tasks/os_interaction/task.py", line 275, in init
+ os.path.basename(file)
AttributeError: 'str' object has no attribute 'removesuffix'
<module 'src.server.tasks.os_interaction' from '/root/work/AgentBenchV0.2/src/server/tasks/os_interaction/init.py'> src.server.tasks.os_interaction.OSInteraction
<class 'src.server.tasks.os_interaction.task.OSInteraction'>
Traceback (most recent call last):
File "/root/anaconda3/envs/py38/lib/python3.8/runpy.py", line 192, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/root/anaconda3/envs/py38/lib/python3.8/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/root/work/AgentBenchV0.2/src/server/task_worker.py", line 256, in
asyncio_task = InstanceFactory.parse_obj(conf[args.name]).create()
File "/root/work/AgentBenchV0.2/src/typings/general.py", line 37, in create
return getattr(mod, self.module.split(".")[-1])(**self.parameters)
File "/root/work/AgentBenchV0.2/src/server/tasks/os_interaction/task.py", line 275, in init
+ os.path.basename(file)
AttributeError: 'str' object has no attribute 'removesuffix'
/root/anaconda3/envs/py38/lib/python3.8/site-packages/requests/init.py:109: RequestsDependencyWarning: urllib3 (2.0.5) or chardet (3.0.4)/charset_normalizer (3.2.0) doesn't match a supported version!
warnings.warn(
/root/anaconda3/envs/py38/lib/python3.8/site-packages/requests/init.py:109: RequestsDependencyWarning: urllib3 (2.0.5) or chardet (3.0.4)/charset_normalizer (3.2.0) doesn't match a supported version!
warnings.warn(
<module 'src.server.tasks.os_interaction' from '/root/work/AgentBenchV0.2/src/server/tasks/os_interaction/init.py'> src.server.tasks.os_interaction.OSInteraction
<class 'src.server.tasks.os_interaction.task.OSInteraction'>
Traceback (most recent call last):
File "/root/anaconda3/envs/py38/lib/python3.8/runpy.py", line 192, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/root/anaconda3/envs/py38/lib/python3.8/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/root/work/AgentBenchV0.2/src/server/task_worker.py", line 256, in
asyncio_task = InstanceFactory.parse_obj(conf[args.name]).create()
File "/root/work/AgentBenchV0.2/src/typings/general.py", line 37, in create
return getattr(mod, self.module.split(".")[-1])(**self.parameters)
File "/root/work/AgentBenchV0.2/src/server/tasks/os_interaction/task.py", line 275, in init
+ os.path.basename(file)
AttributeError: 'str' object has no attribute 'removesuffix'
<module 'src.server.tasks.os_interaction' from '/root/work/AgentBenchV0.2/src/server/tasks/os_interaction/init.py'> src.server.tasks.os_interaction.OSInteraction
<class 'src.server.tasks.os_interaction.task.OSInteraction'>
Traceback (most recent call last):
File "/root/anaconda3/envs/py38/lib/python3.8/runpy.py", line 192, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/root/anaconda3/envs/py38/lib/python3.8/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/root/work/AgentBenchV0.2/src/server/task_worker.py", line 256, in
asyncio_task = InstanceFactory.parse_obj(conf[args.name]).create()
File "/root/work/AgentBenchV0.2/src/typings/general.py", line 37, in create
return getattr(mod, self.module.split(".")[-1])(**self.parameters)
File "/root/work/AgentBenchV0.2/src/server/tasks/os_interaction/task.py", line 275, in init
+ os.path.basename(file)
python -m src.assigner 后
访问os-std就会报错
<class 'src.client.task.TaskClient'>
TaskClient created: os-std (http://localhost:5000/api)
Traceback (most recent call last):
File "/root/anaconda3/envs/py38/lib/python3.8/runpy.py", line 192, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/root/anaconda3/envs/py38/lib/python3.8/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/root/work/AgentBenchV0.2/src/assigner.py", line 402, in
Assigner(value, args.retry).start()
File "/root/work/AgentBenchV0.2/src/assigner.py", line 74, in init
self.task_indices[task] = self.tasks[task].get_indices()
File "/root/work/AgentBenchV0.2/src/client/task.py", line 31, in get_indices
raise AgentBenchException(result.text, result.status_code, self.name)
src.typings.exception.AgentBenchException: ('{"detail":"Error: Task does not exist"}', 400, 'os-std')
The text was updated successfully, but these errors were encountered: