-
Notifications
You must be signed in to change notification settings - Fork 20
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
其他数据集训练出错 #5
Comments
|
好的,感谢您的指导
…------------------ 原始邮件 ------------------
发件人: "Coopercoppers/PFN" ***@***.***>;
发送时间: 2021年10月30日(星期六) 下午4:02
***@***.***>;
***@***.******@***.***>;
主题: Re: [Coopercoppers/PFN] 其他数据集训练出错 (Issue #5)
rc_head_label应该是subj_head, obj_head, relation。你这上边写的1,1是subj_head, subj_tail,这是不对的。
不需要对实体进行类型划分的就设成None
这个可能是你的句子tokenize之后长度超过了512,检查下你的数据,看看哪个超了。
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or unsubscribe.
Triage notifications on the go with GitHub Mobile for iOS or Android.
|
不好意思再次打扰您。
我按照您说的检查出错的句子的长度,发现我一个批次的句子都没有超过128的,但是tokenizer(x)后num_tokens=588,确实大于512.
我不是很明白tokenizer的原理,好像我将批次降低之后他仍然会超过。
这个bug改了一下午了,很是苦恼,烦请您指导一下该从哪里改,谢谢
…------------------ 原始邮件 ------------------
发件人: "Coopercoppers/PFN" ***@***.***>;
发送时间: 2021年10月30日(星期六) 下午4:24
***@***.***>;
***@***.******@***.***>;
主题: Re: [Coopercoppers/PFN] 其他数据集训练出错 (Issue #5)
Closed #5.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or unsubscribe.
Triage notifications on the go with GitHub Mobile for iOS or Android.
|
你把tokenizer后长度为588的那个句子拿出来打印一下再分析。 |
这个我之前已经试过了,很奇怪的是,我的tokenizer()是对一个字符list进行的,最后得到了一个一维的输出,就是这个588的向量,在我的理解中,tokenizer应该是对每个句子进行的吧。。。
可能之前的数据处理还是有问题,谢谢您的回复
…------------------ 原始邮件 ------------------
发件人: "Coopercoppers/PFN" ***@***.***>;
发送时间: 2021年10月30日(星期六) 晚上6:07
***@***.***>;
***@***.******@***.***>;
主题: Re: [Coopercoppers/PFN] 其他数据集训练出错 (Issue #5)
你把tokenizer后长度为588的那个句子拿出来打印一下再分析。
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or unsubscribe.
Triage notifications on the go with GitHub Mobile for iOS or Android.
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
我按照您对数据的处理(nytAndWebnlg),应用在其他数据集上:
我猜测您dataloader之中
` subj = entity[subj_idx]
的含义是实体变成
[1, 1, 'None', 16, 20, 'None'],两个数字是实体的单词起始和结束下标,None是类型 关系是:
rc_head_labels = [1, 1, '/location/location/contains],rc_tail_labels= [16, 20, '/location/location/contains]`,即头实体和尾实体的单词下标对和关系类型。我想请教的是:
Process finished with exit code 1
The text was updated successfully, but these errors were encountered: