fix uie dataloader memory overflow #3381
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
linjieccc
reviewed
Sep 28, 2022
model_zoo/uie/utils.py
Outdated
| @@ -229,8 +229,7 @@ def reader(data_path, max_seq_len=512): | |||
| cur_result_list = [] | |||
|
|
|||
| for result in result_list: | |||
| if result['start'] + 1 <= max_content_len < result[ | |||
| 'end']: | |||
| if result['start'] + 1 <= max_content_len < result['end'] and result['end'] - result['start'] <= max_content_len : | |||
There was a problem hiding this comment.
对于抽取目标超过最大长度限制result['end'] - result['start']的情况如果能给个warning的信息可能更好一些
There was a problem hiding this comment.
最新的commit已添加相关warning
westfish
changed the title
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
PR types
Bug fixes
PR changes
Models
Description
分析了一下bug出现的原因：
当输入过长时，在数据加载时reader会对输入的实例进行分割，返回多个实例， 正常情况max_content_len是不会变化的，它的作用是分割content为两部分， 即cur_content = content[:max_content_len] res_content = content[max_content_len:] 。
但是，当待抽取对象result的start id小于max_content_len且end id小于max_content_len，那么待抽取result就将位于cur_content和res_content两个地方，所以为了让整个result位于同一个地方，就会有一个特殊处理，也即进入if result['start'] + 1 <= max_content_len < result[ 'end']语句， 并设置max_content_len = result['start']，从而让待抽取对象位于新实例的开始部分。
但是问题是，如果待抽取对象result过长，超过了max_content_len，即使执行了上一步骤之后，更新后的start id和end id仍然会位于两个不同的地方，就会再次进入if result['start'] + 1 <= max_content_len < result[ 'end']语句，从而无限循环，得到无限多的空实例，最终内存溢出。
所以结论就是，要考虑到待抽取对象result会出现超过max_content_len的情况，在这种情况下，我们是无法得到一个可以包含待抽取对象result的实例的，即对重制max_content_len对条件增加限制，也即if result['start'] + 1 <= max_content_len < result['end'] and result['end']-result['start'] <= max_content_len 。
经过bad case测试，增加测试后不再出现内存溢出的问题。