Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
在数据集处理时,如果训练集中
input
过长,超过了max_length
,这种数据虽然在tokenizer
部分被标记成了-100
,但仍然会参与向前计算和损失函数的计算,这会浪费计算资源,因此写了一个filter_out_all_negative_labels
函数来过滤掉负标签的样本。这样使得负样本不会参与向前传播的计算和损失函数的计算。During dataset processing, if the
input
in the training set is too long and exceedsmax_length
, such data, although marked as-100
in the tokenizer section, will still participate in the forward computation and loss function calculation, which can waste computational resources. Therefore, a function namedfilter_out_all_negative_labels
was written to filter out samples with negative labels. This ensures that samples with negative labels do not participate in forward propagation and loss function calculations.