You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
When the new process_list is shorter than op_record, Python does not raise an error for out-of-range indices but rather truncates to the maximum available length, thus len(prefix_process) < len(self.op_record). Similarly, the zip function terminates at the shorter iterable's length. This results in the check_ops_to_skip function incorrectly assuming that the recorded operators match the prefix of the current operators list.
As the title suggests, the issue lies in the following code.
data-juicer/data_juicer/utils/ckpt_utils.py
Lines 86 to 101 in c85e024
When the new
process_list
is shorter thanop_record
, Python does not raise an error for out-of-range indices but rather truncates to the maximum available length, thuslen(prefix_process) < len(self.op_record)
. Similarly, thezip
function terminates at the shorter iterable's length. This results in thecheck_ops_to_skip
function incorrectly assuming that the recorded operators match the prefix of the current operators list.Is that the case? @HYLcool @yxdyc
The text was updated successfully, but these errors were encountered: