Running tokenizer on dataset (num_proc=16): 0%| | 0/1000 [00:00<?, ? examples/s]Process ForkPoolWorker-18:
Traceback (most recent call last):
File "/mnt/bn/lesqa-quality-data-yg/project/anaconda3/envs/llamafactory/lib/python3.11/site-packages/multiprocess/process.py", line 314, in _bootstrap
self.run()
File "/mnt/bn/lesqa-quality-data-yg/project/anaconda3/envs/llamafactory/lib/python3.11/site-packages/multiprocess/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/mnt/bn/lesqa-quality-data-yg/project/anaconda3/envs/llamafactory/lib/python3.11/site-packages/multiprocess/pool.py", line 114, in worker
task = get()
^^^^^
File "/mnt/bn/lesqa-quality-data-yg/project/anaconda3/envs/llamafactory/lib/python3.11/site-packages/multiprocess/queues.py", line 370, in get
return _ForkingPickler.loads(res)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/mnt/bn/lesqa-quality-data-yg/project/anaconda3/envs/llamafactory/lib/python3.11/site-packages/dill/_dill.py", line 303, in loads
return load(file, ignore, **kwds)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/mnt/bn/lesqa-quality-data-yg/project/anaconda3/envs/llamafactory/lib/python3.11/site-packages/dill/_dill.py", line 289, in load
return Unpickler(file, ignore=ignore, **kwds).load()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/mnt/bn/lesqa-quality-data-yg/project/anaconda3/envs/llamafactory/lib/python3.11/site-packages/dill/_dill.py", line 444, in load
obj = StockUnpickler.load(self)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/mnt/bn/lesqa-quality-data-yg/project/anaconda3/envs/llamafactory/lib/python3.11/site-packages/dill/_dill.py", line 593, in _create_type
return typeobj(*args)
^^^^^^^^^^^^^^
File "/mnt/bn/lesqa-quality-data-yg/project/anaconda3/envs/llamafactory/lib/python3.11/typing.py", line 2992, in __new__
raise TypeError('cannot inherit from both a TypedDict type '
TypeError: cannot inherit from both a TypedDict type and a non-TypedDict base class
Running tokenizer on dataset (num_proc=16): 0%| | 0/1000 [00:02<?, ? examples/s]
[rank0]: Traceback (most recent call last):
[rank0]: File "/mnt/bn/iesqa-moe-teenager-data-yg/chenzixuan/LLaMA-Factory/src/llamafactory/launcher.py", line 23, in <module>
[rank0]: launch()
[rank0]: File "/mnt/bn/iesqa-moe-teenager-data-yg/chenzixuan/LLaMA-Factory/src/llamafactory/launcher.py", line 19, in launch
[rank0]: run_exp()
[rank0]: File "/mnt/bn/iesqa-moe-teenager-data-yg/chenzixuan/LLaMA-Factory/src/llamafactory/train/tuner.py", line 110, in run_exp
[rank0]: _training_function(config={"args": args, "callbacks": callbacks})
[rank0]: File "/mnt/bn/iesqa-moe-teenager-data-yg/chenzixuan/LLaMA-Factory/src/llamafactory/train/tuner.py", line 72, in _training_function
[rank0]: run_sft(model_args, data_args, training_args, finetuning_args, generating_args, callbacks)
[rank0]: File "/mnt/bn/iesqa-moe-teenager-data-yg/chenzixuan/LLaMA-Factory/src/llamafactory/train/sft/workflow.py", line 51, in run_sft
[rank0]: dataset_module = get_dataset(template, model_args, data_args, training_args, stage="sft", **tokenizer_module)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/mnt/bn/iesqa-moe-teenager-data-yg/chenzixuan/LLaMA-Factory/src/llamafactory/data/loader.py", line 315, in get_dataset
[rank0]: dataset = _get_preprocessed_dataset(
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/mnt/bn/iesqa-moe-teenager-data-yg/chenzixuan/LLaMA-Factory/src/llamafactory/data/loader.py", line 256, in _get_preprocessed_dataset
[rank0]: dataset = dataset.map(
[rank0]: ^^^^^^^^^^^^
[rank0]: File "/mnt/bn/lesqa-quality-data-yg/project/anaconda3/envs/llamafactory/lib/python3.11/site-packages/datasets/arrow_dataset.py", line 557, in wrapper
[rank0]: out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/mnt/bn/lesqa-quality-data-yg/project/anaconda3/envs/llamafactory/lib/python3.11/site-packages/datasets/arrow_dataset.py", line 3171, in map
[rank0]: for rank, done, content in iflatmap_unordered(
[rank0]: File "/mnt/bn/lesqa-quality-data-yg/project/anaconda3/envs/llamafactory/lib/python3.11/site-packages/datasets/utils/py_utils.py", line 721, in iflatmap_unordered
[rank0]: raise RuntimeError(
[rank0]: RuntimeError: One of the subprocesses has abruptly died during map operation.To debug the error, disable multiprocessing.
[rank0]:[W826 17:05:49.980127574 ProcessGroupNCCL.cpp:1250] Warning: WARNING: process group has NOT been destroyed before we destruct ProcessGroupNCCL. On normal program exit, the application should call destroy_process_group to ensure that any pending NCCL operations have finished in this process. In rare cases this process can exit before this point and block the progress of another member of the process group. This constraint has always been present, but this warning has only been added since PyTorch 2.4 (function operator())
Error