preprocess_dataset dataset.map crashed with TypeError: cannot pickle 'builtins.CoreBPE' object

[songkq]

During preprocess_dataset mapping the dataset, it crashed. Could you please give some advice?

Spawning 32 processes
Running tokenizer on dataset (num_proc=32):   0%|          | 0/11694 [00:00<?, ? examples/s]

[hiyouga]

use preprocessing_num_workers=1

[songkq]

@hiyouga preprocessing_num_workers=1 can work but a little slow.
I'm confusing why preprocessing_num_workers cannot be set to be greater than 1 with the commit 2780792 .
Since preprocessing_num_workers > 1 can work well with this commit 513e1f1.

[hiyouga]

This problem is related to the tiktoken tokenizer, looks like you are using the Qwen-7B model.

Related issue: huggingface/datasets#5536 huggingface/datasets#5769

[songkq]

@hiyouga Thanks.

[songkq]

使用替代方案GPT2Tokenizer支持多线程：https://huggingface.co/vonjack/Qwen-LLaMAfied-HFTok-7B-Chat/tree/main