checking existence of file before copy

[wj-Mcat]

PR types

Bug fixes

PR changes

Models

Description

try to fix: #2577

[guoshengCS]

The other tokenizers' failure caused by auto-saving in from_pretrained also need fix.

[wj-Mcat]

After update the save_resouce method, there are only three tokenizers will occur the error.

• error table

+-----------------------+------------------------------------------------------+----------+
| bert-class            | resouce                                              | result   |
+=======================+======================================================+==========+
| BertJapaneseTokenizer | cl-tohoku/bert-base-japanese                         | False    |
+-----------------------+------------------------------------------------------+----------+
| BertJapaneseTokenizer | cl-tohoku/bert-base-japanese-whole-word-masking      | False    |
+-----------------------+------------------------------------------------------+----------+
| BertJapaneseTokenizer | cl-tohoku/bert-base-japanese-char                    | False    |
+-----------------------+------------------------------------------------------+----------+
| BertJapaneseTokenizer | cl-tohoku/bert-base-japanese-char-whole-word-masking | False    |
+-----------------------+------------------------------------------------------+----------+
| ChineseBertTokenizer  | ChineseBERT-base                                     | False    |
+-----------------------+------------------------------------------------------+----------+
| ErnieTokenizer        | rocketqa-v1-marco-query-encoder                      | False    |
+-----------------------+------------------------------------------------------+----------+
| ErnieTokenizer        | rocketqa-v1-marco-para-encoder                       | False    |
+-----------------------+------------------------------------------------------+----------+
| ErnieTokenizer        | rocketqa-v1-marco-cross-encoder                      | False    |
+-----------------------+------------------------------------------------------+----------+

• BertJapaneseTokenzier and LukeTokenizer require additional package to make it run.
• ChineseBertTokenizer raise an exception in initialized stage. you can refer to ChineseBertTokenizer got an unexpected keyword argument 'tokenizer_class' #2584
• ErnieTokenizer lose the vocab_file: https://bj.bcebos.com/paddlenlp/models/transformers/rocketqa/rocketqa-v1-marco-vocab.txt

[wj-Mcat]

Please take some time to review this PR. @guoshengCS

[guoshengCS]

LGTM

[guoshengCS]

After update the save_resouce method, there are only three tokenizers will occur the error.

• error table

+-----------------------+------------------------------------------------------+----------+
| bert-class            | resouce                                              | result   |
+=======================+======================================================+==========+
| BertJapaneseTokenizer | cl-tohoku/bert-base-japanese                         | False    |
+-----------------------+------------------------------------------------------+----------+
| BertJapaneseTokenizer | cl-tohoku/bert-base-japanese-whole-word-masking      | False    |
+-----------------------+------------------------------------------------------+----------+
| BertJapaneseTokenizer | cl-tohoku/bert-base-japanese-char                    | False    |
+-----------------------+------------------------------------------------------+----------+
| BertJapaneseTokenizer | cl-tohoku/bert-base-japanese-char-whole-word-masking | False    |
+-----------------------+------------------------------------------------------+----------+
| ChineseBertTokenizer  | ChineseBERT-base                                     | False    |
+-----------------------+------------------------------------------------------+----------+
| ErnieTokenizer        | rocketqa-v1-marco-query-encoder                      | False    |
+-----------------------+------------------------------------------------------+----------+
| ErnieTokenizer        | rocketqa-v1-marco-para-encoder                       | False    |
+-----------------------+------------------------------------------------------+----------+
| ErnieTokenizer        | rocketqa-v1-marco-cross-encoder                      | False    |
+-----------------------+------------------------------------------------------+----------+

• BertJapaneseTokenzier and LukeTokenizer require additional package to make it run.
• ChineseBertTokenizer raise an exception in initialized stage. you can refer to ChineseBertTokenizer got an unexpected keyword argument 'tokenizer_class' #2584
• ErnieTokenizer lose the vocab_file: https://bj.bcebos.com/paddlenlp/models/transformers/rocketqa/rocketqa-v1-marco-vocab.txt

Get it. Thanks