ValueError: char_to_token() is not available when using Python based tokenizers #88
Comments
|
env ： Name Version Build Channel |
{{ message }}
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
error log：
Namespace(aux_loss=True, backbone='resnet101', batch_size=4, bbox_loss_coef=5, ce_loss_coef=1, clevr_ann_path='', clevr_img_path='', clip_max_norm=0.1, coco_path='/data_SSD1/lhxiao/transvg/ln_data/other/images/mscoco/images/', combine_datasets=['refexp'], combine_datasets_val=['refexp'], contrastive_align_loss=True, contrastive_align_loss_coef=1, contrastive_loss=False, contrastive_loss_coef=0.1, contrastive_loss_hdim=64, dataset_config='configs/refcoco.json', dec_layers=6, device='cuda', dice_loss_coef=1, dilation=False, dim_feedforward=2048, dist_backend='nccl', dist_url='env://', distributed=True, do_qa=False, dropout=0.1, ema=True, ema_decay=0.9998, enc_layers=6, eos_coef=0.1, epoch_chunks=-1, epochs=5, eval=False, eval_skip=1, fraction_warmup_steps=0.01, freeze_text_encoder=False, frozen_weights=None, giou_loss_coef=2, gpu=0, hidden_dim=256, load='/data_SSD1/lhxiao/mdetr/checkpoint/pretrain/pretrained_resnet101_checkpoint.pth', lr=5e-05, lr_backbone=1e-05, lr_drop=3, mask_loss_coef=1, mask_model='none', masks=False, modulated_lvis_ann_path='', nheads=8, no_detection=False, num_queries=100, num_workers=2, optimizer='adam', output_dir='/data_SSD1/lhxiao/mdetr/output/v01', pass_pos_and_query=True, phrasecut_ann_path='', phrasecut_orig_ann_path='', position_embedding='sine', pre_norm=False, predict_final=False, qa_loss_coef=1, rank=0, refexp_ann_path='/data_SSD1/lhxiao/mdetr/mdetr_annotations/OpenSource/', refexp_dataset_name='refcoco', remove_difficult=False, resume='', run_name='', schedule='linear_with_warmup', seed=42, set_cost_bbox=5, set_cost_class=1, set_cost_giou=2, set_loss='hungarian', split_qa_heads=False, start_epoch=0, temperature_NCE=0.07, test=False, test_type='test', text_encoder_lr=1e-05, text_encoder_type='roberta-base', vg_ann_path='', vg_img_path='', weight_decay=0.0001, world_size=2)
Some weights of the model checkpoint at roberta-base were not used when initializing RobertaModel: ['lm_head.layer_norm.weight', 'lm_head.layer_norm.bias', 'lm_head.dense.bias', 'lm_head.bias', 'lm_head.decoder.weight', 'lm_head.dense.weight']
Some weights of the model checkpoint at roberta-base were not used when initializing RobertaModel: ['lm_head.bias', 'lm_head.layer_norm.bias', 'lm_head.decoder.weight', 'lm_head.dense.bias', 'lm_head.layer_norm.weight', 'lm_head.dense.weight']
number of params: 185160324
loading annotations into memory...
Done (t=1.61s)
creating index...
index created!
loading annotations into memory...
Done (t=0.09s)
creating index...
index created!
loading from /data_SSD1/lhxiao/mdetr/checkpoint/pretrain/pretrained_resnet101_checkpoint.pth
Start training
Starting epoch 0
/home/mmc_xiaolinhui/mmc_226_exp_202206/mdetr/models/position_encoding.py:41: UserWarning: floordiv is deprecated, and its behavior will change in a future version of pytorch. It currently rounds toward 0 (like the 'trunc' function NOT 'floor'). This results in incorrect rounding for negative values. To keep the current behavior, use torch.div(a, b, rounding_mode='trunc'), or for actual floor division, use torch.div(a, b, rounding_mode='floor').
dim_t = self.temperature ** (2 * (dim_t // 2) / self.num_pos_feats)
/home/mmc_xiaolinhui/mmc_226_exp_202206/mdetr/models/position_encoding.py:41: UserWarning: floordiv is deprecated, and its behavior will change in a future version of pytorch. It currently rounds toward 0 (like the 'trunc' function NOT 'floor'). This results in incorrect rounding for negative values. To keep the current behavior, use torch.div(a, b, rounding_mode='trunc'), or for actual floor division, use torch.div(a, b, rounding_mode='floor').
dim_t = self.temperature ** (2 * (dim_t // 2) / self.num_pos_feats)
Traceback (most recent call last):
File "main.py", line 591, in
main(args)
File "main.py", line 494, in main
train_stats = train_one_epoch(
File "/home/mmc_xiaolinhui/mmc_226_exp_202206/mdetr/engine.py", line 73, in train_one_epoch
loss_dict.update(criterion(outputs, targets, positive_map))
File "/home/mmc_xiaolinhui/anaconda3/envs/mdetr_env/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1110, in _call_impl
return forward_call(*input, **kwargs)
File "/home/mmc_xiaolinhui/mmc_226_exp_202206/mdetr/models/mdetr.py", line 679, in forward
losses.update(self.get_loss(loss, outputs, targets, positive_map, indices, num_boxes))
File "/home/mmc_xiaolinhui/mmc_226_exp_202206/mdetr/models/mdetr.py", line 655, in get_loss
return loss_map[loss](outputs, targets, positive_map, indices, num_boxes, **kwargs)
File "/home/mmc_xiaolinhui/mmc_226_exp_202206/mdetr/models/mdetr.py", line 518, in loss_contrastive_align
beg_pos = tokenized.char_to_token(i, beg)
File "/home/mmc_xiaolinhui/anaconda3/envs/mdetr_env/lib/python3.8/site-packages/transformers/tokenization_utils_base.py", line 547, in char_to_token
raise ValueError("char_to_token() is not available when using Python based tokenizers")
ValueError: char_to_token() is not available when using Python based tokenizers
Traceback (most recent call last):
File "main.py", line 591, in
main(args)
**File "main.py", line 494, in main
train_stats = train_one_epoch(
File "/home/mmc_xiaolinhui/mmc_226_exp_202206/mdetr/engine.py", line 73, in train_one_epoch
loss_dict.update(criterion(outputs, targets, positive_map))
File "/home/mmc_xiaolinhui/anaconda3/envs/mdetr_env/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1110, in _call_impl
return forward_call(*input, **kwargs)
File "/home/mmc_xiaolinhui/mmc_226_exp_202206/mdetr/models/mdetr.py", line 679, in forward
losses.update(self.get_loss(loss, outputs, targets, positive_map, indices, num_boxes))
File "/home/mmc_xiaolinhui/mmc_226_exp_202206/mdetr/models/mdetr.py", line 655, in get_loss
return loss_map[loss](outputs, targets, positive_map, indices, num_boxes, kwargs)
File "/home/mmc_xiaolinhui/mmc_226_exp_202206/mdetr/models/mdetr.py", line 518, in loss_contrastive_align
beg_pos = tokenized.char_to_token(i, beg)
File "/home/mmc_xiaolinhui/anaconda3/envs/mdetr_env/lib/python3.8/site-packages/transformers/tokenization_utils_base.py", line 547, in char_to_token
raise ValueError("char_to_token() is not available when using Python based tokenizers")
ValueError: char_to_token() is not available when using Python based tokenizers
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 3789675 closing signal SIGTERM
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 3789672) of binary: /home/mmc_xiaolinhui/anaconda3/envs/mdetr_env/bin/python
env ：
Name Version Build Channel
_libgcc_mutex 0.1 conda_forge https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge
_openmp_mutex 4.5 2_gnu https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge
bzip2 1.0.8 h7f98852_4 https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge
ca-certificates 2022.12.7 ha878542_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge
certifi 2022.12.7 pypi_0 pypi
charset-normalizer 2.1.1 pypi_0 pypi
click 8.1.3 pypi_0 pypi
cloudpickle 2.2.0 pypi_0 pypi
coloredlogs 15.0.1 pypi_0 pypi
contourpy 1.0.6 pypi_0 pypi
cycler 0.11.0 pypi_0 pypi
cython 0.29.32 pypi_0 pypi
filelock 3.8.2 pypi_0 pypi
flatbuffers 22.12.6 pypi_0 pypi
fonttools 4.38.0 pypi_0 pypi
huggingface-hub 0.0.8 pypi_0 pypi
humanfriendly 10.0 pypi_0 pypi
idna 3.4 pypi_0 pypi
joblib 1.2.0 pypi_0 pypi
kiwisolver 1.4.4 pypi_0 pypi
ld_impl_linux-64 2.39 hcc3a1bd_1 https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge
libffi 3.4.2 h7f98852_5 https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge
libgcc-ng 12.2.0 h65d4601_19 https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge
libgomp 12.2.0 h65d4601_19 https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge
libnsl 2.0.0 h7f98852_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge
libsqlite 3.40.0 h753d276_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge
libuuid 2.32.1 h7f98852_1000 https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge
libzlib 1.2.13 h166bdaf_4 https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge
matplotlib 3.6.2 pypi_0 pypi
mpmath 1.2.1 pypi_0 pypi
ncurses 6.3 h27087fc_1 https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge
numpy 1.23.5 pypi_0 pypi
onnx 1.13.0 pypi_0 pypi
onnxruntime 1.13.1 pypi_0 pypi
openssl 3.0.7 h0b41bf4_1 https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge
packaging 22.0 pypi_0 pypi
panopticapi 0.1 pypi_0 pypi
pillow 9.3.0 pypi_0 pypi
pip 22.3.1 pyhd8ed1ab_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge
prettytable 3.5.0 pypi_0 pypi
protobuf 3.20.3 pypi_0 pypi
pycocotools 2.0 pypi_0 pypi
pyparsing 3.0.9 pypi_0 pypi
python 3.8.15 h4a9ceb5_0_cpython https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge
python-dateutil 2.8.2 pypi_0 pypi
pyyaml 6.0 pypi_0 pypi
readline 8.1.2 h0f457ee_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge
regex 2022.10.31 pypi_0 pypi
requests 2.28.1 pypi_0 pypi
sacremoses 0.0.53 pypi_0 pypi
scipy 1.9.3 pypi_0 pypi
setuptools 65.5.1 pyhd8ed1ab_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge
six 1.16.0 pypi_0 pypi
submitit 1.4.5 pypi_0 pypi
sympy 1.11.1 pypi_0 pypi
timm 0.6.12 pypi_0 pypi
tk 8.6.12 h27826a3_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge
tokenizers 0.10.3 pypi_0 pypi
torch 1.11.0+cu113 pypi_0 pypi
torchaudio 0.11.0+cu113 pypi_0 pypi
torchvision 0.12.0+cu113 pypi_0 pypi
tqdm 4.64.1 pypi_0 pypi
transformers 4.6.0 pypi_0 pypi
typing-extensions 4.4.0 pypi_0 pypi
urllib3 1.26.13 pypi_0 pypi
wcwidth 0.2.5 pypi_0 pypi
wheel 0.38.4 pyhd8ed1ab_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge
xmltodict 0.13.0 pypi_0 pypi
xz 5.2.6 h166bdaf_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge
The text was updated successfully, but these errors were encountered: