[04/07 15:54:35 ape.modeling.ape_deta.deformable_detr]: Setting eval dataset to: cvppp_coco cvppp_coco_val 0
[04/07 15:54:35 ape.modeling.ape_deta.deformable_detr]: Setting eval entity to: cvppp_coco cvppp_coco_val thing
[04/07 15:54:35 d2.data.datasets.coco]: Loaded 42 images in COCO format from /home/maer/disk16T/xiongrui/APE-datasets/cvppp_coco/coco/annotations/instances_leaf_val.json
[04/07 15:54:35 d2.data.build]: Distribution of instances among all 1 categories:
|  category  | #instances   |
|:----------:|:-------------|
|    leaf    | 554          |
|            |              |
[04/07 15:54:35 d2.data.dataset_mapper]: [DatasetMapper] Augmentations used in inference: [ResizeShortestEdge(short_edge_length=(1024, 1024), max_size=1024)]
[04/07 15:54:35 d2.data.common]: Serializing the dataset using: <class 'detectron2.data.common._TorchSerializedList'>
[04/07 15:54:35 d2.data.common]: Serializing 42 elements to byte tensors and concatenating them all ...
[04/07 15:54:35 d2.data.common]: Serialized dataset takes 0.18 MiB
[04/07 15:54:35 ape.evaluation.evaluator]: Start inference on 21 batches
Traceback (most recent call last):
  File "/home/maer/disk16T/xiongrui/xr/APE/tools/train_net.py", line 750, in <module>
    launch(
  File "/home/maer/disk16T/xiongrui/detectron2/detectron2/engine/launch.py", line 69, in launch
    mp.start_processes(
  File "/home/maer/anaconda3/envs/ape/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 198, in start_processes
    while not context.join():
  File "/home/maer/anaconda3/envs/ape/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 160, in join
    raise ProcessRaisedException(msg, error_index, failed_process.pid)
torch.multiprocessing.spawn.ProcessRaisedException: 

-- Process 1 terminated with the following error:
Traceback (most recent call last):
  File "/home/maer/anaconda3/envs/ape/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 69, in _wrap
    fn(i, *args)
  File "/home/maer/disk16T/xiongrui/detectron2/detectron2/engine/launch.py", line 123, in _distributed_worker
    main_func(*args)
  File "/home/maer/disk16T/xiongrui/xr/APE/tools/train_net.py", line 743, in main
    print(do_test(cfg, model, eval_only=True))
  File "/home/maer/disk16T/xiongrui/xr/APE/tools/train_net.py", line 534, in do_test
    ret = inference_on_dataset(
  File "/home/maer/disk16T/xiongrui/xr/APE/ape/evaluation/evaluator.py", line 83, in inference_on_dataset
    outputs = model(inputs)
  File "/home/maer/anaconda3/envs/ape/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
File "/home/maer/anaconda3/envs/ape/lib/python3.9/site-packages/torch/nn/parallel/distributed.py", line 1008, in forward
    output = self._run_ddp_forward(*inputs, **kwargs)
  File "/home/maer/anaconda3/envs/ape/lib/python3.9/site-packages/torch/nn/parallel/distributed.py", line 969, in _run_ddp_forward
    return module_to_run(*inputs[0], **kwargs[0])
  File "/home/maer/anaconda3/envs/ape/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/maer/disk16T/xiongrui/xr/APE/ape/modeling/ape_deta/ape_deta.py", line 36, in forward
    losses = self.model_vision(batched_inputs, do_postprocess=do_postprocess)
  File "/home/maer/anaconda3/envs/ape/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/maer/disk16T/xiongrui/xr/APE/ape/modeling/ape_deta/deformable_detr_segm_vl.py", line 256, in forward
    outputs_l = self.model_language.forward_text(text_list, cache=cache)
  File "/home/maer/anaconda3/envs/ape/lib/python3.9/site-packages/torch/amp/autocast_mode.py", line 12, in decorate_autocast
    return func(*args, **kwargs)
  File "/home/maer/anaconda3/envs/ape/lib/python3.9/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
    return func(*args, **kwargs)
  File "/home/maer/disk16T/xiongrui/xr/APE/ape/modeling/text/clip_wrapper_eva02.py", line 107, in forward_text
    encoder_outputs_x, encoder_outputs_xx = self.custom_encode_text(
  File "/home/maer/anaconda3/envs/ape/lib/python3.9/site-packages/torch/amp/autocast_mode.py", line 12, in decorate_autocast
    return func(*args, **kwargs)
  File "/home/maer/anaconda3/envs/ape/lib/python3.9/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
    return func(*args, **kwargs)
  File "/home/maer/disk16T/xiongrui/xr/APE/ape/modeling/text/clip_wrapper_eva02.py", line 137, in custom_encode_text
    x = m.transformer(x, attn_mask=m.attn_mask)
  File "/home/maer/anaconda3/envs/ape/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/maer/disk16T/xiongrui/xr/APE/ape/modeling/text/eva02_clip/transformer.py", line 516, in forward
    x = r(x, attn_mask=attn_mask)
  File "/home/maer/anaconda3/envs/ape/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/maer/disk16T/xiongrui/xr/APE/ape/modeling/text/eva02_clip/transformer.py", line 481, in forward
    x = x + self.ls_1(self.attention(self.ln_1(x), attn_mask=attn_mask))
  File "/home/maer/disk16T/xiongrui/xr/APE/ape/modeling/text/eva02_clip/transformer.py", line 475, in attention
    attn_mask = attn_mask.to(x.dtype) if attn_mask is not None else None
RuntimeError: CUDA error: invalid configuration argument