Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

【论文复现】模型动转静报错: Input(Ids) of LookupTableV2Op should not be null #43225

Closed
fuqianya opened this issue Jun 6, 2022 · 11 comments
Labels
PFCC Paddle Framework Contributor Club,https://github.com/PaddlePaddle/community/tree/master/pfcc status/close 已关闭 type/others 其他问题

Comments

@fuqianya
Copy link
Contributor

fuqianya commented Jun 6, 2022

问题描述 Please describe your issue

运行环境:

  • Python 3.6

  • paddlepaddle-gpu 2.3.0

模型由动态图模式转为静态图的时候,报如下的错:

ValueError: (InvalidArgument) Input(Ids) of LookupTableV2Op should not be null.
[Hint: Expected ctx->HasInput("Ids") == true, but received ctx->HasInput("Ids"):0 != true:1.] (at /paddle/paddle/fluid/operators/lookup_table_v2_op.cc:35)

动转静代码:
https://github.com/Mixture-of-Rookie/UNITER-Paddle/blob/main/export_model.py

模型组网代码:
https://github.com/Mixture-of-Rookie/UNITER-Paddle/blob/main/models/uniter.py#L184

具体报错如下:

W0606 15:19:12.259377  5653 gpu_context.cc:278] Please NOTE: device: 0, GPU Compute Capability: 6.1, Driver API Version: 11.6, Runtime API Version: 11.1                                      
W0606 15:19:12.262272  5653 gpu_context.cc:306] device: 0, cuDNN Version: 8.0.                                                                                                                
W0606 15:19:12.470620  5653 gpu_context.cc:506] WARNING: device: . The installed Paddle is compiled with CUDNN 8.1, but CUDNN version in your machine is 8.0, which may cause serious incompat
ible bug. Please recompile or reinstall Paddle with compatible CUDNN version.                                                                                                                 
/home/fuqian/Downloads/Software/anaconda3/envs/2020-UNITER-ECCV/lib/python3.6/site-packages/paddle/fluid/dygraph/layers.py:1483: UserWarning: Skip loading for rank_output.weight. rank_output
.weight is not found in the provided dict.                                                                                                                                                    
  warnings.warn(("Skip loading for {}. ".format(key) + str(err)))                                                                                                                             
/home/fuqian/Downloads/Software/anaconda3/envs/2020-UNITER-ECCV/lib/python3.6/site-packages/paddle/fluid/dygraph/layers.py:1483: UserWarning: Skip loading for rank_output.bias. rank_output.b
ias is not found in the provided dict.                                                                                                                                                        
  warnings.warn(("Skip loading for {}. ".format(key) + str(err)))                                                                                                                             
/home/fuqian/Downloads/Software/anaconda3/envs/2020-UNITER-ECCV/lib/python3.6/site-packages/paddle/fluid/dygraph/dygraph_to_static/grad_transformer.py:88: UserWarning: paddle.no_grad is only
 supported for inference model, and not supported for training under @to_static.                                                                                                              
  "paddle.no_grad is only supported for inference model, and not supported for training under @to_static."                                                                                    
Traceback (most recent call last):                                                                                                                                                            
  File "export_model.py", line 115, in <module>                                                                                                                                               
    export(args, cfg)                                                                                                                                                                         
  File "export_model.py", line 109, in export                                                                                                                                                 
    paddle.jit.save(model, os.path.join(args.out_dir, "inference"))                                                                                                                           
  File "/home/fuqian/Downloads/Software/anaconda3/envs/2020-UNITER-ECCV/lib/python3.6/site-packages/paddle/fluid/dygraph/jit.py", line 629, in wrapper                                        
    func(layer, path, input_spec, **configs)
  File "/home/fuqian/Downloads/Software/anaconda3/envs/2020-UNITER-ECCV/lib/python3.6/site-packages/decorator.py", line 232, in fun
    return caller(func, *(extras + args), **kw)
  File "/home/fuqian/Downloads/Software/anaconda3/envs/2020-UNITER-ECCV/lib/python3.6/site-packages/paddle/fluid/wrapped_decorator.py", line 25, in __impl__
    return wrapped_func(*args, **kwargs)
  File "/home/fuqian/Downloads/Software/anaconda3/envs/2020-UNITER-ECCV/lib/python3.6/site-packages/paddle/fluid/dygraph/base.py", line 51, in __impl__
    return func(*args, **kwargs)
  File "/home/fuqian/Downloads/Software/anaconda3/envs/2020-UNITER-ECCV/lib/python3.6/site-packages/paddle/fluid/dygraph/jit.py", line 857, in save
    inner_input_spec, with_hook=with_hook)
  File "/home/fuqian/Downloads/Software/anaconda3/envs/2020-UNITER-ECCV/lib/python3.6/site-packages/paddle/fluid/dygraph/dygraph_to_static/program_translator.py", line 528, in concrete_progr
am_specify_input_spec
    *desired_input_spec, with_hook=with_hook)

  File "/home/fuqian/Downloads/Software/anaconda3/envs/2020-UNITER-ECCV/lib/python3.6/site-packages/paddle/fluid/dygraph/dygraph_to_static/program_translator.py", line 436, in get_concrete_program
    concrete_program, partial_program_layer = self._program_cache[cache_key]
  File "/home/fuqian/Downloads/Software/anaconda3/envs/2020-UNITER-ECCV/lib/python3.6/site-packages/paddle/fluid/dygraph/dygraph_to_static/program_translator.py", line 801, in __getitem__
    self._caches[item_id] = self._build_once(item)
  File "/home/fuqian/Downloads/Software/anaconda3/envs/2020-UNITER-ECCV/lib/python3.6/site-packages/paddle/fluid/dygraph/dygraph_to_static/program_translator.py", line 790, in _build_once
    **cache_key.kwargs)
  File "/home/fuqian/Downloads/Software/anaconda3/envs/2020-UNITER-ECCV/lib/python3.6/site-packages/decorator.py", line 232, in fun
    return caller(func, *(extras + args), **kw)
  File "/home/fuqian/Downloads/Software/anaconda3/envs/2020-UNITER-ECCV/lib/python3.6/site-packages/paddle/fluid/wrapped_decorator.py", line 25, in __impl__
    return wrapped_func(*args, **kwargs)
  File "/home/fuqian/Downloads/Software/anaconda3/envs/2020-UNITER-ECCV/lib/python3.6/site-packages/paddle/fluid/dygraph/base.py", line 51, in __impl__
    return func(*args, **kwargs)
  File "/home/fuqian/Downloads/Software/anaconda3/envs/2020-UNITER-ECCV/lib/python3.6/site-packages/paddle/fluid/dygraph/dygraph_to_static/program_translator.py", line 740, in from_func_spec
    error_data.raise_new_exception()
  File "/home/fuqian/Downloads/Software/anaconda3/envs/2020-UNITER-ECCV/lib/python3.6/site-packages/paddle/fluid/dygraph/dygraph_to_static/error.py", line 336, in raise_new_exception
    six.exec_("raise new_exception from None")
  File "<string>", line 1, in <module>
ValueError: In transformed code:

    File "/home/fuqian/Documents/Research/Multi-Modal-Pretraining/2020-UNITER-ECCV/UNITER-Paddle/models/uniter_retrieval.py", line 58, in forward                                    [30/1046]
        return self.compute_score(batch, compute_loss)                                                                                                                                        
    File "/home/fuqian/Documents/Research/Multi-Modal-Pretraining/2020-UNITER-ECCV/UNITER-Paddle/models/uniter_retrieval.py", line 71, in compute_score                                       
        output_all_encoded_layers=False)                                                                                                                                                      
    File "/home/fuqian/Documents/Research/Multi-Modal-Pretraining/2020-UNITER-ECCV/UNITER-Paddle/models/uniter.py", line 324, in forward                                                      
        gather_index, img_masks, txt_type_ids, img_type_ids)                                                                                                                                  
    File "/home/fuqian/Documents/Research/Multi-Modal-Pretraining/2020-UNITER-ECCV/UNITER-Paddle/models/uniter.py", line 291, in _compute_img_txt_embeddings                                  
        input_ids, position_ids, txt_type_ids)
    File "/home/fuqian/Documents/Research/Multi-Modal-Pretraining/2020-UNITER-ECCV/UNITER-Paddle/models/uniter.py", line 273, in _compute_txt_embeddings
        output = self.embeddings(input_ids, position_ids, txt_type_ids)
    File "/home/fuqian/Documents/Research/Multi-Modal-Pretraining/2020-UNITER-ECCV/UNITER-Paddle/models/uniter.py", line 202, in forward
            # token_type_ids = paddle.zeros_like(input_ids)

        words_embeddings = self.word_embeddings(input_ids)
        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ <--- HERE
        position_embeddings = self.position_embeddings(position_ids)
        token_type_embeddings = self.token_type_embeddings(token_type_ids)

    File "/home/fuqian/Downloads/Software/anaconda3/envs/2020-UNITER-ECCV/lib/python3.6/site-packages/paddle/fluid/dygraph/layers.py", line 930, in __call__
        return self._dygraph_call_func(*inputs, **kwargs)
    File "/home/fuqian/Downloads/Software/anaconda3/envs/2020-UNITER-ECCV/lib/python3.6/site-packages/paddle/fluid/dygraph/layers.py", line 915, in _dygraph_call_func
        outputs = self.forward(*inputs, **kwargs)
    File "/home/fuqian/Downloads/Software/anaconda3/envs/2020-UNITER-ECCV/lib/python3.6/site-packages/paddle/nn/layer/common.py", line 1469, in forward
        name=self._name)
    File "/home/fuqian/Downloads/Software/anaconda3/envs/2020-UNITER-ECCV/lib/python3.6/site-packages/paddle/nn/functional/input.py", line 229, in embedding
        'padding_idx': padding_idx
    File "/home/fuqian/Downloads/Software/anaconda3/envs/2020-UNITER-ECCV/lib/python3.6/site-packages/paddle/fluid/layer_helper.py", line 44, in append_op
        return self.main_program.current_block().append_op(*args, **kwargs)
    File "/home/fuqian/Downloads/Software/anaconda3/envs/2020-UNITER-ECCV/lib/python3.6/site-packages/paddle/fluid/framework.py", line 3621, in append_op
        attrs=kwargs.get("attrs", None))
    File "/home/fuqian/Downloads/Software/anaconda3/envs/2020-UNITER-ECCV/lib/python3.6/site-packages/paddle/fluid/framework.py", line 2764, in __init__
        self.desc.infer_shape(self.block.desc)

    ValueError: (InvalidArgument) Input(Ids) of LookupTableV2Op should not be null.
  [Hint: Expected ctx->HasInput("Ids") == true, but received ctx->HasInput("Ids"):0 != true:1.] (at /paddle/paddle/fluid/operators/lookup_table_v2_op.cc:35)
  [operator < lookup_table_v2 > error]

尝试如下:

  • 调试输出 input_ids, 发现不是null

    var expand_0.tmp_0 : LOD_TENSOR.shape(1, 40).dtype(int64).stop_gradient(False)
    
@fuqianya fuqianya added status/new-issue 新建 type/others 其他问题 labels Jun 6, 2022
@0x45f
Copy link
Contributor

0x45f commented Jun 7, 2022

你好,方便提供下完整的动转静导出运行命令吗?

@paddle-bot-old paddle-bot-old bot added status/following-up 跟进中 and removed status/new-issue 新建 labels Jun 7, 2022
@fuqianya
Copy link
Contributor Author

fuqianya commented Jun 8, 2022

你好,动转静的导出命令为: python export_model.py --cfg_file configs/retrieval_train.yaml
但是需要事先下载预训练权重,参考https://github.com/Mixture-of-Rookie/UNITER-Paddle 中的 五、快速开始 --> step3: 挂载数据

@0x45f
Copy link
Contributor

0x45f commented Jun 9, 2022

你好,动转静的导出命令为: python export_model.py --cfg_file configs/retrieval_train.yaml。 但是需要事先下载预训练权重,参考https://github.com/Mixture-of-Rookie/UNITER-Paddle 中的 五、快速开始 --> step3: 挂载数据

感谢,我们已经定位到了具体问题,目前动转静中ifelse配合dict使用有bug,我们看下如何修复
image

@0x45f
Copy link
Contributor

0x45f commented Jun 10, 2022

你好,动转静的导出命令为: python export_model.py --cfg_file configs/retrieval_train.yaml。 但是需要事先下载预训练权重,参考https://github.com/Mixture-of-Rookie/UNITER-Paddle 中的 五、快速开始 --> step3: 挂载数据

另外有个问题想要请教下,代码会走到compute_score函数中,但是batch['sample_size']是None,可以检查看下batch['sample_size']有被赋值吗?
image

@fuqianya
Copy link
Contributor Author

你好,非常感谢您的时间~
代码的执行逻辑是这样的:
Screenshot from 2022-06-11 10-32-46
执行53行(将compute_loss设置为False),获得scores。然后执行54行,得到的hard_batch中包含了sample_size

@0x45f
Copy link
Contributor

0x45f commented Jun 13, 2022

你好,非常感谢您的时间~ 代码的执行逻辑是这样的: Screenshot from 2022-06-11 10-32-46 执行53行(将compute_loss设置为False),获得scores。然后执行54行,得到的hard_batch中包含了sample_size

模型导出时self.training=False会走到else分支,但是compute_loss=True,仍然会走计算loss的逻辑。这里调用compute_score时需要将compute_loss设置为Flase吗?

@fuqianya
Copy link
Contributor Author

是的,需要将compute_loss设置为False

@0x45f
Copy link
Contributor

0x45f commented Jun 14, 2022

是的,需要将compute_loss设置为False

ifelse配合dict使用的问题修复PR #43348 已经合入develop分支。
在模型代码中将compute_loss设置为False,并且在model/bert.py中做如下两处修改已经能够正确save,可以安装develop分支的paddle后尝试进行导出。
image
image

@luotao1 luotao1 added the PFCC Paddle Framework Contributor Club,https://github.com/PaddlePaddle/community/tree/master/pfcc label Aug 15, 2022
@fuqianya
Copy link
Contributor Author

fuqianya commented Sep 5, 2022

老师您好,按照您的指导,我已经成功的导出了静态模型。

但是,我在进行模型推理的时候又遇到了如下的问题:

image

当输入x为多个时,输入的格式应该是怎样的?
也就是说,当我的input_names为多个时,输入的input_tensor应该是什么格式?

image

image

我进行了如下尝试:
(1)输入的xdict,即keytensor_name,而valuetensor。但是我得到了如下了错误提示:

image

(2)根据(1)中的错误,更改输入xlist,如下:

image

但还是不行,报错如下:

image

@fuqianya
Copy link
Contributor Author

fuqianya commented Sep 5, 2022

导出模型和推理的代码都在:https://github.com/Mixture-of-Rookie/UNITER-Paddle

@fuqianya
Copy link
Contributor Author

fuqianya commented Sep 6, 2022

您好,我目前已经完成了TIPC的整个流程,只差GPU下的推理了。

我在GPU下的推理中遇到了如下的错误:

➜ python deploy/inference_python/infer.py --use-gpu True                                                                                                                            [79/1753]
^BzTraceback (most recent call last):
  File "deploy/inference_python/infer.py", line 213, in <module>
    infer_main(args)
  File "deploy/inference_python/infer.py", line 185, in infer_main
    output = inference_engine.run(data)
  File "deploy/inference_python/infer.py", line 119, in run
    self.predictor.run()
OSError: In user code:

    File "tools/export_model.py", line 115, in <module>
      export(args, cfg)
    File "tools/export_model.py", line 109, in export
      paddle.jit.save(model, os.path.join(args.out_dir, "inference"))
    File "/home/fuqian/Downloads/Software/anaconda3/envs/2020-UNITER-ECCV/lib/python3.6/site-packages/paddle/fluid/dygraph/jit.py", line 643, in wrapper
      func(layer, path, input_spec, **configs)
    File "/home/fuqian/Downloads/Software/anaconda3/envs/2020-UNITER-ECCV/lib/python3.6/site-packages/decorator.py", line 232, in fun
      return caller(func, *(extras + args), **kw)
    File "/home/fuqian/Downloads/Software/anaconda3/envs/2020-UNITER-ECCV/lib/python3.6/site-packages/paddle/fluid/wrapped_decorator.py", line 26, in __impl__
      return wrapped_func(*args, **kwargs)
    File "/home/fuqian/Downloads/Software/anaconda3/envs/2020-UNITER-ECCV/lib/python3.6/site-packages/paddle/fluid/dygraph/base.py", line 52, in __impl__
      return func(*args, **kwargs)
    File "/home/fuqian/Downloads/Software/anaconda3/envs/2020-UNITER-ECCV/lib/python3.6/site-packages/paddle/fluid/dygraph/jit.py", line 921, in save
      inner_input_spec, with_hook=with_hook)
    File "/home/fuqian/Downloads/Software/anaconda3/envs/2020-UNITER-ECCV/lib/python3.6/site-packages/paddle/fluid/dygraph/dygraph_to_static/program_translator.py", line 578, in concrete_pr
ogram_specify_input_spec
      is_train=self._is_train_mode())
    File "/home/fuqian/Downloads/Software/anaconda3/envs/2020-UNITER-ECCV/lib/python3.6/site-packages/paddle/fluid/dygraph/dygraph_to_static/program_translator.py", line 483, in get_concret
e_program
      concrete_program, partial_program_layer = self._program_cache[cache_key]
    File "/home/fuqian/Downloads/Software/anaconda3/envs/2020-UNITER-ECCV/lib/python3.6/site-packages/paddle/fluid/dygraph/dygraph_to_static/program_translator.py", line 953, in __getitem__
      self._caches[item_id] = self._build_once(item)
    File "/home/fuqian/Downloads/Software/anaconda3/envs/2020-UNITER-ECCV/lib/python3.6/site-packages/paddle/fluid/dygraph/dygraph_to_static/program_translator.py", line 942, in _build_once
      **cache_key.kwargs)
    File "/home/fuqian/Downloads/Software/anaconda3/envs/2020-UNITER-ECCV/lib/python3.6/site-packages/decorator.py", line 232, in fun
      return caller(func, *(extras + args), **kw)
    File "/home/fuqian/Downloads/Software/anaconda3/envs/2020-UNITER-ECCV/lib/python3.6/site-packages/paddle/fluid/wrapped_decorator.py", line 26, in __impl__
      return wrapped_func(*args, **kwargs)
    File "/home/fuqian/Downloads/Software/anaconda3/envs/2020-UNITER-ECCV/lib/python3.6/site-packages/paddle/fluid/dygraph/base.py", line 52, in __impl__
      return func(*args, **kwargs)
    File "/home/fuqian/Downloads/Software/anaconda3/envs/2020-UNITER-ECCV/lib/python3.6/site-packages/paddle/fluid/dygraph/dygraph_to_static/program_translator.py", line 886, in from_func_s
pec
      outputs = static_func(*inputs)
    File "/home/fuqian/Documents/Research/Multi-Modal-Pretraining/2020-UNITER-ECCV/UNITER-Paddle/models/uniter_retrieval.py", line 50, in forward
      if self.training and compute_loss:
    File "/home/fuqian/Downloads/Software/anaconda3/envs/2020-UNITER-ECCV/lib/python3.6/site-packages/paddle/fluid/dygraph/dygraph_to_static/convert_operators.py", line 320, in convert_ife$
se
      return_name_ids)
    File "/home/fuqian/Downloads/Software/anaconda3/envs/2020-UNITER-ECCV/lib/python3.6/site-packages/paddle/fluid/dygraph/dygraph_to_static/convert_operators.py", line 376, in _run_py_ife$
se
      py_outs = true_fn() if pred else false_fn()
    File "/home/fuqian/Documents/Research/Multi-Modal-Pretraining/2020-UNITER-ECCV/UNITER-Paddle/models/uniter_retrieval.py", line 58, in forward
      return self.compute_score(batch, compute_loss)
    File "/home/fuqian/Documents/Research/Multi-Modal-Pretraining/2020-UNITER-ECCV/UNITER-Paddle/models/uniter_retrieval.py", line 68, in compute_score
      sequence_output = self.uniter(input_ids, position_ids,
    File "/home/fuqian/Downloads/Software/anaconda3/envs/2020-UNITER-ECCV/lib/python3.6/site-packages/paddle/fluid/dygraph/layers.py", line 950, in __call__
      return self._dygraph_call_func(*inputs, **kwargs)
    File "/home/fuqian/Downloads/Software/anaconda3/envs/2020-UNITER-ECCV/lib/python3.6/site-packages/paddle/fluid/dygraph/layers.py", line 935, in _dygraph_call_func
      outputs = self.forward(*inputs, **kwargs)
    File "/home/fuqian/Documents/Research/Multi-Modal-Pretraining/2020-UNITER-ECCV/UNITER-Paddle/models/uniter.py", line 312, in forward
      if input_ids is None:
    File "/home/fuqian/Downloads/Software/anaconda3/envs/2020-UNITER-ECCV/lib/python3.6/site-packages/paddle/fluid/dygraph/dygraph_to_static/convert_operators.py", line 320, in convert_ife$
se
      return_name_ids)
    File "/home/fuqian/Downloads/Software/anaconda3/envs/2020-UNITER-ECCV/lib/python3.6/site-packages/paddle/fluid/dygraph/dygraph_to_static/convert_operators.py", line 376, in _run_py_ife$
se
      py_outs = true_fn() if pred else false_fn()
    File "/home/fuqian/Documents/Research/Multi-Modal-Pretraining/2020-UNITER-ECCV/UNITER-Paddle/models/uniter.py", line 316, in forward
      elif img_feat is None:
    File "/home/fuqian/Downloads/Software/anaconda3/envs/2020-UNITER-ECCV/lib/python3.6/site-packages/paddle/fluid/dygraph/dygraph_to_static/convert_operators.py", line 320, in convert_ife$
se
      return_name_ids)
    File "/home/fuqian/Downloads/Software/anaconda3/envs/2020-UNITER-ECCV/lib/python3.6/site-packages/paddle/fluid/dygraph/dygraph_to_static/convert_operators.py", line 376, in _run_py_ife$
se
      py_outs = true_fn() if pred else false_fn()
    File "/home/fuqian/Documents/Research/Multi-Modal-Pretraining/2020-UNITER-ECCV/UNITER-Paddle/models/uniter.py", line 321, in forward
      embedding_output = self._compute_img_txt_embeddings(
    File "/home/fuqian/Documents/Research/Multi-Modal-Pretraining/2020-UNITER-ECCV/UNITER-Paddle/models/uniter.py", line 297, in _compute_img_txt_embeddings
      embedding_output = paddle_gather(paddle.concat([txt_emb, img_emb], axis=1),
    File "/home/fuqian/Documents/Research/Multi-Modal-Pretraining/2020-UNITER-ECCV/UNITER-Paddle/utils/io_utils.py", line 72, in paddle_gather
      index_flatten = index.flatten()
    File "/home/fuqian/Downloads/Software/anaconda3/envs/2020-UNITER-ECCV/lib/python3.6/site-packages/paddle/tensor/manipulation.py", line 1497, in flatten
      "stop_axis": stop_axis
    File "/home/fuqian/Downloads/Software/anaconda3/envs/2020-UNITER-ECCV/lib/python3.6/site-packages/paddle/fluid/layer_helper.py", line 45, in append_op
      return self.main_program.current_block().append_op(*args, **kwargs)
    File "/home/fuqian/Downloads/Software/anaconda3/envs/2020-UNITER-ECCV/lib/python3.6/site-packages/paddle/fluid/framework.py", line 3828, in append_op
      attrs=kwargs.get("attrs", None))
    File "/home/fuqian/Downloads/Software/anaconda3/envs/2020-UNITER-ECCV/lib/python3.6/site-packages/paddle/fluid/framework.py", line 2736, in __init__
      for frame in traceback.extract_stack():

    ExternalError: CUDA error(700), an illegal memory access was encountered.
      [Hint: 'cudaErrorIllegalAddress'. The device encountered a load or store instruction on an invalid memory address. This leaves the process in an inconsistentstate and any further CUDA
 work will return the same error. To continue using CUDA, the process must be terminated and relaunched. ] (at /paddle/paddle/phi/backends/gpu/cuda/cuda_info.cc:252)
      [operator < flatten_contiguous_range > error]
terminate called after throwing an instance of 'phi::enforce::EnforceNotMet'
  what():  (External) CUDA error(700), an illegal memory access was encountered.
  [Hint: 'cudaErrorIllegalAddress'. The device encountered a load or store instruction on an invalid memory address. This leaves the process in an inconsistentstate and any further CUDA wor
k will return the same error. To continue using CUDA, the process must be terminated and relaunched. ] (at /paddle/paddle/fluid/platform/device/gpu/gpu_info.cc:289)



--------------------------------------
C++ Traceback (most recent call last):
--------------------------------------
0   paddle::AnalysisPredictor::~AnalysisPredictor()
1   paddle::AnalysisPredictor::~AnalysisPredictor()
2   paddle::memory::allocation::StreamSafeCUDAAllocator::ReleaseImpl(phi::Place const&)
3   paddle::memory::allocation::AutoGrowthBestFitAllocator::FreeIdleChunks()
4   paddle::memory::allocation::CUDAAllocator::FreeImpl(phi::Allocation*)
5   paddle::platform::RecordedGpuMallocHelper::Free(void*, unsigned long)

----------------------
Error Message Summary:
----------------------
FatalError: `Process abort signal` is detected by the operating system.
  [TimeInfo: *** Aborted at 1662429696 (unix time) try "date -d @1662429696" if you are using GNU date ***]
  [SignalInfo: *** SIGABRT (@0x3e800006aff) received by PID 27391 (TID 0x7fb3d1daf0c0) from PID 27391 ***]

[1]    27391 abort (core dumped)  python deploy/inference_python/infer.py --use-gpu True

@paddle-bot paddle-bot bot added status/close 已关闭 and removed status/following-up 跟进中 labels Sep 21, 2022
@fuqianya fuqianya reopened this Sep 22, 2022
@paddle-bot paddle-bot bot added status/reopen 重新打开 and removed status/close 已关闭 labels Sep 22, 2022
@paddle-bot paddle-bot bot added status/close 已关闭 and removed status/reopen 重新打开 labels Sep 26, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
PFCC Paddle Framework Contributor Club,https://github.com/PaddlePaddle/community/tree/master/pfcc status/close 已关闭 type/others 其他问题
Projects
None yet
Development

No branches or pull requests

3 participants