Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[TF-TRT] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory #52287

Open
ProtossDragoon opened this issue Oct 7, 2021 · 0 comments
Open

[TF-TRT] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory #52287

ProtossDragoon opened this issue Oct 7, 2021 · 0 comments
Assignees
@bixia1
Labels
comp:gpu:tensorrt Issues specific to TensorRT stat:awaiting tensorflower Status - Awaiting response from tensorflower type:bug Bug

Comments

@ProtossDragoon
Copy link
Contributor

ProtossDragoon commented Oct 7, 2021 •
edited

System information

  • Have I written custom code (as opposed to using a stock example script provided in TensorFlow): yes
  • OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Jetson AGX Xavier Jetpack Version 4.5.1 ubuntu 18.04
  • Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device:
  • TensorFlow installed from (source or binary): nvidia container install
  • TensorFlow version (use command below): 7.1.3
  • Python version: 3.6.9
  • Bazel version (if compiling from source):
  • GCC/Compiler version (if compiling from source):
  • CUDA/cuDNN version: CUDA 10.2 cuDNN 8.0
  • GPU model and memory: Jetson AGX Xavier, 16GB

Describe the current behavior

Hi, while using TF-TRT, always I faced on an error 2021-10-04 00:54:11.478005: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory. when I called converter.build(input_fn=my_input_fn) method. (either the error raised at runtime when not call build() method for pre-build engine.)

(image : trials, parameter tuning minimum_segment_size and so on.)
image

But any of them couldn't remove that error message.

My entire source code wasn't different with any other example source codes.

# memory growing mode (already tried fixed gpu memory mode either.)
gpus = tf.config.experimental.list_physical_devices('GPU')
tf.config.experimental.set_memory_growth(gpus[0], True)

#https://github.com/tensorflow/tensorrt/issues/195
#https://github.com/tensorflow/tensorrt/issues/200
env = os.environ.get('TF_DEBUG_TRT_ALLOW_INEFFICIENT_TRANSPOSE')
new_env = os.environ['TF_DEBUG_TRT_ALLOW_INEFFICIENT_TRANSPOSE'] = "1"
print(f'Setting environment variable... TF_DEBUG_TRT_ALLOW_INEFFICIENT_TRANSPOSE={new_env}... (before {env})')

# Conversion Parameters 
conversion_params = trt.TrtConversionParams(
    precision_mode=QUANTIZATION_MODE, # 'INT8' or 'FP16'
    max_workspace_size_bytes=MAX_GPU_TRTENGINE_SIZE_MB<<20, # 2048 ... etc.
    minimum_segment_size=MINIMUM_SEGMENT_SIZE, # 10 ... etc.
    maximum_cached_engines=MAXIMUM_CACHED_ENGINE, # 20 ... etc.
    use_calibration=True # This argument is ignored if precision_mode is not INT8.
    )

converter = trt.TrtGraphConverterV2(
    input_saved_model_dir=SAVED_MODEL_DIR,
    input_saved_model_tags=SAVED_MODEL_TAG_SET,
    input_saved_model_signature_key=SAVED_MODEL_SIGNATURE_DEF,
    conversion_params=conversion_params
    )

if QUANTIZATION_MODE.lower() == 'int8':
    print('Convert with calibration')
    converter.convert(
        calibration_input_fn=lambda:input_fn(args) # args : my command line input
        # input_fn returns numpy array [1, 320, 320, 3] shaped.
        ) # Maybe converter.convert() works fine...?
else:
    print('Convert without calibration')
    converter.convert()

print('[Info] Convert complete.')
print('[Info] Now prebuild trt-engine.')
converter.build(input_fn=lambda:input_fn(args)) # here raises error message, but process doesn't shut down.

optimized_saved_model_fulldir = parse_tftrt_path(args)

# Save the model to the disk
converter.save(optimized_saved_model_fulldir)
print('[Info] writing savedmodel complete.')

I found that optimized model does not improving model inference speed, so I guess the error is not just a warning. Maybe It's a bug.

Thanks!


Describe the expected behavior

  • no warning
  • inference speed improvement

Contributing

  • Do you want to contribute a PR? (yes/no): no
  • Briefly describe your candidate solution(if contributing):

Standalone code to reproduce the issue
Provide a reproducible test case that is the bare minimum necessary to generate
the problem. If possible, please share a link to Colab/Jupyter/any notebook.

COLAB's TF-TRT is unstable.

Other info / logs Include any logs or source code that would be helpful to
diagnose the problem. If including tracebacks, please include the full
traceback. Large logs and files should be attached.

model signature

==============================
2021-10-07 12:58:34.474165: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.10.2
The given SavedModel contains the following tag-sets:
'serve'

2021-10-07 12:59:08.535907: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.10.2
The given SavedModel MetaGraphDef contains SignatureDefs with the following keys:
SignatureDef key: "__saved_model_init_op"
SignatureDef key: "serving_default"

2021-10-07 12:59:41.622920: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.10.2
The given SavedModel SignatureDef contains the following input(s):
  inputs['input_1'] tensor_info:
      dtype: DT_FLOAT
      shape: (-1, -1, -1, 3)
      name: serving_default_input_1:0
The given SavedModel SignatureDef contains the following output(s):
  outputs['softmax'] tensor_info:
      dtype: DT_FLOAT
      shape: (-1, -1, -1, 9)
      name: StatefulPartitionedCall:0
Method name is: tensorflow/serving/predict
==============================

Entire program log description below

2021-10-07 12:58:32.487612: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcuda.so.1
2021-10-07 12:58:32.513042: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1001] ARM64 does not support NUMA - returning NUMA node zero
2021-10-07 12:58:32.513711: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1734] Found device 0 with properties: 
pciBusID: 0000:00:00.0 name: Xavier computeCapability: 7.2
coreClock: 1.377GHz coreCount: 8 deviceMemorySize: 31.17GiB deviceMemoryBandwidth: 82.08GiB/s
2021-10-07 12:58:32.514587: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.10.2
2021-10-07 12:58:32.515218: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcublas.so.10
2021-10-07 12:58:32.515752: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcublasLt.so.10
2021-10-07 12:58:32.516480: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcufft.so.10
2021-10-07 12:58:32.522466: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcurand.so.10
2021-10-07 12:58:32.535133: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcusolver.so.10
2021-10-07 12:58:32.543363: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcusparse.so.10
2021-10-07 12:58:32.546243: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudnn.so.8
2021-10-07 12:58:32.546781: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1001] ARM64 does not support NUMA - returning NUMA node zero
2021-10-07 12:58:32.547194: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1001] ARM64 does not support NUMA - returning NUMA node zero
2021-10-07 12:58:32.547471: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1872] Adding visible gpu devices: 0

Setting environment variable... TF_DEBUG_TRT_ALLOW_INEFFICIENT_TRANSPOSE=1... (before None)
2021-10-07 13:00:15.205272: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libnvinfer.so.7
Convert with calibration
2021-10-07 13:01:31.023349: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1001] ARM64 does not support NUMA - returning NUMA node zero
2021-10-07 13:01:31.023722: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1734] Found device 0 with properties: 
pciBusID: 0000:00:00.0 name: Xavier computeCapability: 7.2
coreClock: 1.377GHz coreCount: 8 deviceMemorySize: 31.17GiB deviceMemoryBandwidth: 82.08GiB/s
2021-10-07 13:01:31.024083: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1001] ARM64 does not support NUMA - returning NUMA node zero
2021-10-07 13:01:31.024338: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1001] ARM64 does not support NUMA - returning NUMA node zero
2021-10-07 13:01:31.024470: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1872] Adding visible gpu devices: 0
2021-10-07 13:01:31.025038: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.10.2
2021-10-07 13:01:36.768804: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1258] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-10-07 13:01:36.768996: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1264]      0 
2021-10-07 13:01:36.769069: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1277] 0:   N 
2021-10-07 13:01:36.769398: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1001] ARM64 does not support NUMA - returning NUMA node zero
2021-10-07 13:01:36.769747: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1001] ARM64 does not support NUMA - returning NUMA node zero
2021-10-07 13:01:36.769976: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1001] ARM64 does not support NUMA - returning NUMA node zero
2021-10-07 13:01:36.770180: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1418] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 18830 MB memory) -> physical GPU (device: 0, name: Xavier, pci bus id: 0000:00:00.0, compute capability: 7.2)
2021-10-07 13:06:42.865817: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1001] ARM64 does not support NUMA - returning NUMA node zero
2021-10-07 13:06:42.866173: I tensorflow/core/grappler/devices.cc:69] Number of eligible GPUs (core count >= 8, compute capability >= 0.0): 1
2021-10-07 13:06:42.866754: I tensorflow/core/grappler/clusters/single_machine.cc:357] Starting new session
2021-10-07 13:06:42.870918: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1001] ARM64 does not support NUMA - returning NUMA node zero
2021-10-07 13:06:42.871261: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1734] Found device 0 with properties: 
pciBusID: 0000:00:00.0 name: Xavier computeCapability: 7.2
coreClock: 1.377GHz coreCount: 8 deviceMemorySize: 31.17GiB deviceMemoryBandwidth: 82.08GiB/s
2021-10-07 13:06:42.871593: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1001] ARM64 does not support NUMA - returning NUMA node zero
2021-10-07 13:06:42.871875: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1001] ARM64 does not support NUMA - returning NUMA node zero
2021-10-07 13:06:42.872000: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1872] Adding visible gpu devices: 0
2021-10-07 13:06:42.872134: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1258] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-10-07 13:06:42.872198: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1264]      0 
2021-10-07 13:06:42.872245: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1277] 0:   N 
2021-10-07 13:06:42.872483: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1001] ARM64 does not support NUMA - returning NUMA node zero
2021-10-07 13:06:42.872836: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1001] ARM64 does not support NUMA - returning NUMA node zero
2021-10-07 13:06:42.873158: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1418] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 18830 MB memory) -> physical GPU (device: 0, name: Xavier, pci bus id: 0000:00:00.0, compute capability: 7.2)
2021-10-07 13:06:42.874915: I tensorflow/core/platform/profile_utils/cpu_utils.cc:114] CPU Frequency: 31250000 Hz
2021-10-07 13:06:47.236124: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:1171] Optimization results for grappler item: graph_to_optimize
  function_optimizer: Graph size after: 1216 nodes (901), 1824 edges (1509), time = 176.228ms.
  function_optimizer: function_optimizer did nothing. time = 2.022ms.

2021-10-07 13:07:39.201871: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1001] ARM64 does not support NUMA - returning NUMA node zero
2021-10-07 13:07:39.202299: I tensorflow/core/grappler/devices.cc:69] Number of eligible GPUs (core count >= 8, compute capability >= 0.0): 1
2021-10-07 13:07:39.202699: I tensorflow/core/grappler/clusters/single_machine.cc:357] Starting new session
2021-10-07 13:07:39.210304: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1001] ARM64 does not support NUMA - returning NUMA node zero
2021-10-07 13:07:39.210650: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1734] Found device 0 with properties: 
pciBusID: 0000:00:00.0 name: Xavier computeCapability: 7.2
coreClock: 1.377GHz coreCount: 8 deviceMemorySize: 31.17GiB deviceMemoryBandwidth: 82.08GiB/s
2021-10-07 13:07:39.211093: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1001] ARM64 does not support NUMA - returning NUMA node zero
2021-10-07 13:07:39.211425: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1001] ARM64 does not support NUMA - returning NUMA node zero
2021-10-07 13:07:39.211551: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1872] Adding visible gpu devices: 0
2021-10-07 13:07:39.211696: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1258] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-10-07 13:07:39.211766: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1264]      0 
2021-10-07 13:07:39.211813: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1277] 0:   N 
2021-10-07 13:07:39.212127: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1001] ARM64 does not support NUMA - returning NUMA node zero
2021-10-07 13:07:39.212488: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1001] ARM64 does not support NUMA - returning NUMA node zero
2021-10-07 13:07:39.212722: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1418] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 18830 MB memory) -> physical GPU (device: 0, name: Xavier, pci bus id: 0000:00:00.0, compute capability: 7.2)
2021-10-07 13:07:45.495807: I tensorflow/compiler/tf2tensorrt/segment/segment.cc:790] There are 30 ops of 8 different types in the graph that are not converted to TensorRT: Identity, ResizeNearestNeighbor, Placeholder, NoOp, Mul, Shape, DataFormatVecPermute, StridedSlice, (For more information see https://docs.nvidia.com/deeplearning/frameworks/tf-trt-user-guide/index.html#supported-ops).
2021-10-07 13:07:46.085000: I tensorflow/compiler/tf2tensorrt/convert/convert_graph.cc:759] Number of TensorRT candidate segments: 6
2021-10-07 13:07:46.227369: I tensorflow/compiler/tf2tensorrt/convert/convert_graph.cc:853] Replaced segment 0 consisting of 424 nodes by TRTEngineOp_0_0.
2021-10-07 13:07:46.232840: I tensorflow/compiler/tf2tensorrt/convert/convert_graph.cc:853] Replaced segment 1 consisting of 22 nodes by StatefulPartitionedCall/model/TRTEngineOp_0_1.
2021-10-07 13:07:46.234022: I tensorflow/compiler/tf2tensorrt/convert/convert_graph.cc:853] Replaced segment 2 consisting of 22 nodes by StatefulPartitionedCall/model/TRTEngineOp_0_2.
2021-10-07 13:07:46.234935: I tensorflow/compiler/tf2tensorrt/convert/convert_graph.cc:853] Replaced segment 3 consisting of 22 nodes by StatefulPartitionedCall/model/TRTEngineOp_0_3.
2021-10-07 13:07:46.236717: I tensorflow/compiler/tf2tensorrt/convert/convert_graph.cc:853] Replaced segment 4 consisting of 22 nodes by StatefulPartitionedCall/model/TRTEngineOp_0_4.
2021-10-07 13:07:46.237427: I tensorflow/compiler/tf2tensorrt/convert/convert_graph.cc:853] Replaced segment 5 consisting of 27 nodes by TRTEngineOp_0_5.
2021-10-07 13:07:47.073017: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:1171] Optimization results for grappler item: tf_graph
  constant_folding: Graph size after: 558 nodes (-624), 1200 edges (-624), time = 554.18ms.
  layout: Graph size after: 595 nodes (37), 1237 edges (37), time = 328.333ms.
  constant_folding: Graph size after: 587 nodes (-8), 1229 edges (-8), time = 230.304ms.
  TensorRTOptimizer: Graph size after: 54 nodes (-533), 62 edges (-1167), time = 975.276ms.
  constant_folding: Graph size after: 54 nodes (0), 62 edges (0), time = 21.947ms.
Optimization results for grappler item: StatefulPartitionedCall/model/TRTEngineOp_0_1_native_segment
  constant_folding: Graph size after: 26 nodes (0), 25 edges (0), time = 26.112ms.
  layout: Graph size after: 26 nodes (0), 25 edges (0), time = 42.376ms.
  constant_folding: Graph size after: 26 nodes (0), 25 edges (0), time = 31.746ms.
  TensorRTOptimizer: Graph size after: 26 nodes (0), 25 edges (0), time = 5.574ms.
  constant_folding: Graph size after: 26 nodes (0), 25 edges (0), time = 22.56ms.
Optimization results for grappler item: StatefulPartitionedCall/model/TRTEngineOp_0_2_native_segment
  constant_folding: Graph size after: 26 nodes (0), 25 edges (0), time = 6.176ms.
  layout: Graph size after: 26 nodes (0), 25 edges (0), time = 9.286ms.
  constant_folding: Graph size after: 26 nodes (0), 25 edges (0), time = 8.636ms.
  TensorRTOptimizer: Graph size after: 26 nodes (0), 25 edges (0), time = 0.877ms.
  constant_folding: Graph size after: 26 nodes (0), 25 edges (0), time = 6.713ms.
Optimization results for grappler item: StatefulPartitionedCall/model/TRTEngineOp_0_4_native_segment
  constant_folding: Graph size after: 26 nodes (0), 25 edges (0), time = 4.232ms.
  layout: Graph size after: 26 nodes (0), 25 edges (0), time = 4.541ms.
  constant_folding: Graph size after: 26 nodes (0), 25 edges (0), time = 4.265ms.
  TensorRTOptimizer: Graph size after: 26 nodes (0), 25 edges (0), time = 0.405ms.
  constant_folding: Graph size after: 26 nodes (0), 25 edges (0), time = 4.41ms.
Optimization results for grappler item: TRTEngineOp_0_5_native_segment
  constant_folding: Graph size after: 29 nodes (0), 40 edges (0), time = 4.414ms.
  layout: Graph size after: 29 nodes (0), 40 edges (0), time = 4.662ms.
  constant_folding: Graph size after: 29 nodes (0), 40 edges (0), time = 4.751ms.
  TensorRTOptimizer: Graph size after: 29 nodes (0), 40 edges (0), time = 0.367ms.
  constant_folding: Graph size after: 29 nodes (0), 40 edges (0), time = 4.544ms.
Optimization results for grappler item: StatefulPartitionedCall/model/TRTEngineOp_0_3_native_segment
  constant_folding: Graph size after: 26 nodes (0), 25 edges (0), time = 4.687ms.
  layout: Graph size after: 26 nodes (0), 25 edges (0), time = 7.218ms.
  constant_folding: Graph size after: 26 nodes (0), 25 edges (0), time = 4.93ms.
  TensorRTOptimizer: Graph size after: 26 nodes (0), 25 edges (0), time = 0.535ms.
  constant_folding: Graph size after: 26 nodes (0), 25 edges (0), time = 4.577ms.
Optimization results for grappler item: TRTEngineOp_0_0_native_segment
  constant_folding: Graph size after: 431 nodes (0), 440 edges (0), time = 57.452ms.
  layout: Graph size after: 431 nodes (0), 440 edges (0), time = 88.399ms.
  constant_folding: Graph size after: 431 nodes (0), 440 edges (0), time = 70.943ms.
  TensorRTOptimizer: Graph size after: 431 nodes (0), 440 edges (0), time = 8.772ms.
  constant_folding: Graph size after: 431 nodes (0), 440 edges (0), time = 66.01ms.

Found 30 files belonging to 1 classes.
2021-10-07 13:40:34.221298: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:176] None of the MLIR Optimization Passes are enabled (registered 2)
step 1/30, image shape : (1, 320, 320, 3)
2021-10-07 13:40:35.773429: I tensorflow/compiler/tf2tensorrt/common/utils.cc:58] Linked TensorRT version: 7.1.3
2021-10-07 13:40:35.773914: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libnvinfer.so.7
2021-10-07 13:40:35.773981: I tensorflow/compiler/tf2tensorrt/common/utils.cc:60] Loaded TensorRT version: 7.1.3
2021-10-07 13:40:35.785002: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libnvinfer_plugin.so.7
2021-10-07 13:41:56.956338: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudnn.so.8
2021-10-07 13:41:56.964140: I tensorflow/stream_executor/cuda/cuda_dnn.cc:380] Loaded cuDNN version 8000
2021-10-07 13:42:03.715476: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcublas.so.10
step 2/30, image shape : (1, 320, 320, 3)
step 3/30, image shape : (1, 320, 320, 3)
step 4/30, image shape : (1, 320, 320, 3)
step 5/30, image shape : (1, 320, 320, 3)
step 6/30, image shape : (1, 320, 320, 3)
step 7/30, image shape : (1, 320, 320, 3)
step 8/30, image shape : (1, 320, 320, 3)
step 9/30, image shape : (1, 320, 320, 3)
step 10/30, image shape : (1, 320, 320, 3)
step 11/30, image shape : (1, 320, 320, 3)
step 12/30, image shape : (1, 320, 320, 3)
step 13/30, image shape : (1, 320, 320, 3)
step 14/30, image shape : (1, 320, 320, 3)
step 15/30, image shape : (1, 320, 320, 3)
step 16/30, image shape : (1, 320, 320, 3)
step 17/30, image shape : (1, 320, 320, 3)
step 18/30, image shape : (1, 320, 320, 3)
step 19/30, image shape : (1, 320, 320, 3)
step 20/30, image shape : (1, 320, 320, 3)
step 21/30, image shape : (1, 320, 320, 3)
step 22/30, image shape : (1, 320, 320, 3)
step 23/30, image shape : (1, 320, 320, 3)
step 24/30, image shape : (1, 320, 320, 3)
step 25/30, image shape : (1, 320, 320, 3)
step 26/30, image shape : (1, 320, 320, 3)
step 27/30, image shape : (1, 320, 320, 3)
step 28/30, image shape : (1, 320, 320, 3)
step 29/30, image shape : (1, 320, 320, 3)
step 30/30, image shape : (1, 320, 320, 3)
[Info] Convert complete.
[Info] Now prebuild trt-engine.
Found 30 files belonging to 1 classes.
step 1/30, image shape : (1, 320, 320, 3)
2021-10-07 14:21:54.501273: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:54.518895: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:54.520264: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:54.521836: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:54.523049: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:54.525338: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
step 2/30, image shape : (1, 320, 320, 3)
2021-10-07 14:21:54.553969: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:54.555540: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:54.556665: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:54.558038: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:54.559088: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:54.559983: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
step 3/30, image shape : (1, 320, 320, 3)
2021-10-07 14:21:54.594004: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:54.597923: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:54.599552: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:54.601033: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:54.602305: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:54.604312: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
step 4/30, image shape : (1, 320, 320, 3)
2021-10-07 14:21:54.634213: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:54.638674: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:54.640178: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:54.642291: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:54.643825: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:54.649495: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
step 5/30, image shape : (1, 320, 320, 3)
2021-10-07 14:21:54.668430: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:54.671339: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:54.672554: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:54.678498: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:54.679625: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:54.681399: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
step 6/30, image shape : (1, 320, 320, 3)
2021-10-07 14:21:54.699913: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:54.703150: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:54.705529: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:54.708076: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:54.711813: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:54.713090: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
step 7/30, image shape : (1, 320, 320, 3)
2021-10-07 14:21:54.748968: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:54.750285: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:54.751345: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:54.752453: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:54.753522: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:54.754501: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
step 8/30, image shape : (1, 320, 320, 3)
2021-10-07 14:21:54.773218: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:54.774877: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:54.776059: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:54.777162: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:54.778034: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:54.779149: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
step 9/30, image shape : (1, 320, 320, 3)
2021-10-07 14:21:54.794973: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:54.796168: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:54.797143: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:54.797970: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:54.798687: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:54.799433: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
step 10/30, image shape : (1, 320, 320, 3)
2021-10-07 14:21:54.848015: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:54.849429: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:54.850419: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:54.851397: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:54.852383: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:54.853300: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
step 11/30, image shape : (1, 320, 320, 3)
2021-10-07 14:21:54.890036: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:54.892058: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:54.905496: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:54.906838: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:54.907802: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:54.908979: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
step 12/30, image shape : (1, 320, 320, 3)
2021-10-07 14:21:54.931153: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:54.933730: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:54.935208: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:54.936988: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:54.938206: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:54.945270: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
step 13/30, image shape : (1, 320, 320, 3)
2021-10-07 14:21:54.996327: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:54.997990: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:54.999318: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.000395: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.001443: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.002548: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
step 14/30, image shape : (1, 320, 320, 3)
2021-10-07 14:21:55.026953: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.028467: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.029830: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.030897: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.033084: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.034140: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
step 15/30, image shape : (1, 320, 320, 3)
2021-10-07 14:21:55.049957: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.051390: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.052484: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.053757: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.054732: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.055674: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
step 16/30, image shape : (1, 320, 320, 3)
2021-10-07 14:21:55.072601: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.075016: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.077017: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.078947: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.080625: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.081977: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
step 17/30, image shape : (1, 320, 320, 3)
2021-10-07 14:21:55.122032: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.123231: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.124173: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.125291: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.126088: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.126915: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
step 18/30, image shape : (1, 320, 320, 3)
2021-10-07 14:21:55.142622: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.144997: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.146100: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.147124: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.147956: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.148818: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
step 19/30, image shape : (1, 320, 320, 3)
2021-10-07 14:21:55.163239: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.165280: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.167056: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.170588: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.173496: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.174742: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
step 20/30, image shape : (1, 320, 320, 3)
2021-10-07 14:21:55.191675: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.193147: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.195249: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.196687: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.197986: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.198907: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
step 21/30, image shape : (1, 320, 320, 3)
2021-10-07 14:21:55.223086: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.224427: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.225536: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.226596: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.227501: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.228680: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
step 22/30, image shape : (1, 320, 320, 3)
2021-10-07 14:21:55.243688: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.245667: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.247469: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.253290: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.255746: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.258155: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
step 23/30, image shape : (1, 320, 320, 3)
2021-10-07 14:21:55.272156: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.273568: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.274988: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.276136: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.277656: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.278708: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
step 24/30, image shape : (1, 320, 320, 3)
2021-10-07 14:21:55.299802: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.301459: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.303199: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.304482: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.305795: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.306835: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
step 25/30, image shape : (1, 320, 320, 3)
2021-10-07 14:21:55.322027: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.324808: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.326183: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.327298: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.328234: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.329430: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
step 26/30, image shape : (1, 320, 320, 3)
2021-10-07 14:21:55.347076: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.348356: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.351511: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.354776: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.357402: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.358476: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
step 27/30, image shape : (1, 320, 320, 3)
2021-10-07 14:21:55.378031: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.383370: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.386487: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.387798: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.393018: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.395249: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
step 28/30, image shape : (1, 320, 320, 3)
2021-10-07 14:21:55.426818: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.431675: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.434095: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.436594: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.437868: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.438853: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
step 29/30, image shape : (1, 320, 320, 3)
2021-10-07 14:21:55.454118: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.455327: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.456274: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.460399: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.461846: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.462862: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
step 30/30, image shape : (1, 320, 320, 3)
2021-10-07 14:21:55.475968: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.477193: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.478811: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.479840: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.480687: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
2021-10-07 14:21:55.481721: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger INVALID_ARGUMENT: Cannot set empty memory.
[Info] writing savedmodel complete.
@sushreebarsa sushreebarsa added the comp:gpu:tensorrt Issues specific to TensorRT label Oct 7, 2021
@sachinprasadhs sachinprasadhs added the stat:awaiting tensorflower Status - Awaiting response from tensorflower label Oct 8, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@bixia1 bixia1

Labels
comp:gpu:tensorrt Issues specific to TensorRT stat:awaiting tensorflower Status - Awaiting response from tensorflower type:bug Bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@bixia1 @ProtossDragoon @sachinprasadhs @sushreebarsa