Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

resource exhausted? #33

Open
mezoderma opened this issue Jan 12, 2024 · 1 comment
Open

resource exhausted? #33

mezoderma opened this issue Jan 12, 2024 · 1 comment

Comments

@mezoderma
Copy link

mezoderma commented Jan 12, 2024 •
edited
Loading

Dear Song,

Thank you for the DeepTE.
I have the following error with test dataset. The CUDA driver is installed (below). Does it mean that I need more GPU?

Thank you in advance,
Ilya

(deepte) ilbor@ilbor-MS-7C91:~/deepTE$ python ~/soft/DeepTE/DeepTE.py -d working_dir -o output_dir -i input_test.fasta -sp P -m_dir ~/miniconda3/envs/deepte/models/Plants_model/ > deepte.log
Using TensorFlow backend.
start time is 1705071754.2253158
Step1: transfer fasta data to CNN input data
Step2: classify TEs
Step2: 2) domain information is not exist

WARNING:tensorflow:From /home/ilbor/miniconda3/envs/deepte/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:517: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead.

WARNING:tensorflow:From /home/ilbor/miniconda3/envs/deepte/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:4138: The name tf.random_uniform is deprecated. Please use tf.random.uniform instead.

WARNING:tensorflow:From /home/ilbor/miniconda3/envs/deepte/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:3976: The name tf.nn.max_pool is deprecated. Please use tf.nn.max_pool2d instead.

WARNING:tensorflow:From /home/ilbor/miniconda3/envs/deepte/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:131: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead.

WARNING:tensorflow:From /home/ilbor/miniconda3/envs/deepte/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:133: The name tf.placeholder_with_default is deprecated. Please use tf.compat.v1.placeholder_with_default instead.

WARNING:tensorflow:From /home/ilbor/miniconda3/envs/deepte/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:3445: calling dropout (from tensorflow.python.ops.nn_ops) with keep_prob is deprecated and will be removed in a future version.
Instructions for updating:
Please use `rate` instead of `keep_prob`. Rate should be set to `rate = 1 - keep_prob`.
WARNING:tensorflow:From /home/ilbor/miniconda3/envs/deepte/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:174: The name tf.get_default_session is deprecated. Please use tf.compat.v1.get_default_session instead.

2024-01-12 18:02:34.381101: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA
2024-01-12 18:02:34.402921: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 3799960000 Hz
2024-01-12 18:02:34.403531: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x55a227c90350 executing computations on platform Host. Devices:
2024-01-12 18:02:34.403553: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): <undefined>, <undefined>
2024-01-12 18:02:34.404829: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcuda.so.1
2024-01-12 18:02:34.460689: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2024-01-12 18:02:34.460974: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x55a227c901a0 executing computations on platform CUDA. Devices:
2024-01-12 18:02:34.460998: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): NVIDIA GeForce GT 710, Compute Capability 3.5
2024-01-12 18:02:34.461124: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2024-01-12 18:02:34.461352: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties: 
name: NVIDIA GeForce GT 710 major: 3 minor: 5 memoryClockRate(GHz): 0.954
pciBusID: 0000:2b:00.0
2024-01-12 18:02:34.461488: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
2024-01-12 18:02:34.462408: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10
2024-01-12 18:02:34.463463: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10
2024-01-12 18:02:34.463606: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10
2024-01-12 18:02:34.464545: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10
2024-01-12 18:02:34.465080: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10
2024-01-12 18:02:34.467037: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
2024-01-12 18:02:34.467115: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2024-01-12 18:02:34.467369: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2024-01-12 18:02:34.467562: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0
2024-01-12 18:02:34.467582: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
2024-01-12 18:02:34.467879: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:
2024-01-12 18:02:34.467888: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187]      0 
2024-01-12 18:02:34.467892: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0:   N 
2024-01-12 18:02:34.467948: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2024-01-12 18:02:34.468168: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2024-01-12 18:02:34.468377: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 1394 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce GT 710, pci bus id: 0000:2b:00.0, compute capability: 3.5)
WARNING:tensorflow:From /home/ilbor/miniconda3/envs/deepte/lib/python3.6/site-packages/keras/optimizers.py:790: The name tf.train.Optimizer is deprecated. Please use tf.compat.v1.train.Optimizer instead.

WARNING:tensorflow:From /home/ilbor/miniconda3/envs/deepte/lib/python3.6/site-packages/tensorflow/python/ops/math_grad.py:1250: add_dispatch_support.<locals>.wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
2024-01-12 18:02:46.980585: W tensorflow/core/common_runtime/bfc_allocator.cc:314] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.2KiB (rounded to 1280).  Current allocation summary follows.
2024-01-12 18:02:46.980645: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (256): 	Total Chunks: 36, Chunks in use: 36. 9.0KiB allocated for chunks. 9.0KiB in use in bin. 192B client-requested in use in bin.
2024-01-12 18:02:46.980674: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (512): 	Total Chunks: 17, Chunks in use: 17. 9.8KiB allocated for chunks. 9.8KiB in use in bin. 8.3KiB client-requested in use in bin.
2024-01-12 18:02:46.980685: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (1024): 	Total Chunks: 16, Chunks in use: 16. 19.8KiB allocated for chunks. 19.8KiB in use in bin. 18.5KiB client-requested in use in bin.
2024-01-12 18:02:46.980693: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (2048): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2024-01-12 18:02:46.980702: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (4096): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2024-01-12 18:02:46.980710: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (8192): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2024-01-12 18:02:46.980718: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (16384): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2024-01-12 18:02:46.980726: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (32768): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2024-01-12 18:02:46.980735: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (65536): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2024-01-12 18:02:46.980745: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (131072): 	Total Chunks: 5, Chunks in use: 5. 880.0KiB allocated for chunks. 880.0KiB in use in bin. 878.9KiB client-requested in use in bin.
2024-01-12 18:02:46.980754: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (262144): 	Total Chunks: 4, Chunks in use: 4. 1.55MiB allocated for chunks. 1.55MiB in use in bin. 1.54MiB client-requested in use in bin.
2024-01-12 18:02:46.980762: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (524288): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2024-01-12 18:02:46.980770: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (1048576): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2024-01-12 18:02:46.980778: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (2097152): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2024-01-12 18:02:46.980786: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (4194304): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2024-01-12 18:02:46.980794: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (8388608): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2024-01-12 18:02:46.980803: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (16777216): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2024-01-12 18:02:46.980811: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (33554432): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2024-01-12 18:02:46.980819: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (67108864): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2024-01-12 18:02:46.980828: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (134217728): 	Total Chunks: 5, Chunks in use: 5. 1.10GiB allocated for chunks. 1.10GiB in use in bin. 1.10GiB client-requested in use in bin.
2024-01-12 18:02:46.980838: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (268435456): 	Total Chunks: 1, Chunks in use: 1. 267.78MiB allocated for chunks. 267.78MiB in use in bin. 224.78MiB client-requested in use in bin.
2024-01-12 18:02:46.980847: I tensorflow/core/common_runtime/bfc_allocator.cc:780] Bin for 1.2KiB was 1.0KiB, Chunk State: 
2024-01-12 18:02:46.980854: I tensorflow/core/common_runtime/bfc_allocator.cc:793] Next region of size 1461846016
2024-01-12 18:02:46.980864: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x2300d40000 next 1 of size 1280
2024-01-12 18:02:46.980871: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x2300d40500 next 2 of size 256
2024-01-12 18:02:46.980878: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x2300d40600 next 3 of size 256
2024-01-12 18:02:46.980885: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x2300d40700 next 4 of size 256
2024-01-12 18:02:46.980892: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x2300d40800 next 5 of size 256
2024-01-12 18:02:46.980899: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x2300d40900 next 6 of size 256
2024-01-12 18:02:46.980906: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x2300d40a00 next 7 of size 512
2024-01-12 18:02:46.980913: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x2300d40c00 next 8 of size 256
2024-01-12 18:02:46.980919: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x2300d40d00 next 9 of size 256
2024-01-12 18:02:46.980926: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x2300d40e00 next 10 of size 1024
2024-01-12 18:02:46.980933: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x2300d41200 next 11 of size 256
2024-01-12 18:02:46.980940: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x2300d41300 next 12 of size 256
2024-01-12 18:02:46.980947: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x2300d41400 next 13 of size 768
2024-01-12 18:02:46.980954: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x2300d41700 next 14 of size 256
2024-01-12 18:02:46.980960: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x2300d41800 next 15 of size 256
2024-01-12 18:02:46.980967: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x2300d41900 next 16 of size 512
2024-01-12 18:02:46.980973: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x2300d41b00 next 17 of size 235699200
2024-01-12 18:02:46.980979: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x230ee09700 next 18 of size 1536
2024-01-12 18:02:46.980986: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x230ee09d00 next 19 of size 405248
2024-01-12 18:02:46.980992: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x230ee6cc00 next 20 of size 1024
2024-01-12 18:02:46.980998: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x230ee6d000 next 21 of size 512
2024-01-12 18:02:46.981003: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x230ee6d200 next 22 of size 256
2024-01-12 18:02:46.981009: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x230ee6d300 next 23 of size 768
2024-01-12 18:02:46.981015: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x230ee6d600 next 24 of size 512
2024-01-12 18:02:46.981021: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x230ee6d800 next 25 of size 1280
2024-01-12 18:02:46.981027: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x230ee6dd00 next 26 of size 180224
2024-01-12 18:02:46.981033: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x230ee99d00 next 35 of size 256
2024-01-12 18:02:46.981039: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x230ee99e00 next 36 of size 1536
2024-01-12 18:02:46.981045: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x230ee9a400 next 33 of size 512
2024-01-12 18:02:46.981051: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x230ee9a600 next 34 of size 235699200
2024-01-12 18:02:46.981057: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x231cf62200 next 29 of size 1024
2024-01-12 18:02:46.981063: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x231cf62600 next 30 of size 405248
2024-01-12 18:02:46.981068: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x231cfc5500 next 32 of size 768
2024-01-12 18:02:46.981074: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x231cfc5800 next 31 of size 256
2024-01-12 18:02:46.981080: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x231cfc5900 next 27 of size 180224
2024-01-12 18:02:46.981086: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x231cff1900 next 28 of size 512
2024-01-12 18:02:46.981092: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x231cff1b00 next 37 of size 1280
2024-01-12 18:02:46.981097: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x231cff2000 next 38 of size 256
2024-01-12 18:02:46.981103: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x231cff2100 next 39 of size 256
2024-01-12 18:02:46.981109: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x231cff2200 next 40 of size 256
2024-01-12 18:02:46.981115: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x231cff2300 next 41 of size 256
2024-01-12 18:02:46.981120: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x231cff2400 next 42 of size 256
2024-01-12 18:02:46.981126: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x231cff2500 next 43 of size 1536
2024-01-12 18:02:46.981132: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x231cff2b00 next 44 of size 1280
2024-01-12 18:02:46.981138: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x231cff3000 next 45 of size 180224
2024-01-12 18:02:46.981143: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x231d01f000 next 46 of size 256
2024-01-12 18:02:46.981149: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x231d01f100 next 47 of size 256
2024-01-12 18:02:46.981155: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x231d01f200 next 48 of size 256
2024-01-12 18:02:46.981161: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x231d01f300 next 49 of size 1280
2024-01-12 18:02:46.981167: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x231d01f800 next 50 of size 256
2024-01-12 18:02:46.981172: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x231d01f900 next 51 of size 235699200
2024-01-12 18:02:46.981178: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x232b0e7500 next 52 of size 256
2024-01-12 18:02:46.981184: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x232b0e7600 next 53 of size 1536
2024-01-12 18:02:46.981190: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x232b0e7c00 next 54 of size 256
2024-01-12 18:02:46.981195: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x232b0e7d00 next 55 of size 235699200
2024-01-12 18:02:46.981201: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x23391af900 next 56 of size 256
2024-01-12 18:02:46.981207: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x23391afa00 next 57 of size 256
2024-01-12 18:02:46.981213: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x23391afb00 next 58 of size 256
2024-01-12 18:02:46.981218: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x23391afc00 next 59 of size 256
2024-01-12 18:02:46.981224: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x23391afd00 next 60 of size 405248
2024-01-12 18:02:46.981230: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x2339212c00 next 61 of size 256
2024-01-12 18:02:46.981236: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x2339212d00 next 62 of size 256
2024-01-12 18:02:46.981242: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x2339212e00 next 63 of size 180224
2024-01-12 18:02:46.981247: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x233923ee00 next 64 of size 256
2024-01-12 18:02:46.981253: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x233923ef00 next 65 of size 512
2024-01-12 18:02:46.981259: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x233923f100 next 66 of size 512
2024-01-12 18:02:46.981265: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x233923f300 next 67 of size 405248
2024-01-12 18:02:46.981270: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x23392a2200 next 68 of size 256
2024-01-12 18:02:46.981276: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x23392a2300 next 69 of size 512
2024-01-12 18:02:46.981282: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x23392a2500 next 70 of size 512
2024-01-12 18:02:46.981288: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x23392a2700 next 71 of size 256
2024-01-12 18:02:46.981293: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x23392a2800 next 72 of size 256
2024-01-12 18:02:46.981299: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x23392a2900 next 73 of size 768
2024-01-12 18:02:46.981305: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x23392a2c00 next 74 of size 768
2024-01-12 18:02:46.981311: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x23392a2f00 next 75 of size 256
2024-01-12 18:02:46.981317: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x23392a3000 next 76 of size 1024
2024-01-12 18:02:46.981322: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x23392a3400 next 77 of size 1024
2024-01-12 18:02:46.981328: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x23392a3800 next 78 of size 1024
2024-01-12 18:02:46.981335: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x23392a3c00 next 79 of size 1536
2024-01-12 18:02:46.981344: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x23392a4200 next 80 of size 235699200
2024-01-12 18:02:46.981349: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x234736be00 next 81 of size 180224
2024-01-12 18:02:46.981354: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x2347397e00 next 82 of size 512
2024-01-12 18:02:46.981360: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x2347398000 next 83 of size 512
2024-01-12 18:02:46.981367: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x2347398200 next 18446744073709551615 of size 280788480
2024-01-12 18:02:46.981372: I tensorflow/core/common_runtime/bfc_allocator.cc:809]      Summary of in-use Chunks by size: 
2024-01-12 18:02:46.981380: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 36 Chunks of size 256 totalling 9.0KiB
2024-01-12 18:02:46.981387: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 12 Chunks of size 512 totalling 6.0KiB
2024-01-12 18:02:46.981393: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 5 Chunks of size 768 totalling 3.8KiB
2024-01-12 18:02:46.981398: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 6 Chunks of size 1024 totalling 6.0KiB
2024-01-12 18:02:46.981404: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 5 Chunks of size 1280 totalling 6.2KiB
2024-01-12 18:02:46.981409: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 5 Chunks of size 1536 totalling 7.5KiB
2024-01-12 18:02:46.981415: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 5 Chunks of size 180224 totalling 880.0KiB
2024-01-12 18:02:46.981420: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 4 Chunks of size 405248 totalling 1.55MiB
2024-01-12 18:02:46.981425: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 5 Chunks of size 235699200 totalling 1.10GiB
2024-01-12 18:02:46.981431: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 1 Chunks of size 280788480 totalling 267.78MiB
2024-01-12 18:02:46.981436: I tensorflow/core/common_runtime/bfc_allocator.cc:816] Sum Total of in-use chunks: 1.36GiB
2024-01-12 18:02:46.981441: I tensorflow/core/common_runtime/bfc_allocator.cc:818] total_region_allocated_bytes_: 1461846016 memory_limit_: 1461846016 available bytes: 0 curr_region_allocation_bytes_: 2923692032
2024-01-12 18:02:46.981451: I tensorflow/core/common_runtime/bfc_allocator.cc:824] Stats: 
Limit:                  1461846016
InUse:                  1461846016
MaxInUse:               1461846016
NumAllocs:                      94
MaxAllocSize:            280788480

2024-01-12 18:02:46.981462: W tensorflow/core/common_runtime/bfc_allocator.cc:319] *************************************************************************************************xxx
Traceback (most recent call last):
  File "/home/ilbor/miniconda3/envs/deepte/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1356, in _do_call
    return fn(*args)
  File "/home/ilbor/miniconda3/envs/deepte/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1341, in _run_fn
    options, feed_dict, fetch_list, target_list, run_metadata)
  File "/home/ilbor/miniconda3/envs/deepte/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1429, in _call_tf_sessionrun
    run_metadata)
tensorflow.python.framework.errors_impl.InternalError: 2 root error(s) found.
  (0) Internal: Dst tensor is not initialized.
	 [[{{node _arg_Placeholder_11_0_1}}]]
	 [[Assign_20/_275]]
  (1) Internal: Dst tensor is not initialized.
	 [[{{node _arg_Placeholder_11_0_1}}]]
0 successful operations.
0 derived errors ignored.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ilbor/soft/DeepTE/DeepTE.py", line 274, in <module>
    main()
  File "/home/ilbor/soft/DeepTE/DeepTE.py", line 249, in main
    pipeline_no_m.classify_pipeline(model_dir, input_CNN_data_file, temp_store_opt_dir, sp_type,te_fam,prop_thr)
  File "/home/ilbor/soft/DeepTE/scripts/DeepTE_pipeline_no_modification.py", line 339, in classify_pipeline
    predict_te(model_file_dic[model_name], model_name, x_all_test_list, y_all_test_nm_list,input_spe_type,y_all_test_nm_list,prop_thr)
  File "/home/ilbor/soft/DeepTE/scripts/DeepTE_pipeline_no_modification.py", line 210, in predict_te
    model = load_model(model)
  File "/home/ilbor/miniconda3/envs/deepte/lib/python3.6/site-packages/keras/engine/saving.py", line 419, in load_model
    model = _deserialize_model(f, custom_objects, compile)
  File "/home/ilbor/miniconda3/envs/deepte/lib/python3.6/site-packages/keras/engine/saving.py", line 325, in _deserialize_model
    model.optimizer.set_weights(optimizer_weight_values)
  File "/home/ilbor/miniconda3/envs/deepte/lib/python3.6/site-packages/keras/optimizers.py", line 134, in set_weights
    K.batch_set_value(weight_value_tuples)
  File "/home/ilbor/miniconda3/envs/deepte/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py", line 2470, in batch_set_value
    get_session().run(assign_ops, feed_dict=feed_dict)
  File "/home/ilbor/miniconda3/envs/deepte/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 950, in run
    run_metadata_ptr)
  File "/home/ilbor/miniconda3/envs/deepte/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1173, in _run
    feed_dict_tensor, options, run_metadata)
  File "/home/ilbor/miniconda3/envs/deepte/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1350, in _do_run
    run_metadata)
  File "/home/ilbor/miniconda3/envs/deepte/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1370, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InternalError: 2 root error(s) found.
  (0) Internal: Dst tensor is not initialized.
	 [[{{node _arg_Placeholder_11_0_1}}]]
	 [[Assign_20/_275]]
  (1) Internal: Dst tensor is not initialized.
	 [[{{node _arg_Placeholder_11_0_1}}]]
0 successful operations.
0 derived errors ignored.

(deepte) ilbor@ilbor-MS-7C91:~/deepTE$ nvidia-smi
Fri Jan 12 17:43:51 2024       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.223.02   Driver Version: 470.223.02   CUDA Version: 11.4     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  Off  | 00000000:2B:00.0 N/A |                  N/A |
| 33%   41C    P0    N/A /  N/A |    397MiB /  1993MiB |     N/A      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+
@mezoderma
Copy link
Author

Although with my own data (1.2 Mb vs 6 Mb test data) I got another one error:

(deepte) ilbor@ilbor-MS-7C91:~/deepTE$ python ~/soft/DeepTE/DeepTE.py -d working_dir -o output_dir -i consensi.fa.classified -sp M -m_dir ~/miniconda3/envs/deepte/models/Metazoans_model/
Using TensorFlow backend.
start time is 1705069063.0767448
Step1: transfer fasta data to CNN input data
Step2: classify TEs
Step2: 2) domain information is not exist
WARNING:tensorflow:From /home/ilbor/miniconda3/envs/deepte/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:517: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead.

WARNING:tensorflow:From /home/ilbor/miniconda3/envs/deepte/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:4138: The name tf.random_uniform is deprecated. Please use tf.random.uniform instead.

WARNING:tensorflow:From /home/ilbor/miniconda3/envs/deepte/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:3976: The name tf.nn.max_pool is deprecated. Please use tf.nn.max_pool2d instead.

WARNING:tensorflow:From /home/ilbor/miniconda3/envs/deepte/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:131: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead.

WARNING:tensorflow:From /home/ilbor/miniconda3/envs/deepte/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:133: The name tf.placeholder_with_default is deprecated. Please use tf.compat.v1.placeholder_with_default instead.

WARNING:tensorflow:From /home/ilbor/miniconda3/envs/deepte/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:3445: calling dropout (from tensorflow.python.ops.nn_ops) with keep_prob is deprecated and will be removed in a future version.
Instructions for updating:
Please use `rate` instead of `keep_prob`. Rate should be set to `rate = 1 - keep_prob`.
WARNING:tensorflow:From /home/ilbor/miniconda3/envs/deepte/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:174: The name tf.get_default_session is deprecated. Please use tf.compat.v1.get_default_session instead.

2024-01-12 17:17:43.223107: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA
2024-01-12 17:17:43.246928: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 3799960000 Hz
2024-01-12 17:17:43.247580: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x55a56f2514e0 executing computations on platform Host. Devices:
2024-01-12 17:17:43.247602: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): <undefined>, <undefined>
2024-01-12 17:17:43.248847: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcuda.so.1
2024-01-12 17:17:43.312145: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2024-01-12 17:17:43.312433: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x55a56f251330 executing computations on platform CUDA. Devices:
2024-01-12 17:17:43.312460: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): NVIDIA GeForce GT 710, Compute Capability 3.5
2024-01-12 17:17:43.312583: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2024-01-12 17:17:43.312806: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties: 
name: NVIDIA GeForce GT 710 major: 3 minor: 5 memoryClockRate(GHz): 0.954
pciBusID: 0000:2b:00.0
2024-01-12 17:17:43.312956: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
2024-01-12 17:17:43.313858: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10
2024-01-12 17:17:43.314896: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10
2024-01-12 17:17:43.315037: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10
2024-01-12 17:17:43.315941: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10
2024-01-12 17:17:43.316455: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10
2024-01-12 17:17:43.318378: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
2024-01-12 17:17:43.318446: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2024-01-12 17:17:43.318694: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2024-01-12 17:17:43.318888: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0
2024-01-12 17:17:43.318908: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
2024-01-12 17:17:43.319468: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:
2024-01-12 17:17:43.319478: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187]      0 
2024-01-12 17:17:43.319481: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0:   N 
2024-01-12 17:17:43.319541: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2024-01-12 17:17:43.319758: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2024-01-12 17:17:43.319966: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 1428 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce GT 710, pci bus id: 0000:2b:00.0, compute capability: 3.5)
WARNING:tensorflow:From /home/ilbor/miniconda3/envs/deepte/lib/python3.6/site-packages/keras/optimizers.py:790: The name tf.train.Optimizer is deprecated. Please use tf.compat.v1.train.Optimizer instead.

WARNING:tensorflow:From /home/ilbor/miniconda3/envs/deepte/lib/python3.6/site-packages/tensorflow/python/ops/math_grad.py:1250: add_dispatch_support.<locals>.wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
2024-01-12 17:17:55.812661: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10
2024-01-12 17:17:55.901100: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
2024-01-12 17:17:57.287230: W tensorflow/core/common_runtime/bfc_allocator.cc:237] Allocator (GPU_0_bfc) ran out of memory trying to allocate 299.89MiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2024-01-12 17:19:03.452000: W tensorflow/core/common_runtime/bfc_allocator.cc:314] Allocator (GPU_0_bfc) ran out of memory trying to allocate 900B (rounded to 1024).  Current allocation summary follows.
2024-01-12 17:19:03.452051: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (256): 	Total Chunks: 51, Chunks in use: 51. 12.8KiB allocated for chunks. 12.8KiB in use in bin. 264B client-requested in use in bin.
2024-01-12 17:19:03.452063: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (512): 	Total Chunks: 22, Chunks in use: 22. 12.8KiB allocated for chunks. 12.8KiB in use in bin. 10.8KiB client-requested in use in bin.
2024-01-12 17:19:03.452073: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (1024): 	Total Chunks: 19, Chunks in use: 19. 22.5KiB allocated for chunks. 22.5KiB in use in bin. 21.0KiB client-requested in use in bin.
2024-01-12 17:19:03.452082: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (2048): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2024-01-12 17:19:03.452091: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (4096): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2024-01-12 17:19:03.452099: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (8192): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2024-01-12 17:19:03.452109: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (16384): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2024-01-12 17:19:03.452117: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (32768): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2024-01-12 17:19:03.452125: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (65536): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2024-01-12 17:19:03.452135: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (131072): 	Total Chunks: 5, Chunks in use: 5. 880.0KiB allocated for chunks. 880.0KiB in use in bin. 878.9KiB client-requested in use in bin.
2024-01-12 17:19:03.452145: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (262144): 	Total Chunks: 5, Chunks in use: 5. 1.93MiB allocated for chunks. 1.93MiB in use in bin. 1.93MiB client-requested in use in bin.
2024-01-12 17:19:03.452155: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (524288): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2024-01-12 17:19:03.452163: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (1048576): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2024-01-12 17:19:03.452172: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (2097152): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2024-01-12 17:19:03.452181: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (4194304): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2024-01-12 17:19:03.452189: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (8388608): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2024-01-12 17:19:03.452198: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (16777216): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2024-01-12 17:19:03.452206: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (33554432): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2024-01-12 17:19:03.452215: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (67108864): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2024-01-12 17:19:03.452224: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (134217728): 	Total Chunks: 5, Chunks in use: 5. 1.10GiB allocated for chunks. 1.10GiB in use in bin. 1.10GiB client-requested in use in bin.
2024-01-12 17:19:03.452234: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (268435456): 	Total Chunks: 1, Chunks in use: 1. 301.76MiB allocated for chunks. 301.76MiB in use in bin. 224.78MiB client-requested in use in bin.
2024-01-12 17:19:03.452243: I tensorflow/core/common_runtime/bfc_allocator.cc:780] Bin for 1.0KiB was 1.0KiB, Chunk State: 
2024-01-12 17:19:03.452251: I tensorflow/core/common_runtime/bfc_allocator.cc:793] Next region of size 1497890816
2024-01-12 17:19:03.452261: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x2300d40000 next 1 of size 1280
2024-01-12 17:19:03.452269: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x2300d40500 next 2 of size 256
2024-01-12 17:19:03.452276: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x2300d40600 next 3 of size 256
2024-01-12 17:19:03.452283: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x2300d40700 next 4 of size 256
2024-01-12 17:19:03.452290: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x2300d40800 next 5 of size 256
2024-01-12 17:19:03.452297: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x2300d40900 next 6 of size 256
2024-01-12 17:19:03.452305: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x2300d40a00 next 7 of size 512
2024-01-12 17:19:03.452313: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x2300d40c00 next 8 of size 256
2024-01-12 17:19:03.452320: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x2300d40d00 next 9 of size 256
2024-01-12 17:19:03.452327: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x2300d40e00 next 10 of size 1024
2024-01-12 17:19:03.452334: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x2300d41200 next 11 of size 256
2024-01-12 17:19:03.452342: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x2300d41300 next 12 of size 256
2024-01-12 17:19:03.452349: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x2300d41400 next 13 of size 768
2024-01-12 17:19:03.452356: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x2300d41700 next 14 of size 256
2024-01-12 17:19:03.452363: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x2300d41800 next 15 of size 256
2024-01-12 17:19:03.452370: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x2300d41900 next 16 of size 512
2024-01-12 17:19:03.452377: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x2300d41b00 next 17 of size 512
2024-01-12 17:19:03.452384: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x2300d41d00 next 18 of size 1280
2024-01-12 17:19:03.452391: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x2300d42200 next 19 of size 180224
2024-01-12 17:19:03.452399: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x2300d6e200 next 20 of size 405248
2024-01-12 17:19:03.452407: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x2300dd1100 next 21 of size 1536
2024-01-12 17:19:03.452414: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x2300dd1700 next 22 of size 256
2024-01-12 17:19:03.452421: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x2300dd1800 next 23 of size 512
2024-01-12 17:19:03.452429: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x2300dd1a00 next 24 of size 235699200
2024-01-12 17:19:03.452436: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x230ee99600 next 25 of size 1024
2024-01-12 17:19:03.452443: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x230ee99a00 next 26 of size 768
2024-01-12 17:19:03.452450: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x230ee99d00 next 35 of size 256
2024-01-12 17:19:03.452457: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x230ee99e00 next 36 of size 1536
2024-01-12 17:19:03.452464: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x230ee9a400 next 32 of size 512
2024-01-12 17:19:03.452471: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x230ee9a600 next 33 of size 235699200
2024-01-12 17:19:03.452478: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x231cf62200 next 31 of size 1024
2024-01-12 17:19:03.452485: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x231cf62600 next 27 of size 405248
2024-01-12 17:19:03.452492: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x231cfc5500 next 28 of size 768
2024-01-12 17:19:03.452499: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x231cfc5800 next 29 of size 256
2024-01-12 17:19:03.452506: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x231cfc5900 next 34 of size 180224
2024-01-12 17:19:03.452513: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x231cff1900 next 30 of size 512
2024-01-12 17:19:03.452520: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x231cff1b00 next 37 of size 1280
2024-01-12 17:19:03.452527: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x231cff2000 next 38 of size 256
2024-01-12 17:19:03.452534: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x231cff2100 next 39 of size 256
2024-01-12 17:19:03.452541: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x231cff2200 next 40 of size 256
2024-01-12 17:19:03.452548: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x231cff2300 next 41 of size 256
2024-01-12 17:19:03.452554: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x231cff2400 next 42 of size 256
2024-01-12 17:19:03.452561: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x231cff2500 next 43 of size 256
2024-01-12 17:19:03.452568: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x231cff2600 next 44 of size 1024
2024-01-12 17:19:03.452575: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x231cff2a00 next 45 of size 1024
2024-01-12 17:19:03.452582: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x231cff2e00 next 46 of size 512
2024-01-12 17:19:03.452589: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x231cff3000 next 47 of size 256
2024-01-12 17:19:03.452596: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x231cff3100 next 48 of size 256
2024-01-12 17:19:03.452603: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x231cff3200 next 49 of size 256
2024-01-12 17:19:03.452610: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x231cff3300 next 50 of size 256
2024-01-12 17:19:03.452617: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x231cff3400 next 51 of size 256
2024-01-12 17:19:03.452624: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x231cff3500 next 52 of size 180224
2024-01-12 17:19:03.452631: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x231d01f500 next 53 of size 1280
2024-01-12 17:19:03.452638: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x231d01fa00 next 54 of size 235699200
2024-01-12 17:19:03.452645: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x232b0e7600 next 55 of size 256
2024-01-12 17:19:03.452652: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x232b0e7700 next 56 of size 256
2024-01-12 17:19:03.452658: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x232b0e7800 next 57 of size 256
2024-01-12 17:19:03.452665: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x232b0e7900 next 58 of size 1536
2024-01-12 17:19:03.452672: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x232b0e7f00 next 59 of size 235699200
2024-01-12 17:19:03.452679: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x23391afb00 next 60 of size 256
2024-01-12 17:19:03.452686: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x23391afc00 next 61 of size 1536
2024-01-12 17:19:03.452693: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x23391b0200 next 62 of size 1280
2024-01-12 17:19:03.452700: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x23391b0700 next 63 of size 512
2024-01-12 17:19:03.452707: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x23391b0900 next 64 of size 405248
2024-01-12 17:19:03.452714: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x2339213800 next 65 of size 256
2024-01-12 17:19:03.452721: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x2339213900 next 66 of size 256
2024-01-12 17:19:03.452728: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x2339213a00 next 67 of size 180224
2024-01-12 17:19:03.452735: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x233923fa00 next 68 of size 256
2024-01-12 17:19:03.452742: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x233923fb00 next 69 of size 405248
2024-01-12 17:19:03.452749: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x23392a2a00 next 70 of size 256
2024-01-12 17:19:03.452756: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x23392a2b00 next 71 of size 512
2024-01-12 17:19:03.452762: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x23392a2d00 next 72 of size 512
2024-01-12 17:19:03.452768: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x23392a2f00 next 73 of size 768
2024-01-12 17:19:03.452773: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x23392a3200 next 74 of size 768
2024-01-12 17:19:03.452779: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x23392a3500 next 75 of size 256
2024-01-12 17:19:03.452785: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x23392a3600 next 76 of size 256
2024-01-12 17:19:03.452791: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x23392a3700 next 77 of size 256
2024-01-12 17:19:03.452797: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x23392a3800 next 90 of size 256
2024-01-12 17:19:03.452803: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x23392a3900 next 93 of size 256
2024-01-12 17:19:03.452809: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x23392a3a00 next 84 of size 256
2024-01-12 17:19:03.452815: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x23392a3b00 next 94 of size 256
2024-01-12 17:19:03.452820: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x23392a3c00 next 95 of size 256
2024-01-12 17:19:03.452826: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x23392a3d00 next 79 of size 256
2024-01-12 17:19:03.452832: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x23392a3e00 next 81 of size 256
2024-01-12 17:19:03.452838: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x23392a3f00 next 92 of size 512
2024-01-12 17:19:03.452844: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x23392a4100 next 80 of size 256
2024-01-12 17:19:03.452849: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x23392a4200 next 85 of size 256
2024-01-12 17:19:03.452855: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x23392a4300 next 86 of size 1024
2024-01-12 17:19:03.452861: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x23392a4700 next 88 of size 256
2024-01-12 17:19:03.452866: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x23392a4800 next 89 of size 256
2024-01-12 17:19:03.452872: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x23392a4900 next 87 of size 768
2024-01-12 17:19:03.452878: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x23392a4c00 next 83 of size 256
2024-01-12 17:19:03.452884: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x23392a4d00 next 91 of size 256
2024-01-12 17:19:03.452890: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x23392a4e00 next 82 of size 512
2024-01-12 17:19:03.452895: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x23392a5000 next 78 of size 512
2024-01-12 17:19:03.452901: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x23392a5200 next 96 of size 405248
2024-01-12 17:19:03.452907: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x2339308100 next 97 of size 180224
2024-01-12 17:19:03.452913: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x2339334100 next 98 of size 235699200
2024-01-12 17:19:03.452919: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x23473fbd00 next 99 of size 256
2024-01-12 17:19:03.452924: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x23473fbe00 next 100 of size 768
2024-01-12 17:19:03.452930: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x23473fc100 next 101 of size 1024
2024-01-12 17:19:03.452936: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x23473fc500 next 102 of size 1024
2024-01-12 17:19:03.452943: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x23473fc900 next 103 of size 1280
2024-01-12 17:19:03.452948: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x23473fce00 next 104 of size 512
2024-01-12 17:19:03.452954: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x23473fd000 next 111 of size 256
2024-01-12 17:19:03.452960: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x23473fd100 next 112 of size 1024
2024-01-12 17:19:03.452966: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x23473fd500 next 110 of size 512
2024-01-12 17:19:03.452972: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0x23473fd700 next 18446744073709551615 of size 316418304
2024-01-12 17:19:03.452979: I tensorflow/core/common_runtime/bfc_allocator.cc:809]      Summary of in-use Chunks by size: 
2024-01-12 17:19:03.452986: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 51 Chunks of size 256 totalling 12.8KiB
2024-01-12 17:19:03.452994: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 15 Chunks of size 512 totalling 7.5KiB
2024-01-12 17:19:03.453000: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 7 Chunks of size 768 totalling 5.2KiB
2024-01-12 17:19:03.453006: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 9 Chunks of size 1024 totalling 9.0KiB
2024-01-12 17:19:03.453012: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 6 Chunks of size 1280 totalling 7.5KiB
2024-01-12 17:19:03.453018: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 4 Chunks of size 1536 totalling 6.0KiB
2024-01-12 17:19:03.453024: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 5 Chunks of size 180224 totalling 880.0KiB
2024-01-12 17:19:03.453031: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 5 Chunks of size 405248 totalling 1.93MiB
2024-01-12 17:19:03.453037: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 5 Chunks of size 235699200 totalling 1.10GiB
2024-01-12 17:19:03.453044: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 1 Chunks of size 316418304 totalling 301.76MiB
2024-01-12 17:19:03.453050: I tensorflow/core/common_runtime/bfc_allocator.cc:816] Sum Total of in-use chunks: 1.39GiB
2024-01-12 17:19:03.453056: I tensorflow/core/common_runtime/bfc_allocator.cc:818] total_region_allocated_bytes_: 1497890816 memory_limit_: 1497890816 available bytes: 0 curr_region_allocation_bytes_: 2995781632
2024-01-12 17:19:03.453068: I tensorflow/core/common_runtime/bfc_allocator.cc:824] Stats: 
Limit:                  1497890816
InUse:                  1497890816
MaxInUse:               1497890816
NumAllocs:                    1314
MaxAllocSize:            316421120

2024-01-12 17:19:03.453081: W tensorflow/core/common_runtime/bfc_allocator.cc:319] ***********************************************************************************************xxxxx
2024-01-12 17:19:03.453107: W tensorflow/core/framework/op_kernel.cc:1479] OP_REQUIRES failed at constant_op.cc:77 : Resource exhausted: OOM when allocating tensor of shape [225] and type float
2024-01-12 17:19:03.453134: E tensorflow/core/common_runtime/executor.cc:641] Executor failed to create kernel. Resource exhausted: OOM when allocating tensor of shape [225] and type float
	 [[{{node training_1/Adam/zeros_5}}]]
Traceback (most recent call last):
  File "/home/ilbor/miniconda3/envs/deepte/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1356, in _do_call
    return fn(*args)
  File "/home/ilbor/miniconda3/envs/deepte/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1341, in _run_fn
    options, feed_dict, fetch_list, target_list, run_metadata)
  File "/home/ilbor/miniconda3/envs/deepte/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1429, in _call_tf_sessionrun
    run_metadata)
tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor of shape [225] and type float
	 [[{{node training_1/Adam/zeros_5}}]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ilbor/soft/DeepTE/DeepTE.py", line 274, in <module>
    main()
  File "/home/ilbor/soft/DeepTE/DeepTE.py", line 249, in main
    pipeline_no_m.classify_pipeline(model_dir, input_CNN_data_file, temp_store_opt_dir, sp_type,te_fam,prop_thr)
  File "/home/ilbor/soft/DeepTE/scripts/DeepTE_pipeline_no_modification.py", line 401, in classify_pipeline
    predict_te(model_file_dic[model_name], model_name, x_classI_ipt_test_list, y_classI_ipt_test_nm_list,input_spe_type,y_classI_ipt_test_nm_list,prop_thr)
  File "/home/ilbor/soft/DeepTE/scripts/DeepTE_pipeline_no_modification.py", line 210, in predict_te
    model = load_model(model)
  File "/home/ilbor/miniconda3/envs/deepte/lib/python3.6/site-packages/keras/engine/saving.py", line 419, in load_model
    model = _deserialize_model(f, custom_objects, compile)
  File "/home/ilbor/miniconda3/envs/deepte/lib/python3.6/site-packages/keras/engine/saving.py", line 325, in _deserialize_model
    model.optimizer.set_weights(optimizer_weight_values)
  File "/home/ilbor/miniconda3/envs/deepte/lib/python3.6/site-packages/keras/optimizers.py", line 126, in set_weights
    param_values = K.batch_get_value(params)
  File "/home/ilbor/miniconda3/envs/deepte/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py", line 2420, in batch_get_value
    return get_session().run(ops)
  File "/home/ilbor/miniconda3/envs/deepte/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py", line 206, in get_session
    session.run(tf.variables_initializer(uninitialized_vars))
  File "/home/ilbor/miniconda3/envs/deepte/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 950, in run
    run_metadata_ptr)
  File "/home/ilbor/miniconda3/envs/deepte/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1173, in _run
    feed_dict_tensor, options, run_metadata)
  File "/home/ilbor/miniconda3/envs/deepte/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1350, in _do_run
    run_metadata)
  File "/home/ilbor/miniconda3/envs/deepte/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1370, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor of shape [225] and type float
	 [[node training_1/Adam/zeros_5 (defined at /miniconda3/envs/deepte/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:702) ]]

Original stack trace for 'training_1/Adam/zeros_5':
  File "/soft/DeepTE/DeepTE.py", line 274, in <module>
    main()
  File "/soft/DeepTE/DeepTE.py", line 249, in main
    pipeline_no_m.classify_pipeline(model_dir, input_CNN_data_file, temp_store_opt_dir, sp_type,te_fam,prop_thr)
  File "/soft/DeepTE/scripts/DeepTE_pipeline_no_modification.py", line 401, in classify_pipeline
    predict_te(model_file_dic[model_name], model_name, x_classI_ipt_test_list, y_classI_ipt_test_nm_list,input_spe_type,y_classI_ipt_test_nm_list,prop_thr)
  File "/soft/DeepTE/scripts/DeepTE_pipeline_no_modification.py", line 210, in predict_te
    model = load_model(model)
  File "/miniconda3/envs/deepte/lib/python3.6/site-packages/keras/engine/saving.py", line 419, in load_model
    model = _deserialize_model(f, custom_objects, compile)
  File "/miniconda3/envs/deepte/lib/python3.6/site-packages/keras/engine/saving.py", line 317, in _deserialize_model
    model._make_train_function()
  File "/miniconda3/envs/deepte/lib/python3.6/site-packages/keras/engine/training.py", line 509, in _make_train_function
    loss=self.total_loss)
  File "/miniconda3/envs/deepte/lib/python3.6/site-packages/keras/legacy/interfaces.py", line 91, in wrapper
    return func(*args, **kwargs)
  File "/miniconda3/envs/deepte/lib/python3.6/site-packages/keras/optimizers.py", line 487, in get_updates
    ms = [K.zeros(K.int_shape(p), dtype=K.dtype(p)) for p in params]
  File "/miniconda3/envs/deepte/lib/python3.6/site-packages/keras/optimizers.py", line 487, in <listcomp>
    ms = [K.zeros(K.int_shape(p), dtype=K.dtype(p)) for p in params]
  File "/miniconda3/envs/deepte/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py", line 702, in zeros
    v = tf.zeros(shape=shape, dtype=tf_dtype, name=name)
  File "/miniconda3/envs/deepte/lib/python3.6/site-packages/tensorflow/python/ops/array_ops.py", line 1871, in zeros
    output = _constant_if_small(zero, shape, dtype, name)
  File "/miniconda3/envs/deepte/lib/python3.6/site-packages/tensorflow/python/ops/array_ops.py", line 1829, in _constant_if_small
    return constant(value, shape=shape, dtype=dtype, name=name)
  File "/miniconda3/envs/deepte/lib/python3.6/site-packages/tensorflow/python/framework/constant_op.py", line 246, in constant
    allow_broadcast=True)
  File "/miniconda3/envs/deepte/lib/python3.6/site-packages/tensorflow/python/framework/constant_op.py", line 290, in _constant_impl
    name=name).outputs[0]
  File "/miniconda3/envs/deepte/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py", line 507, in new_func
    return func(*args, **kwargs)
  File "/miniconda3/envs/deepte/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3616, in create_op
    op_def=op_def)
  File "/miniconda3/envs/deepte/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 2005, in __init__
    self._traceback = tf_stack.extract_stack()

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@mezoderma