COM prediction values are NaN #128
Comments
|
Below is my com-predict log.
Predicting on sample 0 |
|
Have you fixed this issue? I've used 1) the provided weight (weights.250-0.00036.hdf5) as well as 2) a newly generated fintuned weight (using the provided pretrained weights.rat.COM.hdf5 and the dannce.mat), however, I always get NaN values for com-predict. (99 NaNs out of 100 predictions). I thought it may have to do with my env setting, but 1) predicting dannce with the provided weight (weights.12000-0.00014.hdf5) and 2) finetuning/predicting (weights.rat.MAX.6cam.hdf5 with the dannce.mat) seem to work fine. I'm puzzled with the problems I'm having with COM. |
|
I fixed this issue by reinstalling dannce with the from the development branch, which had TF2.4 |
Hi! I find the predicted com data are all NaN in the com3d0.mat. the parameters was set as below. Is there any problem in the video data loading?
I use dannce demo video data, the result was fine, but using my own 30s data, the result is NaN. Could anyone do me a favor to check what the problems? Thank you so much!!!
`
(dannce113) F:\testdannce120\dannce\demo\new919>com-train .\com_config_919.yaml
2022-10-15 18:13:11.561307: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library cudart64_110.dll
downfac not found in io.yaml file, falling back to main config
extension not found in io.yaml file, falling back to main config
io_config not found in io.yaml file, falling back to main config
crop_height not found in io.yaml file, falling back to main config
crop_width not found in io.yaml file, falling back to main config
n_channels_in not found in io.yaml file, falling back to main config
camnames not found in io.yaml file, falling back to main config
n_views not found in io.yaml file, falling back to main config
n_channels_out not found in io.yaml file, falling back to main config
batch_size not found in io.yaml file, falling back to main config
sigma not found in io.yaml file, falling back to main config
epochs not found in io.yaml file, falling back to main config
verbose not found in io.yaml file, falling back to main config
loss not found in io.yaml file, falling back to main config
lr not found in io.yaml file, falling back to main config
net not found in io.yaml file, falling back to main config
vid_dir_flag not found in io.yaml file, falling back to main config
metric not found in io.yaml file, falling back to main config
num_validation_per_exp not found in io.yaml file, falling back to main config
debug not found in io.yaml file, falling back to main config
max_num_samples not found in io.yaml file, falling back to main config
train_mode not found in io.yaml file, falling back to main config
com_finetune_weights not found in io.yaml file, falling back to main config
com_train_dir set to: .\COM\train_results
com_predict_dir set to: .\COM\predict_results
dannce_train_dir set to: .\DANNCE\train_results\AVG
dannce_predict_dir set to: .\DANNCE\predict_results
exp set to: [{'label3d_file': './20221015_173137_Label3D_dannce.mat'}]
downfac set to: 4
extension set to: .avi
io_config set to: io.yaml
crop_height set to: [0, 1152]
crop_width set to: [0, 1920]
n_channels_in set to: 1
camnames set to: ['Camera1', 'Camera2', 'Camera3']
n_views set to: 3
n_channels_out set to: 1
batch_size set to: 2
sigma set to: 18
epochs set to: 10
verbose set to: 1
loss set to: mask_nan_keep_loss
lr set to: 5e-5
net set to: unet2d_fullbn
vid_dir_flag set to: False
metric set to: mse
num_validation_per_exp set to: 10
debug set to: False
max_num_samples set to: 100
train_mode set to: finetune
com_finetune_weights set to: ..\markerless_mouse_1\COM\weights
base_config set to: .\com_config_919.yaml
viddir set to: videos
gpu_id set to: 0
immode set to: vid
mono set to: False
mirror set to: False
num_train_per_exp set to: None
augment_hue set to: False
augment_brightness set to: False
augment_hue_val set to: 0.05
augment_bright_val set to: 0.05
augment_rotation_val set to: 5
data_split_seed set to: None
valid_exp set to: None
dsmode set to: nn
augment_shift set to: False
augment_zoom set to: False
augment_shear set to: False
augment_rotation set to: False
augment_shear_val set to: 5
augment_zoom_val set to: 0.05
augment_shift_val set to: 0.05
start_batch set to: 0
chunks set to: None
lockfirst set to: None
load_valid set to: None
drop_landmark set to: None
raw_im_h set to: None
raw_im_w set to: None
n_instances set to: 1
start_sample set to: 0
write_npy set to: None
use_npy set to: False
com_predict_weights set to: None
com_debug set to: None
com_exp set to: None
Setting vid_dir_flag to True.
Setting extension to .avi.
Setting chunks to {'Camera1': array([0]), 'Camera2': array([0]), 'Camera3': array([0])}.
Setting n_channels_in to 3.
Setting raw_im_h to 2560.
Setting raw_im_w to 2560.
Experiment 0 using videos in .\videos
Experiment 0 using camnames: ['Camera1', 'Camera2', 'Camera3']
{'0_Camera1': array([0]), '0_Camera2': array([0]), '0_Camera3': array([0])}
./20221015_173137_Label3D_dannce.mat
Using nn downsampling
TRAIN EXPTS: [0]
Initializing Network...
2022-10-15 18:13:14.584720: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library nvcuda.dll
2022-10-15 18:13:14.646942: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1733] Found device 0 with properties:
pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 3080 computeCapability: 8.6
coreClock: 1.8GHz coreCount: 68 deviceMemorySize: 10.00GiB deviceMemoryBandwidth: 707.88GiB/s
2022-10-15 18:13:14.654599: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library cudart64_110.dll
2022-10-15 18:13:14.654802: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library cublas64_11.dll
2022-10-15 18:13:14.655261: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library cublasLt64_11.dll
2022-10-15 18:13:14.655673: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library cufft64_10.dll
2022-10-15 18:13:14.718061: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library curand64_10.dll
2022-10-15 18:13:14.718307: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library cusolver64_11.dll
2022-10-15 18:13:14.719724: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library cusparse64_11.dll
2022-10-15 18:13:14.720154: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library cudnn64_8.dll
2022-10-15 18:13:14.720614: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1871] Adding visible gpu devices: 0
2022-10-15 18:13:14.732399: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX AVX2
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2022-10-15 18:13:14.736139: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1733] Found device 0 with properties:
pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 3080 computeCapability: 8.6
coreClock: 1.8GHz coreCount: 68 deviceMemorySize: 10.00GiB deviceMemoryBandwidth: 707.88GiB/s
2022-10-15 18:13:14.736483: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1871] Adding visible gpu devices: 0
2022-10-15 18:13:17.018957: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1258] Device interconnect StreamExecutor with strength 1 edge matrix:
2022-10-15 18:13:17.022789: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1264] 0
2022-10-15 18:13:17.050138: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1277] 0: N
2022-10-15 18:13:17.083234: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1418] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 7433 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 3080, pci bus id: 0000:41:00.0, compute capability: 8.6)
E:\anaconda\envs\dannce113\lib\site-packages\tensorflow\python\keras\optimizer_v2\optimizer_v2.py:375: UserWarning: The
lrargument is deprecated, uselearning_rateinstead."The
lrargument is deprecated, uselearning_rateinstead.")COMPLETE
2022-10-15 18:13:18.002074: I tensorflow/core/profiler/lib/profiler_session.cc:126] Profiler session initializing.
2022-10-15 18:13:18.002236: I tensorflow/core/profiler/lib/profiler_session.cc:141] Profiler session started.
2022-10-15 18:13:18.002834: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1611] Profiler found 1 GPUs
2022-10-15 18:13:18.004376: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'cupti64_112.dll'; dlerror: cupti64_112.dll not found
2022-10-15 18:13:18.005366: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'cupti.dll'; dlerror: cupti.dll not found
2022-10-15 18:13:18.005806: E tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1661] function cupti_interface_->Subscribe( &subscriber_, (CUpti_CallbackFunc)ApiCallback, this)failed with error CUPTI could not be loaded or symbol could not be found.
2022-10-15 18:13:18.005906: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session tear down.
2022-10-15 18:13:18.005966: E tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1752] function cupti_interface_->Finalize()failed with error CUPTI could not be loaded or symbol could not be found.
Loading data
Loading new video: .\videos\Camera1\0.avi for 0_Camera1
Loading new video: .\videos\Camera2\0.avi for 0_Camera2
Loading new video: .\videos\Camera3\0.avi for 0_Camera3
f:\testdannce120\dannce\dannce\engine\generator_aux.py:261: RuntimeWarning: invalid value encountered in true_divide
y /= np.max(np.max(y, axis=1), axis=1)[:, np.newaxis, np.newaxis, :]
Loading new video: .\videos\Camera1\0.avi for 0_Camera1
Loading new video: .\videos\Camera2\0.avi for 0_Camera2
Loading new video: .\videos\Camera3\0.avi for 0_Camera3
2022-10-15 18:13:34.113113: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:176] None of the MLIR Optimization Passes are enabled (registered 2)
Epoch 1/10
2022-10-15 18:13:36.067056: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library cudnn64_8.dll
2022-10-15 18:13:39.639960: I tensorflow/stream_executor/cuda/cuda_dnn.cc:359] Loaded cuDNN version 8302
2022-10-15 18:13:44.926827: E tensorflow/core/platform/windows/subprocess.cc:287] Call to CreateProcess failed. Error code: 2
2022-10-15 18:13:44.927055: W tensorflow/stream_executor/gpu/asm_compiler.cc:56] Couldn't invoke ptxas.exe --version
2022-10-15 18:13:44.931278: E tensorflow/core/platform/windows/subprocess.cc:287] Call to CreateProcess failed. Error code: 2
2022-10-15 18:13:44.941195: W tensorflow/stream_executor/gpu/redzone_allocator.cc:314] Internal: Failed to launch ptxas
Relying on driver to perform ptx compilation.
Modify $PATH to customize ptxas location.
This message will be only logged once.
2022-10-15 18:13:45.323821: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library cublas64_11.dll
2022-10-15 18:13:45.324555: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library cublasLt64_11.dll
1/1 [==============================] - 16s 16s/step - loss: 0.0000e+00 - val_loss: 0.0000e+00
Epoch 2/10
2022-10-15 18:13:50.695089: I tensorflow/core/profiler/lib/profiler_session.cc:126] Profiler session initializing.
2022-10-15 18:13:50.696455: I tensorflow/core/profiler/lib/profiler_session.cc:141] Profiler session started.
2022-10-15 18:13:50.697011: E tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1661] function cupti_interface_->Subscribe( &subscriber_, (CUpti_CallbackFunc)ApiCallback, this)failed with error CUPTI could not be loaded or symbol could not be found.
1/1 [==============================] - ETA: 0s - loss: 0.0000e+002022-10-15 18:13:50.843437: I tensorflow/core/profiler/lib/profiler_session.cc:66] Profiler session collecting data.
2022-10-15 18:13:50.843756: E tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1752] function cupti_interface_->Finalize()failed with error CUPTI could not be loaded or symbol could not be found.
2022-10-15 18:13:50.893555: I tensorflow/core/profiler/internal/gpu/cupti_collector.cc:673] GpuTracer has collected 0 callback api events and 0 activity events.
2022-10-15 18:13:50.922415: I tensorflow/core/profiler/lib/profiler_session.cc:159] Profiler session tear down.
1/1 [==============================] - 1s 644ms/step - loss: 0.0000e+00 - val_loss: 0.0000e+00
Epoch 3/10
1/1 [==============================] - 0s 364ms/step - loss: 0.0000e+00 - val_loss: 0.0000e+00
Epoch 4/10
1/1 [==============================] - 0s 365ms/step - loss: 0.0000e+00 - val_loss: 0.0000e+00
Epoch 5/10
1/1 [==============================] - 0s 350ms/step - loss: 0.0000e+00 - val_loss: 0.0000e+00
Epoch 6/10
1/1 [==============================] - 0s 365ms/step - loss: 0.0000e+00 - val_loss: 0.0000e+00
Epoch 7/10
1/1 [==============================] - 0s 380ms/step - loss: 0.0000e+00 - val_loss: 0.0000e+00
Epoch 8/10
1/1 [==============================] - 0s 367ms/step - loss: 0.0000e+00 - val_loss: 0.0000e+00
Epoch 9/10
1/1 [==============================] - 0s 350ms/step - loss: 0.0000e+00 - val_loss: 0.0000e+00
Epoch 10/10
1/1 [==============================] - 0s 367ms/step - loss: 0.0000e+00 - val_loss: 0.0000e+00
Renaming weights file with best epoch description
Saving full model at end of training
`
The text was updated successfully, but these errors were encountered: