You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I'm reprocessing a few dozen 2photon acquisitions and performing fine-tuning on the Ai93 model for each acquisition. We've tested it out and determined that we're going to need to fine-tune a model for each field of view.
We're storing each acquisition in a 3D HDF5 file, then we loop over the parent directories of each acquisition for creating the generator, test generator, and fine tuning objects. We then build the fine-tuning object and run.
This goes well for three to four models before we run into RuntimeErrors caused by Tkinter of all things. I've attached the script we're running. This hasn't always been a problem. We've been able to run a bunch of models overnight before, but that might have been with tiff stacks.
Are there any other users experiencing this issue? I've also included the output for conda list to show my environment.
import os
import sys
import glob
import h5py
import pathlib
from datetime import datetime
sys.path.append('C:\\Users\\greg\\Documents\\Python\\deepinterpolation-0.1.5')
from deepinterpolation.cli.fine_tuning import FineTuning
def main():
loss_fun = "mean_squared_error"
sessions_to_run = [
"D:\\seagate_data_for_ben\\TSeries-08252022-1332-002\\Registered\\Channel1",
"D:\\seagate_data_for_ben\\TSeries-08252022-1332-003\\Registered\\Channel1"
# More sessions, etc.
]
time_deltas = []
for i in range(0, len(sessions_to_run)):
parent_dir = sessions_to_run[i]
finetuning_params = {}
generator_param = {}
generator_test_param = {}
# Grab some info from datafile
data_file_maybe = glob.glob(parent_dir + "\\registered.h5")
assert len(data_file_maybe) == 1, "Either HDF5 file for registered data was not found, or more than one was found. Check parent directory."
data_file = data_file_maybe[0]
f = h5py.File(data_file, "a")
rename = False
try:
total_frames = f['mov'].shape[0]
rename = True
except:
total_frames = f['data'].shape[0]
if total_frames < 14000:
num_training_samples = total_frames
else:
num_training_samples = 14000
if rename:
f["data"] = f["mov"]
del f["mov"]
f.close()
# Parameters used for the validation step each epoch
generator_test_param["name"] = "OphysGenerator"
generator_test_param["pre_frame"] = 30
generator_test_param["post_frame"] = 30
generator_test_param["data_path"] = data_file
generator_test_param["batch_size"] = 10
generator_test_param["start_frame"] = 1
generator_test_param["end_frame"] = -1
generator_test_param["total_samples"] = 500
generator_test_param["randomize"] = 1
generator_test_param["pre_post_omission"] = 0
#Parameters used for the main data generator
generator_param["name"] = "OphysGenerator"
generator_param["pre_frame"] = 30
generator_param["post_frame"] = 30
generator_param["data_path"] = data_file
generator_param["batch_size"] = 10
generator_param["start_frame"] = 1
generator_param["end_frame"] = -1
generator_param["total_samples"] = num_training_samples
generator_param["randomize"] = 1
generator_param["pre_post_omission"] = 0
# Parameters used for the training process
finetuning_params["name"] = "transfer_trainer"
# Change this path to any model you wish to improve
filename = "2019_09_11_23_32_unet_single_1024_mean_absolute_error_Ai93-0450.h5"
local_path = \
os.path.join(
"C:\\Users\\greg\\Documents\\Python\\deepinterpolation-0.1.5\\sample_data",
filename
)
finetuning_params["model_source"] = {
"local_path": local_path
}
steps_per_epoch = 200
finetuning_params["steps_per_epoch"] = steps_per_epoch
finetuning_params[
"period_save"
] = 5
finetuning_params["learning_rate"] = 0.0001
finetuning_params["loss"] = loss_fun
finetuning_params["output_dir"] = parent_dir
finetuning_params["apply_learning_decay"] = False
finetuning_params["caching_validation"] = False
finetuning_params["use_multiprocessing"] = False
finetuning_params["nb_workers"] = 8
args = {
"finetuning_params": finetuning_params,
"generator_params": generator_param,
"test_generator_params": generator_test_param,
"output_full_args": True
}
t_start = datetime.now()
finetuning_obj = FineTuning(input_data=args, args=[])
finetuning_obj.run()
t_stop = datetime.now()
time_deltas.append(t_stop - t_start)
print(str(t_stop - t_start))
if __name__=='__main__':
main()
INFO:FineTuning:fine tuning job finished - finalizing output model
Saved model to disk
0:54:37.463883
INFO:FineTuning:wrote D:\seagate_data_for_ben\TSeries-11262022-1053-001\Registered\Channel1\2023_04_03_22_32_training_full_args.json
INFO:FineTuning:wrote D:\seagate_data_for_ben\TSeries-11262022-1053-001\Registered\Channel1\2023_04_03_22_32_finetuning.json
INFO:FineTuning:wrote D:\seagate_data_for_ben\TSeries-11262022-1053-001\Registered\Channel1\2023_04_03_22_32_generator.json
INFO:FineTuning:wrote D:\seagate_data_for_ben\TSeries-11262022-1053-001\Registered\Channel1\2023_04_03_22_32_test_generator.json
WARNING:tensorflow:`period` argument is deprecated. Please use `save_freq` to specify the frequency in number of batches seen.
WARNING:tensorflow:`period` argument is deprecated. Please use `save_freq` to specify the frequency in number of batches seen.
Exception ignored in: <function Image.__del__ at 0x000001ECA89DFE50>
Traceback (most recent call last):
File "C:\Users\greg\anaconda3\envs\deepinterpolation\lib\tkinter\__init__.py", line 4017, in __del__
self.tk.call('image', 'delete', self.name)
RuntimeError: main thread is not in main loop
Exception ignored in: <function Variable.__del__ at 0x000001ECA89CA550>
Traceback (most recent call last):
File "C:\Users\greg\anaconda3\envs\deepinterpolation\lib\tkinter\__init__.py", line 363, in __del__
if self._tk.getboolean(self._tk.call("info", "exists", self._name)):
RuntimeError: main thread is not in main loop
Exception ignored in: <function Variable.__del__ at 0x000001ECA89CA550>
Traceback (most recent call last):
File "C:\Users\greg\anaconda3\envs\deepinterpolation\lib\tkinter\__init__.py", line 363, in __del__
if self._tk.getboolean(self._tk.call("info", "exists", self._name)):
RuntimeError: main thread is not in main loop
Exception ignored in: <function Variable.__del__ at 0x000001ECA89CA550>
Traceback (most recent call last):
File "C:\Users\greg\anaconda3\envs\deepinterpolation\lib\tkinter\__init__.py", line 363, in __del__
if self._tk.getboolean(self._tk.call("info", "exists", self._name)):
RuntimeError: main thread is not in main loop
Exception ignored in: <function Variable.__del__ at 0x000001ECA89CA550>
Traceback (most recent call last):
File "C:\Users\greg\anaconda3\envs\deepinterpolation\lib\tkinter\__init__.py", line 363, in __del__
if self._tk.getboolean(self._tk.call("info", "exists", self._name)):
RuntimeError: main thread is not in main loop
Exception ignored in: <function Image.__del__ at 0x000001ECA89DFE50>
Traceback (most recent call last):
File "C:\Users\greg\anaconda3\envs\deepinterpolation\lib\tkinter\__init__.py", line 4017, in __del__
self.tk.call('image', 'delete', self.name)
RuntimeError: main thread is not in main loop
Exception ignored in: <function Variable.__del__ at 0x000001ECA89CA550>
Traceback (most recent call last):
File "C:\Users\greg\anaconda3\envs\deepinterpolation\lib\tkinter\__init__.py", line 363, in __del__
if self._tk.getboolean(self._tk.call("info", "exists", self._name)):
RuntimeError: main thread is not in main loop
Exception ignored in: <function Variable.__del__ at 0x000001ECA89CA550>
Traceback (most recent call last):
File "C:\Users\greg\anaconda3\envs\deepinterpolation\lib\tkinter\__init__.py", line 363, in __del__
if self._tk.getboolean(self._tk.call("info", "exists", self._name)):
RuntimeError: main thread is not in main loop
Exception ignored in: <function Variable.__del__ at 0x000001ECA89CA550>
Traceback (most recent call last):
File "C:\Users\greg\anaconda3\envs\deepinterpolation\lib\tkinter\__init__.py", line 363, in __del__
if self._tk.getboolean(self._tk.call("info", "exists", self._name)):
RuntimeError: main thread is not in main loop
Exception ignored in: <function Variable.__del__ at 0x000001ECA89CA550>
Traceback (most recent call last):
File "C:\Users\greg\anaconda3\envs\deepinterpolation\lib\tkinter\__init__.py", line 363, in __del__
if self._tk.getboolean(self._tk.call("info", "exists", self._name)):
RuntimeError: main thread is not in main loop
Tcl_AsyncDelete: async handler deleted by the wrong thread
Hello,
I'm reprocessing a few dozen 2photon acquisitions and performing fine-tuning on the Ai93 model for each acquisition. We've tested it out and determined that we're going to need to fine-tune a model for each field of view.
We're storing each acquisition in a 3D HDF5 file, then we loop over the parent directories of each acquisition for creating the generator, test generator, and fine tuning objects. We then build the fine-tuning object and run.
This goes well for three to four models before we run into RuntimeErrors caused by Tkinter of all things. I've attached the script we're running. This hasn't always been a problem. We've been able to run a bunch of models overnight before, but that might have been with tiff stacks.
These two resources helped me out. It seems that when matplotlib creates the plots for training and validation losses across fine-tuning, it's invoking Tkinter as the back end. Simplest solution for me is to disable creating these plots for now.
Project-MONAI/tutorials#94
https://stackoverflow.com/questions/27147300/matplotlib-tcl-asyncdelete-async-handler-deleted-by-the-wrong-thread
Are there any other users experiencing this issue? I've also included the output for
conda listto show my environment.The text was updated successfully, but these errors were encountered: