train.py crashes

[RyannDaGreat]

When I run train.py, it trains fine for about 1000 or so iterations, and then it crashes due to some threading error involving tkinter.

I ran this command:
(video-generation) G ➜ PlayableVideoGeneration git:(main) python3 train.py --config configs/02_breakout.yaml

And much further down, many minutes later I got this:

step: 984/300000 loss_component_observations_rec:0.039 loss_component_perceptual_loss:0.808 loss_component_hidden_states_rec:0.222 loss_component_states_rec:0.001 loss_component_entropy:0.000 loss_component_action_directions_kl_divergence:0.000 loss_component_action_mutual_information:-0.000 loss_component_action_state_distribution_kl:0.000 avg_observations_rec_loss:0.039 avg_perceptual_loss:0.417 states_rec_loss:0.007 hidden_states_rec_loss:0.222 entropy_loss:0.907 samples_entropy:0.696 action_distribution_entropy:1.053 states_magnitude:0.656 hidden_states_magnitude:0.419 action_directions_mean_magnitude:0.001 action_directions_variance_magnitude:0.676 reconstructed_action_directions_mean_magnitude:0.001 reconstructed_action_directions_variance_magnitude:0.678 action_directions_reconstruction_error:0.000 action_directions_kl_loss:0.034 centroids_mean_magnitude:0.000 average_centroids_distance:0.000 average_action_variations_norm_l2:0.727 action_variations_mean:-0.026 reconstructed_action_directions_kl_loss:0.033 action_mutual_information_loss:-0.000 action_state_distribution_kl_loss:0.000 ground_truth_observations:6.000 gumbel_temperature:0.970 observations_count:7.000 perceptual_loss_r0:0.492 observations_rec_loss_r0:0.028 perceptual_loss_r0_l0:0.492 perceptual_loss_r0_l1:0.085 perceptual_loss_r0_l2:0.165 perceptual_loss_r0_l3:0.161 perceptual_loss_r0_l4:0.061 perceptual_loss_r1:0.246 observations_rec_loss_r1:0.018 perceptual_loss_r1_l0:0.246 perceptual_loss_r1_l1:0.044 perceptual_loss_r1_l2:0.078 perceptual_loss_r1_l3:0.077 perceptual_loss_r1_l4:0.035 perceptual_loss_r2:0.513 observations_rec_loss_r2:0.071 perceptual_loss_r2_l0:0.513 perceptual_loss_r2_l1:0.123 perceptual_loss_r2_l2:0.168 perceptual_loss_r2_l3:0.128 perceptual_loss_r2_l4:0.047 loss:1.071 lr: 0.0004
step: 985/300000 loss_component_observations_rec:0.038 loss_component_perceptual_loss:0.814 loss_component_hidden_states_rec:0.221 loss_component_states_rec:0.002 loss_component_entropy:0.000 loss_component_action_directions_kl_divergence:0.000 loss_component_action_mutual_information:-0.000 loss_component_action_state_distribution_kl:0.000 avg_observations_rec_loss:0.038 avg_perceptual_loss:0.420 states_rec_loss:0.008 hidden_states_rec_loss:0.221 entropy_loss:0.928 samples_entropy:0.676 action_distribution_entropy:1.021 states_magnitude:0.657 hidden_states_magnitude:0.419 action_directions_mean_magnitude:0.001 action_directions_variance_magnitude:0.675 reconstructed_action_directions_mean_magnitude:0.001 reconstructed_action_directions_variance_magnitude:0.678 action_directions_reconstruction_error:0.000 action_directions_kl_loss:0.034 centroids_mean_magnitude:0.000 average_centroids_distance:0.000 average_action_variations_norm_l2:0.693 action_variations_mean:0.036 reconstructed_action_directions_kl_loss:0.033 action_mutual_information_loss:-0.000 action_state_distribution_kl_loss:0.000 ground_truth_observations:6.000 gumbel_temperature:0.970 observations_count:7.000 perceptual_loss_r0:0.484 observations_rec_loss_r0:0.028 perceptual_loss_r0_l0:0.484 perceptual_loss_r0_l1:0.084 perceptual_loss_r0_l2:0.161 perceptual_loss_r0_l3:0.160 perceptual_loss_r0_l4:0.060 perceptual_loss_r1:0.257 observations_rec_loss_r1:0.017 perceptual_loss_r1_l0:0.257 perceptual_loss_r1_l1:0.044 perceptual_loss_r1_l2:0.081 perceptual_loss_r1_l3:0.084 perceptual_loss_r1_l4:0.036 perceptual_loss_r2:0.518 observations_rec_loss_r2:0.068 perceptual_loss_r2_l0:0.518 perceptual_loss_r2_l1:0.119 perceptual_loss_r2_l2:0.169 perceptual_loss_r2_l3:0.134 perceptual_loss_r2_l4:0.051 loss:1.075 lr: 0.0004
step: 986/300000 loss_component_observations_rec:0.039 loss_component_perceptual_loss:0.797 loss_component_hidden_states_rec:0.219 loss_component_states_rec:0.002 loss_component_entropy:0.000 loss_component_action_directions_kl_divergence:0.000 loss_component_action_mutual_information:-0.000 loss_component_action_state_distribution_kl:0.000 avg_observations_rec_loss:0.039 avg_perceptual_loss:0.411 states_rec_loss:0.008 hidden_states_rec_loss:0.219 entropy_loss:0.915 samples_entropy:0.727 action_distribution_entropy:1.049 states_magnitude:0.655 hidden_states_magnitude:0.419 action_directions_mean_magnitude:0.001 action_directions_variance_magnitude:0.676 reconstructed_action_directions_mean_magnitude:0.001 reconstructed_action_directions_variance_magnitude:0.678 action_directions_reconstruction_error:0.000 action_directions_kl_loss:0.034 centroids_mean_magnitude:0.000 average_centroids_distance:0.000 average_action_variations_norm_l2:0.649 action_variations_mean:-0.017 reconstructed_action_directions_kl_loss:0.033 action_mutual_information_loss:-0.000 action_state_distribution_kl_loss:0.000 ground_truth_observations:6.000 gumbel_temperature:0.970 observations_count:7.000 perceptual_loss_r0:0.478 observations_rec_loss_r0:0.030 perceptual_loss_r0_l0:0.478 perceptual_loss_r0_l1:0.084 perceptual_loss_r0_l2:0.160 perceptual_loss_r0_l3:0.155 perceptual_loss_r0_l4:0.060 perceptual_loss_r1:0.248 observations_rec_loss_r1:0.018 perceptual_loss_r1_l0:0.248 perceptual_loss_r1_l1:0.043 perceptual_loss_r1_l2:0.079 perceptual_loss_r1_l3:0.081 perceptual_loss_r1_l4:0.034 perceptual_loss_r2:0.507 observations_rec_loss_r2:0.070 perceptual_loss_r2_l0:0.507 perceptual_loss_r2_l1:0.117 perceptual_loss_r2_l2:0.163 perceptual_loss_r2_l3:0.132 perceptual_loss_r2_l4:0.051 loss:1.057 lr: 0.0004
step: 987/300000 loss_component_observations_rec:0.038 loss_component_perceptual_loss:0.791 loss_component_hidden_states_rec:0.217 loss_component_states_rec:0.001 loss_component_entropy:0.000 loss_component_action_directions_kl_divergence:0.000 loss_component_action_mutual_information:-0.000 loss_component_action_state_distribution_kl:0.000 avg_observations_rec_loss:0.038 avg_perceptual_loss:0.408 states_rec_loss:0.007 hidden_states_rec_loss:0.217 entropy_loss:0.886 samples_entropy:0.665 action_distribution_entropy:0.977 states_magnitude:0.654 hidden_states_magnitude:0.418 action_directions_mean_magnitude:0.001 action_directions_variance_magnitude:0.673 reconstructed_action_directions_mean_magnitude:0.001 reconstructed_action_directions_variance_magnitude:0.676 action_directions_reconstruction_error:0.000 action_directions_kl_loss:0.035 centroids_mean_magnitude:0.000 average_centroids_distance:0.000 average_action_variations_norm_l2:0.616 action_variations_mean:-0.237 reconstructed_action_directions_kl_loss:0.034 action_mutual_information_loss:-0.000 action_state_distribution_kl_loss:0.000 ground_truth_observations:6.000 gumbel_temperature:0.970 observations_count:7.000 perceptual_loss_r0:0.466 observations_rec_loss_r0:0.028 perceptual_loss_r0_l0:0.466 perceptual_loss_r0_l1:0.083 perceptual_loss_r0_l2:0.158 perceptual_loss_r0_l3:0.150 perceptual_loss_r0_l4:0.056 perceptual_loss_r1:0.245 observations_rec_loss_r1:0.017 perceptual_loss_r1_l0:0.245 perceptual_loss_r1_l1:0.043 perceptual_loss_r1_l2:0.077 perceptual_loss_r1_l3:0.078 perceptual_loss_r1_l4:0.036 perceptual_loss_r2:0.513 observations_rec_loss_r2:0.069 perceptual_loss_r2_l0:0.513 perceptual_loss_r2_l1:0.118 perceptual_loss_r2_l2:0.165 perceptual_loss_r2_l3:0.136 perceptual_loss_r2_l4:0.048 loss:1.048 lr: 0.0004
step: 988/300000 loss_component_observations_rec:0.037 loss_component_perceptual_loss:0.813 loss_component_hidden_states_rec:0.218 loss_component_states_rec:0.002 loss_component_entropy:0.000 loss_component_action_directions_kl_divergence:0.000 loss_component_action_mutual_information:-0.000 loss_component_action_state_distribution_kl:0.000 avg_observations_rec_loss:0.037 avg_perceptual_loss:0.419 states_rec_loss:0.008 hidden_states_rec_loss:0.218 entropy_loss:0.922 samples_entropy:0.615 action_distribution_entropy:0.983 states_magnitude:0.656 hidden_states_magnitude:0.417 action_directions_mean_magnitude:0.001 action_directions_variance_magnitude:0.672 reconstructed_action_directions_mean_magnitude:0.001 reconstructed_action_directions_variance_magnitude:0.673 action_directions_reconstruction_error:0.000 action_directions_kl_loss:0.035 centroids_mean_magnitude:0.000 average_centroids_distance:0.000 average_action_variations_norm_l2:0.750 action_variations_mean:0.063 reconstructed_action_directions_kl_loss:0.035 action_mutual_information_loss:-0.000 action_state_distribution_kl_loss:0.000 ground_truth_observations:6.000 gumbel_temperature:0.970 observations_count:7.000 perceptual_loss_r0:0.484 observations_rec_loss_r0:0.026 perceptual_loss_r0_l0:0.484 perceptual_loss_r0_l1:0.084 perceptual_loss_r0_l2:0.164 perceptual_loss_r0_l3:0.157 perceptual_loss_r0_l4:0.059 perceptual_loss_r1:0.244 observations_rec_loss_r1:0.018 perceptual_loss_r1_l0:0.244 perceptual_loss_r1_l1:0.042 perceptual_loss_r1_l2:0.077 perceptual_loss_r1_l3:0.079 perceptual_loss_r1_l4:0.035 perceptual_loss_r2:0.530 observations_rec_loss_r2:0.069 perceptual_loss_r2_l0:0.530 perceptual_loss_r2_l1:0.123 perceptual_loss_r2_l2:0.172 perceptual_loss_r2_l3:0.138 perceptual_loss_r2_l4:0.051 loss:1.070 lr: 0.0004
step: 989/300000 loss_component_observations_rec:0.037 loss_component_perceptual_loss:0.777 loss_component_hidden_states_rec:0.216 loss_component_states_rec:0.001 loss_component_entropy:0.000 loss_component_action_directions_kl_divergence:0.000 loss_component_action_mutual_information:-0.000 loss_component_action_state_distribution_kl:0.000 avg_observations_rec_loss:0.037 avg_perceptual_loss:0.401 states_rec_loss:0.007 hidden_states_rec_loss:0.216 entropy_loss:0.954 samples_entropy:0.645 action_distribution_entropy:1.025 states_magnitude:0.656 hidden_states_magnitude:0.418 action_directions_mean_magnitude:0.001 action_directions_variance_magnitude:0.674 reconstructed_action_directions_mean_magnitude:0.001 reconstructed_action_directions_variance_magnitude:0.674 action_directions_reconstruction_error:0.000 action_directions_kl_loss:0.034 centroids_mean_magnitude:0.000 average_centroids_distance:0.000 average_action_variations_norm_l2:0.599 action_variations_mean:0.092 reconstructed_action_directions_kl_loss:0.034 action_mutual_information_loss:-0.000 action_state_distribution_kl_loss:0.000 ground_truth_observations:6.000 gumbel_temperature:0.970 observations_count:7.000 perceptual_loss_r0:0.476 observations_rec_loss_r0:0.027 perceptual_loss_r0_l0:0.476 perceptual_loss_r0_l1:0.084 perceptual_loss_r0_l2:0.161 perceptual_loss_r0_l3:0.154 perceptual_loss_r0_l4:0.057 perceptual_loss_r1:0.215 observations_rec_loss_r1:0.017 perceptual_loss_r1_l0:0.215 perceptual_loss_r1_l1:0.040 perceptual_loss_r1_l2:0.069 perceptual_loss_r1_l3:0.066 perceptual_loss_r1_l4:0.030 perceptual_loss_r2:0.512 observations_rec_loss_r2:0.067 perceptual_loss_r2_l0:0.512 perceptual_loss_r2_l1:0.119 perceptual_loss_r2_l2:0.166 perceptual_loss_r2_l3:0.129 perceptual_loss_r2_l4:0.052 loss:1.031 lr: 0.0004
step: 990/300000 loss_component_observations_rec:0.037 loss_component_perceptual_loss:0.777 loss_component_hidden_states_rec:0.217 loss_component_states_rec:0.001 loss_component_entropy:0.000 loss_component_action_directions_kl_divergence:0.000 loss_component_action_mutual_information:-0.000 loss_component_action_state_distribution_kl:0.000 avg_observations_rec_loss:0.037 avg_perceptual_loss:0.401 states_rec_loss:0.006 hidden_states_rec_loss:0.217 entropy_loss:0.940 samples_entropy:0.823 action_distribution_entropy:1.053 states_magnitude:0.657 hidden_states_magnitude:0.419 action_directions_mean_magnitude:0.001 action_directions_variance_magnitude:0.677 reconstructed_action_directions_mean_magnitude:0.001 reconstructed_action_directions_variance_magnitude:0.678 action_directions_reconstruction_error:0.000 action_directions_kl_loss:0.034 centroids_mean_magnitude:0.000 average_centroids_distance:0.000 average_action_variations_norm_l2:0.531 action_variations_mean:0.005 reconstructed_action_directions_kl_loss:0.033 action_mutual_information_loss:-0.000 action_state_distribution_kl_loss:0.000 ground_truth_observations:6.000 gumbel_temperature:0.970 observations_count:7.000 perceptual_loss_r0:0.462 observations_rec_loss_r0:0.027 perceptual_loss_r0_l0:0.462 perceptual_loss_r0_l1:0.080 perceptual_loss_r0_l2:0.153 perceptual_loss_r0_l3:0.149 perceptual_loss_r0_l4:0.061 perceptual_loss_r1:0.231 observations_rec_loss_r1:0.018 perceptual_loss_r1_l0:0.231 perceptual_loss_r1_l1:0.040 perceptual_loss_r1_l2:0.073 perceptual_loss_r1_l3:0.073 perceptual_loss_r1_l4:0.034 perceptual_loss_r2:0.509 observations_rec_loss_r2:0.065 perceptual_loss_r2_l0:0.509 perceptual_loss_r2_l1:0.116 perceptual_loss_r2_l2:0.165 perceptual_loss_r2_l3:0.132 perceptual_loss_r2_l4:0.051 loss:1.032 lr: 0.0004
step: 991/300000 loss_component_observations_rec:0.041 loss_component_perceptual_loss:0.843 loss_component_hidden_states_rec:0.222 loss_component_states_rec:0.002 loss_component_entropy:0.000 loss_component_action_directions_kl_divergence:0.000 loss_component_action_mutual_information:-0.000 loss_component_action_state_distribution_kl:0.000 avg_observations_rec_loss:0.041 avg_perceptual_loss:0.435 states_rec_loss:0.008 hidden_states_rec_loss:0.222 entropy_loss:0.906 samples_entropy:0.729 action_distribution_entropy:1.034 states_magnitude:0.654 hidden_states_magnitude:0.419 action_directions_mean_magnitude:0.001 action_directions_variance_magnitude:0.680 reconstructed_action_directions_mean_magnitude:0.001 reconstructed_action_directions_variance_magnitude:0.680 action_directions_reconstruction_error:0.000 action_directions_kl_loss:0.033 centroids_mean_magnitude:0.000 average_centroids_distance:0.000 average_action_variations_norm_l2:0.662 action_variations_mean:-0.087 reconstructed_action_directions_kl_loss:0.033 action_mutual_information_loss:-0.000 action_state_distribution_kl_loss:0.000 ground_truth_observations:6.000 gumbel_temperature:0.970 observations_count:7.000 perceptual_loss_r0:0.506 observations_rec_loss_r0:0.031 perceptual_loss_r0_l0:0.506 perceptual_loss_r0_l1:0.086 perceptual_loss_r0_l2:0.166 perceptual_loss_r0_l3:0.167 perceptual_loss_r0_l4:0.067 perceptual_loss_r1:0.274 observations_rec_loss_r1:0.020 perceptual_loss_r1_l0:0.274 perceptual_loss_r1_l1:0.047 perceptual_loss_r1_l2:0.086 perceptual_loss_r1_l3:0.087 perceptual_loss_r1_l4:0.042 perceptual_loss_r2:0.524 observations_rec_loss_r2:0.071 perceptual_loss_r2_l0:0.524 perceptual_loss_r2_l1:0.123 perceptual_loss_r2_l2:0.172 perceptual_loss_r2_l3:0.135 perceptual_loss_r2_l4:0.048 loss:1.108 lr: 0.0004
step: 992/300000 loss_component_observations_rec:0.039 loss_component_perceptual_loss:0.839 loss_component_hidden_states_rec:0.218 loss_component_states_rec:0.001 loss_component_entropy:0.000 loss_component_action_directions_kl_divergence:0.000 loss_component_action_mutual_information:-0.000 loss_component_action_state_distribution_kl:0.000 avg_observations_rec_loss:0.039 avg_perceptual_loss:0.433 states_rec_loss:0.007 hidden_states_rec_loss:0.218 entropy_loss:0.900 samples_entropy:0.640 action_distribution_entropy:1.012 states_magnitude:0.654 hidden_states_magnitude:0.417 action_directions_mean_magnitude:0.001 action_directions_variance_magnitude:0.679 reconstructed_action_directions_mean_magnitude:0.001 reconstructed_action_directions_variance_magnitude:0.681 action_directions_reconstruction_error:0.000 action_directions_kl_loss:0.033 centroids_mean_magnitude:0.000 average_centroids_distance:0.000 average_action_variations_norm_l2:0.773 action_variations_mean:-0.026 reconstructed_action_directions_kl_loss:0.033 action_mutual_information_loss:-0.000 action_state_distribution_kl_loss:0.000 ground_truth_observations:6.000 gumbel_temperature:0.970 observations_count:7.000 perceptual_loss_r0:0.502 observations_rec_loss_r0:0.028 perceptual_loss_r0_l0:0.502 perceptual_loss_r0_l1:0.086 perceptual_loss_r0_l2:0.166 perceptual_loss_r0_l3:0.165 perceptual_loss_r0_l4:0.065 perceptual_loss_r1:0.261 observations_rec_loss_r1:0.018 perceptual_loss_r1_l0:0.261 perceptual_loss_r1_l1:0.044 perceptual_loss_r1_l2:0.082 perceptual_loss_r1_l3:0.084 perceptual_loss_r1_l4:0.039 perceptual_loss_r2:0.535 observations_rec_loss_r2:0.071 perceptual_loss_r2_l0:0.535 perceptual_loss_r2_l1:0.123 perceptual_loss_r2_l2:0.173 perceptual_loss_r2_l3:0.139 perceptual_loss_r2_l4:0.053 loss:1.097 lr: 0.0004
step: 993/300000 loss_component_observations_rec:0.038 loss_component_perceptual_loss:0.823 loss_component_hidden_states_rec:0.221 loss_component_states_rec:0.001 loss_component_entropy:0.000 loss_component_action_directions_kl_divergence:0.000 loss_component_action_mutual_information:-0.000 loss_component_action_state_distribution_kl:0.000 avg_observations_rec_loss:0.038 avg_perceptual_loss:0.424 states_rec_loss:0.007 hidden_states_rec_loss:0.221 entropy_loss:0.902 samples_entropy:0.650 action_distribution_entropy:1.029 states_magnitude:0.653 hidden_states_magnitude:0.417 action_directions_mean_magnitude:0.001 action_directions_variance_magnitude:0.683 reconstructed_action_directions_mean_magnitude:0.001 reconstructed_action_directions_variance_magnitude:0.684 action_directions_reconstruction_error:0.000 action_directions_kl_loss:0.032 centroids_mean_magnitude:0.000 average_centroids_distance:0.000 average_action_variations_norm_l2:0.737 action_variations_mean:-0.088 reconstructed_action_directions_kl_loss:0.032 action_mutual_information_loss:-0.000 action_state_distribution_kl_loss:0.000 ground_truth_observations:6.000 gumbel_temperature:0.970 observations_count:7.000 perceptual_loss_r0:0.497 observations_rec_loss_r0:0.026 perceptual_loss_r0_l0:0.497 perceptual_loss_r0_l1:0.084 perceptual_loss_r0_l2:0.165 perceptual_loss_r0_l3:0.165 perceptual_loss_r0_l4:0.064 perceptual_loss_r1:0.253 observations_rec_loss_r1:0.017 perceptual_loss_r1_l0:0.253 perceptual_loss_r1_l1:0.043 perceptual_loss_r1_l2:0.078 perceptual_loss_r1_l3:0.083 perceptual_loss_r1_l4:0.039 perceptual_loss_r2:0.522 observations_rec_loss_r2:0.070 perceptual_loss_r2_l0:0.522 perceptual_loss_r2_l1:0.119 perceptual_loss_r2_l2:0.167 perceptual_loss_r2_l3:0.136 perceptual_loss_r2_l4:0.054 loss:1.082 lr: 0.0004
step: 994/300000 loss_component_observations_rec:0.039 loss_component_perceptual_loss:0.840 loss_component_hidden_states_rec:0.223 loss_component_states_rec:0.001 loss_component_entropy:0.000 loss_component_action_directions_kl_divergence:0.000 loss_component_action_mutual_information:-0.000 loss_component_action_state_distribution_kl:0.000 avg_observations_rec_loss:0.039 avg_perceptual_loss:0.433 states_rec_loss:0.007 hidden_states_rec_loss:0.223 entropy_loss:0.921 samples_entropy:0.652 action_distribution_entropy:0.991 states_magnitude:0.654 hidden_states_magnitude:0.418 action_directions_mean_magnitude:0.001 action_directions_variance_magnitude:0.682 reconstructed_action_directions_mean_magnitude:0.001 reconstructed_action_directions_variance_magnitude:0.684 action_directions_reconstruction_error:0.000 action_directions_kl_loss:0.032 centroids_mean_magnitude:0.000 average_centroids_distance:0.000 average_action_variations_norm_l2:0.670 action_variations_mean:0.009 reconstructed_action_directions_kl_loss:0.032 action_mutual_information_loss:-0.000 action_state_distribution_kl_loss:0.000 ground_truth_observations:6.000 gumbel_temperature:0.970 observations_count:7.000 perceptual_loss_r0:0.502 observations_rec_loss_r0:0.029 perceptual_loss_r0_l0:0.502 perceptual_loss_r0_l1:0.085 perceptual_loss_r0_l2:0.166 perceptual_loss_r0_l3:0.167 perceptual_loss_r0_l4:0.063 perceptual_loss_r1:0.259 observations_rec_loss_r1:0.018 perceptual_loss_r1_l0:0.259 perceptual_loss_r1_l1:0.046 perceptual_loss_r1_l2:0.083 perceptual_loss_r1_l3:0.083 perceptual_loss_r1_l4:0.034 perceptual_loss_r2:0.538 observations_rec_loss_r2:0.069 perceptual_loss_r2_l0:0.538 perceptual_loss_r2_l1:0.122 perceptual_loss_r2_l2:0.173 perceptual_loss_r2_l3:0.142 perceptual_loss_r2_l4:0.055 loss:1.103 lr: 0.0004
step: 995/300000 loss_component_observations_rec:0.037 loss_component_perceptual_loss:0.797 loss_component_hidden_states_rec:0.219 loss_component_states_rec:0.001 loss_component_entropy:0.000 loss_component_action_directions_kl_divergence:0.000 loss_component_action_mutual_information:-0.000 loss_component_action_state_distribution_kl:0.000 avg_observations_rec_loss:0.037 avg_perceptual_loss:0.411 states_rec_loss:0.006 hidden_states_rec_loss:0.219 entropy_loss:0.955 samples_entropy:0.738 action_distribution_entropy:1.055 states_magnitude:0.655 hidden_states_magnitude:0.419 action_directions_mean_magnitude:0.001 action_directions_variance_magnitude:0.679 reconstructed_action_directions_mean_magnitude:0.001 reconstructed_action_directions_variance_magnitude:0.680 action_directions_reconstruction_error:0.000 action_directions_kl_loss:0.033 centroids_mean_magnitude:0.000 average_centroids_distance:0.000 average_action_variations_norm_l2:0.549 action_variations_mean:0.042 reconstructed_action_directions_kl_loss:0.033 action_mutual_information_loss:-0.000 action_state_distribution_kl_loss:0.000 ground_truth_observations:6.000 gumbel_temperature:0.970 observations_count:7.000 perceptual_loss_r0:0.454 observations_rec_loss_r0:0.027 perceptual_loss_r0_l0:0.454 perceptual_loss_r0_l1:0.079 perceptual_loss_r0_l2:0.152 perceptual_loss_r0_l3:0.147 perceptual_loss_r0_l4:0.057 perceptual_loss_r1:0.233 observations_rec_loss_r1:0.017 perceptual_loss_r1_l0:0.233 perceptual_loss_r1_l1:0.040 perceptual_loss_r1_l2:0.074 perceptual_loss_r1_l3:0.076 perceptual_loss_r1_l4:0.032 perceptual_loss_r2:0.547 observations_rec_loss_r2:0.069 perceptual_loss_r2_l0:0.547 perceptual_loss_r2_l1:0.124 perceptual_loss_r2_l2:0.176 perceptual_loss_r2_l3:0.146 perceptual_loss_r2_l4:0.054 loss:1.055 lr: 0.0004
step: 996/300000 loss_component_observations_rec:0.037 loss_component_perceptual_loss:0.848 loss_component_hidden_states_rec:0.219 loss_component_states_rec:0.001 loss_component_entropy:0.000 loss_component_action_directions_kl_divergence:0.000 loss_component_action_mutual_information:-0.000 loss_component_action_state_distribution_kl:0.000 avg_observations_rec_loss:0.037 avg_perceptual_loss:0.437 states_rec_loss:0.007 hidden_states_rec_loss:0.219 entropy_loss:0.957 samples_entropy:0.737 action_distribution_entropy:1.052 states_magnitude:0.655 hidden_states_magnitude:0.419 action_directions_mean_magnitude:0.001 action_directions_variance_magnitude:0.685 reconstructed_action_directions_mean_magnitude:0.001 reconstructed_action_directions_variance_magnitude:0.686 action_directions_reconstruction_error:0.000 action_directions_kl_loss:0.032 centroids_mean_magnitude:0.000 average_centroids_distance:0.000 average_action_variations_norm_l2:0.732 action_variations_mean:0.245 reconstructed_action_directions_kl_loss:0.032 action_mutual_information_loss:-0.000 action_state_distribution_kl_loss:0.000 ground_truth_observations:6.000 gumbel_temperature:0.970 observations_count:7.000 perceptual_loss_r0:0.511 observations_rec_loss_r0:0.026 perceptual_loss_r0_l0:0.511 perceptual_loss_r0_l1:0.085 perceptual_loss_r0_l2:0.166 perceptual_loss_r0_l3:0.170 perceptual_loss_r0_l4:0.070 perceptual_loss_r1:0.264 observations_rec_loss_r1:0.017 perceptual_loss_r1_l0:0.264 perceptual_loss_r1_l1:0.043 perceptual_loss_r1_l2:0.081 perceptual_loss_r1_l3:0.086 perceptual_loss_r1_l4:0.043 perceptual_loss_r2:0.535 observations_rec_loss_r2:0.069 perceptual_loss_r2_l0:0.535 perceptual_loss_r2_l1:0.121 perceptual_loss_r2_l2:0.173 perceptual_loss_r2_l3:0.137 perceptual_loss_r2_l4:0.057 loss:1.106 lr: 0.0004
step: 997/300000 loss_component_observations_rec:0.037 loss_component_perceptual_loss:0.797 loss_component_hidden_states_rec:0.220 loss_component_states_rec:0.001 loss_component_entropy:0.000 loss_component_action_directions_kl_divergence:0.000 loss_component_action_mutual_information:-0.000 loss_component_action_state_distribution_kl:0.000 avg_observations_rec_loss:0.037 avg_perceptual_loss:0.411 states_rec_loss:0.007 hidden_states_rec_loss:0.220 entropy_loss:0.953 samples_entropy:0.789 action_distribution_entropy:1.073 states_magnitude:0.657 hidden_states_magnitude:0.420 action_directions_mean_magnitude:0.001 action_directions_variance_magnitude:0.685 reconstructed_action_directions_mean_magnitude:0.001 reconstructed_action_directions_variance_magnitude:0.686 action_directions_reconstruction_error:0.000 action_directions_kl_loss:0.032 centroids_mean_magnitude:0.000 average_centroids_distance:0.000 average_action_variations_norm_l2:0.588 action_variations_mean:0.111 reconstructed_action_directions_kl_loss:0.031 action_mutual_information_loss:-0.000 action_state_distribution_kl_loss:0.000 ground_truth_observations:6.000 gumbel_temperature:0.970 observations_count:7.000 perceptual_loss_r0:0.483 observations_rec_loss_r0:0.026 perceptual_loss_r0_l0:0.483 perceptual_loss_r0_l1:0.083 perceptual_loss_r0_l2:0.162 perceptual_loss_r0_l3:0.159 perceptual_loss_r0_l4:0.060 perceptual_loss_r1:0.230 observations_rec_loss_r1:0.017 perceptual_loss_r1_l0:0.230 perceptual_loss_r1_l1:0.041 perceptual_loss_r1_l2:0.074 perceptual_loss_r1_l3:0.072 perceptual_loss_r1_l4:0.031 perceptual_loss_r2:0.521 observations_rec_loss_r2:0.069 perceptual_loss_r2_l0:0.521 perceptual_loss_r2_l1:0.122 perceptual_loss_r2_l2:0.170 perceptual_loss_r2_l3:0.136 perceptual_loss_r2_l4:0.046 loss:1.056 lr: 0.0004
step: 998/300000 loss_component_observations_rec:0.037 loss_component_perceptual_loss:0.769 loss_component_hidden_states_rec:0.218 loss_component_states_rec:0.001 loss_component_entropy:0.000 loss_component_action_directions_kl_divergence:0.000 loss_component_action_mutual_information:-0.000 loss_component_action_state_distribution_kl:0.000 avg_observations_rec_loss:0.037 avg_perceptual_loss:0.397 states_rec_loss:0.007 hidden_states_rec_loss:0.218 entropy_loss:0.921 samples_entropy:0.684 action_distribution_entropy:1.048 states_magnitude:0.656 hidden_states_magnitude:0.419 action_directions_mean_magnitude:0.001 action_directions_variance_magnitude:0.691 reconstructed_action_directions_mean_magnitude:0.001 reconstructed_action_directions_variance_magnitude:0.694 action_directions_reconstruction_error:0.000 action_directions_kl_loss:0.030 centroids_mean_magnitude:0.000 average_centroids_distance:0.000 average_action_variations_norm_l2:0.663 action_variations_mean:-0.024 reconstructed_action_directions_kl_loss:0.030 action_mutual_information_loss:-0.000 action_state_distribution_kl_loss:0.000 ground_truth_observations:6.000 gumbel_temperature:0.970 observations_count:7.000 perceptual_loss_r0:0.467 observations_rec_loss_r0:0.026 perceptual_loss_r0_l0:0.467 perceptual_loss_r0_l1:0.081 perceptual_loss_r0_l2:0.157 perceptual_loss_r0_l3:0.153 perceptual_loss_r0_l4:0.058 perceptual_loss_r1:0.227 observations_rec_loss_r1:0.017 perceptual_loss_r1_l0:0.227 perceptual_loss_r1_l1:0.040 perceptual_loss_r1_l2:0.074 perceptual_loss_r1_l3:0.071 perceptual_loss_r1_l4:0.032 perceptual_loss_r2:0.496 observations_rec_loss_r2:0.069 perceptual_loss_r2_l0:0.496 perceptual_loss_r2_l1:0.117 perceptual_loss_r2_l2:0.160 perceptual_loss_r2_l3:0.129 perceptual_loss_r2_l4:0.045 loss:1.025 lr: 0.0004
step: 999/300000 loss_component_observations_rec:0.037 loss_component_perceptual_loss:0.788 loss_component_hidden_states_rec:0.220 loss_component_states_rec:0.001 loss_component_entropy:0.000 loss_component_action_directions_kl_divergence:0.000 loss_component_action_mutual_information:-0.000 loss_component_action_state_distribution_kl:0.000 avg_observations_rec_loss:0.037 avg_perceptual_loss:0.407 states_rec_loss:0.007 hidden_states_rec_loss:0.220 entropy_loss:0.918 samples_entropy:0.722 action_distribution_entropy:1.030 states_magnitude:0.657 hidden_states_magnitude:0.420 action_directions_mean_magnitude:0.001 action_directions_variance_magnitude:0.689 reconstructed_action_directions_mean_magnitude:0.001 reconstructed_action_directions_variance_magnitude:0.691 action_directions_reconstruction_error:0.000 action_directions_kl_loss:0.031 centroids_mean_magnitude:0.000 average_centroids_distance:0.000 average_action_variations_norm_l2:0.602 action_variations_mean:-0.057 reconstructed_action_directions_kl_loss:0.030 action_mutual_information_loss:-0.000 action_state_distribution_kl_loss:0.000 ground_truth_observations:6.000 gumbel_temperature:0.970 observations_count:7.000 perceptual_loss_r0:0.477 observations_rec_loss_r0:0.026 perceptual_loss_r0_l0:0.477 perceptual_loss_r0_l1:0.082 perceptual_loss_r0_l2:0.158 perceptual_loss_r0_l3:0.156 perceptual_loss_r0_l4:0.061 perceptual_loss_r1:0.223 observations_rec_loss_r1:0.017 perceptual_loss_r1_l0:0.223 perceptual_loss_r1_l1:0.040 perceptual_loss_r1_l2:0.072 perceptual_loss_r1_l3:0.070 perceptual_loss_r1_l4:0.029 perceptual_loss_r2:0.522 observations_rec_loss_r2:0.069 perceptual_loss_r2_l0:0.522 perceptual_loss_r2_l1:0.122 perceptual_loss_r2_l2:0.166 perceptual_loss_r2_l3:0.130 perceptual_loss_r2_l4:0.057 loss:1.046 lr: 0.0004
Exception ignored in: <function Image.__del__ at 0x7f10c53415f0>
Traceback (most recent call last):
  File "/home/ryan/miniconda3/envs/video-generation/lib/python3.7/tkinter/__init__.py", line 3507, in __del__
    self.tk.call('image', 'delete', self.name)
RuntimeError: main thread is not in main loop
Exception ignored in: <function Image.__del__ at 0x7f10c53415f0>
Traceback (most recent call last):
  File "/home/ryan/miniconda3/envs/video-generation/lib/python3.7/tkinter/__init__.py", line 3507, in __del__
    self.tk.call('image', 'delete', self.name)
RuntimeError: main thread is not in main loop
Exception ignored in: <function Variable.__del__ at 0x7f10c5a7cb90>
Traceback (most recent call last):
  File "/home/ryan/miniconda3/envs/video-generation/lib/python3.7/tkinter/__init__.py", line 332, in __del__
    if self._tk.getboolean(self._tk.call("info", "exists", self._name)):
RuntimeError: main thread is not in main loop
Exception ignored in: <function Image.__del__ at 0x7f10c53415f0>
Traceback (most recent call last):
  File "/home/ryan/miniconda3/envs/video-generation/lib/python3.7/tkinter/__init__.py", line 3507, in __del__
    self.tk.call('image', 'delete', self.name)
RuntimeError: main thread is not in main loop
Exception ignored in: <function Variable.__del__ at 0x7f10c5a7cb90>
Traceback (most recent call last):
  File "/home/ryan/miniconda3/envs/video-generation/lib/python3.7/tkinter/__init__.py", line 332, in __del__
    if self._tk.getboolean(self._tk.call("info", "exists", self._name)):
RuntimeError: main thread is not in main loop
Exception ignored in: <function Image.__del__ at 0x7f10c53415f0>
Traceback (most recent call last):
  File "/home/ryan/miniconda3/envs/video-generation/lib/python3.7/tkinter/__init__.py", line 3507, in __del__
    self.tk.call('image', 'delete', self.name)
RuntimeError: main thread is not in main loop
Exception ignored in: <function Image.__del__ at 0x7f10c53415f0>
Traceback (most recent call last):
  File "/home/ryan/miniconda3/envs/video-generation/lib/python3.7/tkinter/__init__.py", line 3507, in __del__
    self.tk.call('image', 'delete', self.name)
RuntimeError: main thread is not in main loop
Exception ignored in: <function Image.__del__ at 0x7f10c53415f0>
Traceback (most recent call last):
  File "/home/ryan/miniconda3/envs/video-generation/lib/python3.7/tkinter/__init__.py", line 3507, in __del__
    self.tk.call('image', 'delete', self.name)
RuntimeError: main thread is not in main loop
Exception ignored in: <function Image.__del__ at 0x7f10c53415f0>
Traceback (most recent call last):
  File "/home/ryan/miniconda3/envs/video-generation/lib/python3.7/tkinter/__init__.py", line 3507, in __del__
    self.tk.call('image', 'delete', self.name)
RuntimeError: main thread is not in main loop
Exception ignored in: <function Image.__del__ at 0x7f10c53415f0>
Traceback (most recent call last):
  File "/home/ryan/miniconda3/envs/video-generation/lib/python3.7/tkinter/__init__.py", line 3507, in __del__
    self.tk.call('image', 'delete', self.name)
RuntimeError: main thread is not in main loop
Tcl_AsyncDelete: async handler deleted by the wrong thread
[1]    4391 abort (core dumped)  python3 train.py --config configs/02_breakout.yaml
(video-generation) G ➜ PlayableVideoGeneration git:(main) wandb: Program ended successfully.
wandb: Run summary:
wandb:                                                                train/perceptual_loss_r0_l2 0.15621022880077362
wandb:                                                                                      _step 981
wandb:                                       train/loss_component_action_directions_kl_divergence 3.709313273429871e-06
wandb:                                              train/reconstructed_action_directions_kl_loss 0.03681553900241852
wandb:                                                       train/loss_component_perceptual_loss 0.8016525010267893
wandb:                                          train/loss_component_action_state_distribution_kl 0.0
wandb:                                   train/reconstructed_action_directions_variance_magnitude 0.6641947031021118
wandb:                                                                train/perceptual_loss_r0_l0 0.4748656153678894
wandb:                                                            train/avg_observations_rec_loss 0.038134743149081864
wandb:                                                                   train/perceptual_loss_r1 0.24382595717906952
wandb:                                                                   train/perceptual_loss_r2 0.5221801400184631
wandb:                                                                train/perceptual_loss_r1_l1 0.04235634580254555
wandb:                                                                train/perceptual_loss_r0_l3 0.1558036208152771
wandb:                                                                   train/perceptual_loss_r0 0.4748656153678894
wandb:                                                                      train/samples_entropy 0.6838436126708984
wandb:                                                                train/perceptual_loss_r1_l4 0.03565201908349991
wandb:                                                            train/loss_component_states_rec 0.0014105471782386303
wandb:                                                      train/loss_component_observations_rec 0.038134743149081864
wandb:                                                                train/perceptual_loss_r2_l0 0.5221801400184631
wandb:                                                              train/hidden_states_magnitude 0.419323205947876
wandb:                                                           train/average_centroids_distance 1.7627293345867656e-05
wandb:                                                                  train/avg_perceptual_loss 0.413623904188474
wandb:                                       train/reconstructed_action_directions_mean_magnitude 0.0011238184524700046
wandb:                                                                train/perceptual_loss_r0_l4 0.06138833239674568
wandb:                                                                train/perceptual_loss_r1_l2 0.07608731091022491
wandb:                                                                train/perceptual_loss_r2_l1 0.12005559355020523
wandb:                                                                      train/states_rec_loss 0.0070527358911931515
wandb:                                                                train/perceptual_loss_r2_l4 0.05388300120830536
wandb:                                                                   train/gumbel_temperature 0.97057
wandb:                                                       train/action_mutual_information_loss -3.69977205991745e-05
wandb:                                                                   train/observations_count 7.0
wandb:                                                     train/action_directions_mean_magnitude 0.0011720473412424326
wandb:                                                            train/ground_truth_observations 6.0
wandb:                                                                train/perceptual_loss_r0_l1 0.08212797343730927
wandb:                                                               train/hidden_states_rec_loss 0.21878568828105927
wandb:                                                               train/action_variations_mean -0.14023524522781372
wandb:                                                                train/perceptual_loss_r2_l2 0.16710881888866425
wandb:                                                                                 train/loss 1.0599816392899204
wandb:                                                                                   train/lr 0.0004
wandb:                                             train/loss_component_action_mutual_information -5.549658089876175e-06
wandb:                                                                train/perceptual_loss_r1_l3 0.07853354513645172
wandb:                                                                                   _runtime 396.27002787590027
wandb:                                                                     train/states_magnitude 0.6560764312744141
wandb:                                                          train/action_distribution_entropy 1.0813705921173096
wandb:                                                            train/action_directions_kl_loss 0.037093132734298706
wandb:                                                    train/action_state_distribution_kl_loss 8.807628546492197e-06
wandb:                                                             train/observations_rec_loss_r2 0.06988160312175751
wandb:                                                             train/observations_rec_loss_r0 0.02682528644800186
wandb:                                                               train/loss_component_entropy 0.0
wandb:                                                             train/observations_rec_loss_r1 0.01769733987748623
wandb:                                                 train/action_directions_variance_magnitude 0.6632212996482849
wandb:                                                                         train/entropy_loss 0.897321343421936
wandb:                                                                train/perceptual_loss_r2_l3 0.13487893342971802
wandb:                                                                                       step 981
wandb:                                                             train/centroids_mean_magnitude 1.4390966498467606e-05
wandb:                                                                                 _timestamp 1626460241.496398
wandb:                                               train/action_directions_reconstruction_error 2.0398056221893057e-07
wandb:                                                    train/average_action_variations_norm_l2 0.6760526895523071
wandb:                                                     train/loss_component_hidden_states_rec 0.21878568828105927
wandb:                                                                train/perceptual_loss_r1_l0 0.24382595717906952
wandb: Syncing files in wandb/run-20210716_182405-w65gbanw:
wandb:   code/train.py
wandb: plus 8 W&B file(s) and 1 media file(s)
wandb:
wandb: Synced 02_breakout: https://app.wandb.ai/ryanburgert/video-generation/runs/w65gbanw
(video-generation) G ➜ PlayableVideoGeneration git:(main)

Are there any workarounds for this? I've tried finding references to tkinter, and there are none in the code that I can see.
UPDATE: I found that tesnor_displayer uses matplotlib.pyplot, which imports tkinter.

I'm using the provided conda environment and docker container. I get this error consistently each time I try running it.

[willi-menapace]

Dear Ryan,
I have never experienced this issue and cannot reproduce it, but as you say the likely cause is a plotting routine.

The code regularly plots the learned action space for debugging purposes and at iteration 1000 this is done for the first time. Two workarounds may be possible:

1. Line 159 of configs/02_breakout.yaml controls the plotting frequency: action_direction_plotting_freq: 1000
   you can specify a value of 1000000 to avoid plotting at all which should avoid the issue.
2. As suggested here, using the 'Agg' Matplotlib backend could allow plotting on your machine

Best