Optimization loop failed

[DittmanC]

I was trying to run the example data for the predictSV step, however, i got a "Optimization loop failed". can you give me some advice on this issue? thanks

predictSV --hic-5k SKNAS-MboI-allReps-filtered.mcool::/resolutions/5000 --hic-10k SKNAS-MboI-allReps-filtered.mcool::/resolutions/10000 --hic-50k SKNAS -MboI-allReps-filtered.mcool::/resolutions/50000 -O SK-N-AS -g hg38 --balance-type ICE -- output-format full --prob-cutoff-5k 0.8 --prob-cutoff-10k 0.8 --prob-cutoff-50k 0.99999
root INFO @ 05/18/23 15:29:26:

ARGUMENT LIST:

Cool URI at 5kb = SKNAS-MboI-allReps-filtered.mcool::/resolutions/5000

Cool URI at 10kb = SKNAS-MboI-allReps-filtered.mcool::/resolutions/10000

Cool URI at 50kb = SKNAS-MboI-allReps-filtered.mcool::/resolutions/50000

Balance Type = ICE

Reference Genome = hg38

Included Chromosomes = ['#', 'X']

Probability Cutoff for 5kb SVs = 0.8

Probability Cutoff for 10kb SVs = 0.8

Probability Cutoff for 50kb SVs = 0.99999

Output File Prefix = SK-N-AS

Output Format = full

Log file name = SK-N-AS.log

root INFO @ 05/18/23 15:29:26: Predict SVs at 5kb resolution ...
root INFO @ 05/18/23 15:29:27: matched sequencing depth in human at 10Kb: 801 81600.06574439
root INFO @ 05/18/23 15:29:27: Load CNN models from /disk3/users/dittman/.con da/envs/py38/lib/python3.8/site-packages/eaglec/data/bulk/50M-100M ...
2023-05-18 15:29:27.987789: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow bina ry is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instruction s in performance-critical operations: AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-05-18 15:29:27.992225: I tensorflow/core/common_runtime/process_util.cc:146] Creating new thread pool with default inter op setting: 2. Tune using inter_op_parallelism_threads for best performance.
root INFO @ 05/18/23 15:29:29: Done
root INFO @ 05/18/23 15:29:29: Interemediate results at the 5kb resolution wi ll be cached to .SKNAS-MboI-allReps-filtered.mcool.91006977.ICE.None.100000.None

eaglec.scoreUtils INFO @ 05/18/23 16:39:38: (chr1, chr1): Total 303446 candidates left after filter ing
2023-05-18 16:40:15.548456: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:176] None of the MLIR O ptimization Passes are enabled (registered 2)
2023-05-18 16:40:15.569822: I tensorflow/core/platform/profile_utils/cpu_utils.cc:114] CPU Frequency: 21000000 00 Hz
tensorflow WARNING @ 05/18/23 16:41:39: 5 out of the last 1004 calls to <function Model.make_pr edict_function..predict_function at 0x7efff41989d0> triggered tf.function retracing. Tracing is expens ive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) p assing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define yo ur @tf.function outside of the loop. For (2), @tf.function has experimental_relax_shapes=True option that rela xes argument shapes that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/ guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more deta ils.
tensorflow WARNING @ 05/18/23 16:41:40: 6 out of the last 1005 calls to <function Model.make_pr edict_function..predict_function at 0x7efff4198940> triggered tf.function retracing. Tracing is expens ive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) p assing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define yo ur @tf.function outside of the loop. For (2), @tf.function has experimental_relax_shapes=True option that rela xes argument shapes that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/ guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more deta ils.
2023-05-18 16:43:59.907971: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled

eaglec.scoreUtils INFO @ 05/18/23 17:52:42: (chr2, chr2): Total 294222 candidates left after filter ing
eaglec.scoreUtils INFO @ 05/18/23 18:28:21: (chr3, chr3): Total 334652 candidates left after filter ing
2023-05-18 18:32:30.953895: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
2023-05-18 18:33:10.529717: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
2023-05-18 18:33:10.541688: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
2023-05-18 18:33:10.578712: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
2023-05-18 18:33:10.590651: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
2023-05-18 18:33:10.631463: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
2023-05-18 18:33:10.685669: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
2023-05-18 18:33:10.697107: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
2023-05-18 18:33:10.960220: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
2023-05-18 18:33:10.995985: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
2023-05-18 18:33:11.124616: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
eaglec.scoreUtils INFO @ 05/18/23 19:11:34: (chr4, chr4): Total 172673 candidates left after filter ing
eaglec.scoreUtils INFO @ 05/18/23 19:40:44: (chr5, chr5): Total 203266 candidates left after filter ing
eaglec.scoreUtils INFO @ 05/18/23 20:00:29: (chr6, chr6): Total 369756 candidates left after filter ing
2023-05-18 20:05:41.496671: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
eaglec.scoreUtils INFO @ 05/18/23 20:25:16: (chr7, chr7): Total 195819 candidates left after filter ing
2023-05-18 20:28:10.832702: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
2023-05-18 20:28:10.932666: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
2023-05-18 20:28:10.962664: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
2023-05-18 20:28:10.974770: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
2023-05-18 20:28:10.994697: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
2023-05-18 20:28:11.031622: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
2023-05-18 20:28:11.069519: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
2023-05-18 20:28:11.307348: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
2023-05-18 20:28:11.365171: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
2023-05-18 20:28:11.804032: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
2023-05-18 20:28:11.827570: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled
eaglec.scoreUtils INFO @ 05/18/23 20:40:54: (chr8, chr8): Total 180975 candidates left after filter ing
2023-05-18 20:42:53.817573: W tensorflow/core/kernels/data/model_dataset_op.cc:205] Optimization loop failed: Cancelled: Operation was cancelled

[XiaoTaoWang]

It seems that there is a compatibility problem between different versions of TensorFlow. Based on my testing, EagleC performed well with versions 2.3.0, 2.9, and 2.11.0. Which version have you installed?