Issues with pre-computed MSAs while running create_individual_features.py

[Simon-Kristl]

So after more digging and some retry runs, I have realised what the issue is. So I used localcolabfold to precompute the MSAs and get the a3m files for all my 5500 proteins. When I then ran create_individual_features.py for all 5500 proteins it for some reason ran the MSA generation again (if I am correct in assuming that the logs from issue #486 correspond with mine in regards that my process also tries to recompute the MSAs).

0%| | 0/150 [elapsed: 00:00 remaining: ?]
SUBMIT: 0%| | 0/150 [elapsed: 00:00 remaining: ?]
PENDING: 0%| | 0/150 [elapsed: 00:01 remaining: ?]E0602 13:58:08.036076 140208070803200 colabfold.py:219] Sleeping for 7s. Reason: PENDING
RUNNING: 0%| | 0/150 [elapsed: 00:09 remaining: ?]
RUNNING: 5%|▍ | 7/150 [elapsed: 00:09 remaining: 03:10]E0602 13:58:15.969209 140208070803200 colabfold.py:219] Sleeping for 10s. Reason: RUNNING
RUNNING: 5%|▍ | 7/150 [elapsed: 00:20 remaining: 03:10]
RUNNING: 11%|█▏ | 17/150 [elapsed: 00:20 remaining: 02:35]E0602 13:58:26.908018 140208070803200 colabfold.py:219] Sleeping for 10s. Reason: RUNNING
RUNNING: 11%|█▏ | 17/150 [elapsed: 00:31 remaining: 02:35]
RUNNING: 18%|█▊ | 27/150 [elapsed: 00:31 remaining: 02:19]E0602 13:58:37.837252 140208070803200 colabfold.py:219] Sleeping for 5s. Reason: RUNNING
RUNNING: 18%|█▊ | 27/150 [elapsed: 00:37 remaining: 02:19]
RUNNING: 21%|██▏ | 32/150 [elapsed: 00:37 remaining: 02:15]E0602 13:58:43.792493 140208070803200 colabfold.py:219] Sleeping for 7s. Reason: RUNNING
RUNNING: 21%|██▏ | 32/150 [elapsed: 00:45 remaining: 02:15]
RUNNING: 26%|██▌ | 39/150 [elapsed: 00:45 remaining: 02:07]E0602 13:58:51.754858 140208070803200 colabfold.py:219] Sleeping for 10s. Reason: RUNNING
COMPLETE: 26%|██▌ | 39/150 [elapsed: 00:56 remaining: 02:07]
COMPLETE: 100%|██████████| 150/150 [elapsed: 00:56 remaining: 00:00]
COMPLETE: 100%|██████████| 150/150 [elapsed: 00:58 remaining: 00:00]

However when I tried processing a smaller subset of about 60 out of my 5500 precomputed MSAs it ran without issue.

The log for the subset I believe ran truly locally.

I0604 10:38:22.428106 140313531604736 create_individual_features.py:311] Running MMseqs2 for feature generation...
I0604 10:38:23.133966 140313531604736 objects.py:204] You chose to calculate MSA with mmseqs2.
Please also cite: Mirdita M, Schütze K, Moriwaki Y, Heo L, Ovchinnikov S and Steinegger M. ColabFold: Making protein folding accessible to all. Nature Methods (2022) doi: 10.1038/s41592-022-01488-1
I0604 10:38:24.407577 140313531604736 hhsearch.py:85] Launching subprocess "hhsearch -i /tmp/tmp9ki2u8g8/query.a3m -o /tmp/tmp9ki2u8g8/output.hhr -maxseq 1000000 -d /ceph/hpc/data/s26o02-02-users/cf_out/P16422_env/templates_101/pdb70"
I0604 10:38:24.410591 140313531604736 utils.py:36] Started HHsearch query
I0604 10:38:28.880973 140313531604736 utils.py:40] Finished HHsearch query in 4.470 seconds
I0604 10:38:28.884171 140313531604736 templates.py:879] Searching for template for: QEECVCENYKLAVNCFVNNNRQCQCTSVGAQNTVICSKLAAKCLVMKAEMNGSKLGRRAKPEGALQNNDGLYDPDCDESGLFKAKQCNGTSMCWCVNTAGVRRTDKDTEITCSERVRTYWIIIELKHKAREKPYDSKSLRTALQKEITTRYQLDPKFITSILYENNVITIDLVQNSSQKTQNDVDIADVAYYFEKDVKGESLFHSKKMDLTVNGEQLDLDPGQTLIYYVDEKAPEFSMQGLK
I0604 10:38:29.124986 140313531604736 templates.py:267] Found an exact template match 4mzv_A.
I0604 10:38:29.669705 140313531604736 templates.py:267] Found an exact template match 7e5n_A.
I0604 10:38:30.075695 140313531604736 templates.py:267] Found an exact template match 7pee_D.
W0604 10:38:30.086306 140313531604736 templates.py:548] The exact sequence NCTCPTNKMTVCSPDGPGGRCQCRALGSGMAVDCSTLTSKCLLLKARMSAPKPSEHALVDNDGLYDPDCDPEGRFKARQCNQTSVCWCVNSVGVRRTDKGDLSLRCDELVRTHHILIDLRHRPTAGAFNHSDLDAELRRLFRERYRLHPKFVAAVHYEQPTIQIELRQNTSQKAAGDVDIGDAAYYFERDIKGESLFQGRGGLDLRVRGEPLQVERTLIYYLDEIPPKF was not found in 7e5n_C. Realigning the template to the actual sequence.
I0604 10:38:30.086370 140313531604736 kalign.py:66] Aligning 2 sequences
I0604 10:38:30.086819 140313531604736 kalign.py:87] Launching subprocess "kalign -i /tmp/tmpu5455t1b/input.fasta -o /tmp/tmpu5455t1b/output.a3m -format fasta"
I0604 10:38:30.096883 140313531604736 utils.py:36] Started Kalign query
I0604 10:38:30.113757 140313531604736 kalign.py:94] Kalign stdout:

My question now is, why could this happen? Could it be that some of the a3m files are somehow not complete and it therefore runs the MSA generation again? I am sure that the naming is not the issue.
I also noticed that the a3m files were modified when this script ran, is this normal?

I am using version 2.0.3-foss-2023a-CUDA-12.1.1

Old issue I mistakenly thought was the problem.

I am doing a large scale ppi screening on a slurm system. I am using localcolabfold MMseqs2 for MSA generaton and then run the create_individual_features.py as part of the localcolabfold --> alphapulldown pipeline, there were 8 proteins in my test run to process which it did pretty quickly, around 2 minutes. However when I am now attempting to run this script on a set of 5500 proteins I run into an issue where the process for some reason sleeps and the stated reason is "PENDING" or "RUNNING". In the given example below it spends around 49 seconds sleeping of the 58 seconds that it had been running. I am not doing any parallel arrays, only one job is running. I have also redone the test run and it still runs without issue. From job efficiency reports there doesn't seem to be any issues with the given hardware.

create_individual_features_34500014_4294967294_err.txt the log file for the subset of 60 that I believe ran normally
create_individual_features_34176000_4294967294_err.txt this is the log from the affected job
protfeature.sh the used slurm script