Mapping from target to model positions does not contain all positions of internal model numbering

[sacdallago]

Traceback (most recent call last):
  File "/n/groups/marks/software/anaconda_o2/envs/evcouplings_backend_develop/lib/python3.5/site-packages/evcouplings/utils/pipeline.py", line 389, in execute_wrapped
    outcfg = execute(**config)
  File "/n/groups/marks/software/anaconda_o2/envs/evcouplings_backend_develop/lib/python3.5/site-packages/evcouplings/utils/pipeline.py", line 185, in execute
    outcfg = runner(**incfg)
  File "/n/groups/marks/software/anaconda_o2/envs/evcouplings_backend_develop/lib/python3.5/site-packages/evcouplings/complex/protocol.py", line 576, in run
    return PROTOCOLS[kwargs["protocol"]](**kwargs)
  File "/n/groups/marks/software/anaconda_o2/envs/evcouplings_backend_develop/lib/python3.5/site-packages/evcouplings/complex/protocol.py", line 454, in best_hit
    kwargs["paralog_identity_threshold"]
  File "/n/groups/marks/software/anaconda_o2/envs/evcouplings_backend_develop/lib/python3.5/site-packages/evcouplings/complex/protocol.py", line 427, in _load_monomer_info
    annotation_table = read_species_annotation_table(annotations_file)
  File "/n/groups/marks/software/anaconda_o2/envs/evcouplings_backend_develop/lib/python3.5/site-packages/evcouplings/complex/similarity.py", line 61, in read_species_annotation_table
    annotation_file
evcouplings.utils.config.InvalidParameterError: provided annotation file 7a74f5bd34b11f505e287d1fd767fbf9/align_1/7a74f5bd34b11f505e287d1fd767fbf9_annotation.csv has no annotation information
[cd174@login06 runs]$ clearcat 00c7b99d38370a666e9d32e337d25cd0/00c7b99d38370a666e9d32e337d25cd0.failed^C
[cd174@login06 runs]$ cat 00c7b99d38370a666e9d32e337d25cd0/00c7b99d38370a666e9d32e337d25cd0.failed
Traceback (most recent call last):
  File "/n/groups/marks/software/anaconda_o2/envs/evcouplings_backend_develop/lib/python3.5/site-packages/evcouplings/couplings/mapping.py", line 219, in patch_model
    for pos in model.index_list
  File "/n/groups/marks/software/anaconda_o2/envs/evcouplings_backend_develop/lib/python3.5/site-packages/evcouplings/couplings/mapping.py", line 219, in <listcomp>
    for pos in model.index_list
KeyError: 4

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/n/groups/marks/software/anaconda_o2/envs/evcouplings_backend_develop/lib/python3.5/site-packages/evcouplings/utils/pipeline.py", line 389, in execute_wrapped
    outcfg = execute(**config)
  File "/n/groups/marks/software/anaconda_o2/envs/evcouplings_backend_develop/lib/python3.5/site-packages/evcouplings/utils/pipeline.py", line 185, in execute
    outcfg = runner(**incfg)
  File "/n/groups/marks/software/anaconda_o2/envs/evcouplings_backend_develop/lib/python3.5/site-packages/evcouplings/mutate/protocol.py", line 326, in run
    return PROTOCOLS[kwargs["protocol"]](**kwargs)
  File "/n/groups/marks/software/anaconda_o2/envs/evcouplings_backend_develop/lib/python3.5/site-packages/evcouplings/mutate/protocol.py", line 202, in complex
    c = MultiSegmentCouplingsModel(kwargs["model_file"], *segment_objects)
  File "/n/groups/marks/software/anaconda_o2/envs/evcouplings_backend_develop/lib/python3.5/site-packages/evcouplings/couplings/mapping.py", line 388, in __init__
    r.patch_model(model=self)
  File "/n/groups/marks/software/anaconda_o2/envs/evcouplings_backend_develop/lib/python3.5/site-packages/evcouplings/couplings/mapping.py", line 223, in patch_model
    "Mapping from target to model positions does "
ValueError: Mapping from target to model positions does not contain all positions of internal model numbering

Runs with this error:

/n/groups/marks/web/backend/runs/

8dbdbb37177af143b6a2f815e805dd05
80600be20c3735f21011334f3b1235e0
9e1b6ed888a8a15aecce724939bca904
90c3840d96414159e4d37765a38f71e2
2ef90e06afc175f200bac8dab09a9ed5
00c7b99d38370a666e9d32e337d25cd0

That's 7 out of 15 failed jobs.

[sacdallago]

Another job has failed on this. Any update on reason / what should be told to the people that encounter this error?

[thomashopf]

@aggreen any idea why this happens?

[aggreen]

This is a bug on our end. When we define the segments after monomer concatenation, we do so in the following way (line 68 of complex/protocol.py):

    # merge segments - this allows to have more than one segment per
    # "monomer" alignment
    segments_1 = _modify_segments(kwargs["first_segments"], "A")
    segments_2 = _modify_segments(kwargs["second_segments"], "B")
    segments_complex = segments_1 + segments_2
    outcfg["segments"] = [s.to_list() for s in segments_complex]

The problem with this is that it assumes that the same positions that were lowercased in the monomer alignments are lowercased in the concatenated alignment. This is not a valid assumption, because after concatenation we repeat the row and column filtering procedure. This can sometimes result in fewer positions defined in the segments kwarg than actually exist in the model, leading to the error users are seeing.

The good news is that this only seems to affect the mutate stage (specifically the MultiSegmentCouplingsModel class). The CouplingScores.csv file has a number of positions that corresponds to the actual concatenated alignment, with the correct number of positions per segment.

TLDR: tell users that this is an issue with the mutate and dock stages that we are trying to address. The results for the couplings and compare stage, which is probably what they care the most about anyway, are valid

[thomashopf]

Okay so from that I take it makes most sense for now to disable mutate and dock on EVcomplex server runs at the moment, right? Because it kills the entire run if the problem occurs.

[aggreen]

Yeah that seems like a good temporary solution.

…

On Fri, May 24, 2019 at 9:35 AM thomashopf ***@***.***> wrote: Okay so from that I take it makes most sense for now to disable mutate and dock on EVcomplex server runs at the moment, right? Because it kills the entire run if the problem occurs. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#213?email_source=notifications&email_token=AB4KMXWCBVKMKL5BIYACKATPW7VKVA5CNFSM4HC76SXKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODWFLV4Y#issuecomment-495631091>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AB4KMXSDVB7I6XYHOQYKQELPW7VKVANCNFSM4HC76SXA> .

[sacdallago]

@thomashopf @aggreen has that been done, or should I do it? 🤔

[aggreen]

I have not done it

…

On Mon, May 27, 2019 at 5:50 AM Christian Dallago ***@***.***> wrote: @thomashopf <https://github.com/thomashopf> @aggreen <https://github.com/aggreen> has that been done, or should I do it? 🤔 — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#213?email_source=notifications&email_token=AB4KMXQGEFEVS5MFYQAG7H3PXOVIBA5CNFSM4HC76SXKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODWJLK3Y#issuecomment-496153967>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AB4KMXQAKUB4G4XSPQ5ORVLPXOVIBANCNFSM4HC76SXA> .

[thomashopf]

Not done yet - please update if you can

[sacdallago]

Sorry -- My bad, I got lost.

I've changed:

stages:
    - align_1
    - align_2
    - concatenate
    - couplings
    - compare
    - mutate
    - fold

to

stages:
    - align_1
    - align_2
    - concatenate
    - couplings
    - compare

aka. Removed mutate and fold. I assumed docking is fold in @thomashopf 's comment #213 (comment) .

[thomashopf]

Thanks!

[cross12tamu]

I am getting this error as well in a few of my jobs. Let me know if you'd like any of the specific error logs.