[#4525] fix gold.align

[tamuhey]

from #4525

• Fixed the bug described in gold.align seems to be broken #4525
• Made the gold.align code simpler and more efficient

Types of change

bug fix and refactor.

Checklist

• I have submitted the spaCy Contributor Agreement.
• I ran the tests, and all new and existing tests passed.
• My changes don't require a change to the documentation, or if they do, I've added all required information.

[tamuhey]

Below is a notebook on profiling the old and new gold.align functions.
https://gist.github.com/tamuhey/1f16a107030659ff75781889fbc37fd1

[honnibal]

Really appreciate your help @tamuhey ! Thanks so much. I've had a fair bit of trouble with the alignment code over time.

[honnibal]

I merged a bit hastily here, and missed something. Will make a commit to master to address it.

[adrianeboyd]

This has fixed some of the subtok problems I was having (this is great!) but now I have cases where texts aren't being aligned like they were before. At least one problem is when the raw text starts with a whitespace character. A minimal example:

[
  {
    "id":0,
    "paragraphs":[
      {
        "raw":" a",
        "sentences":[
          {
            "tokens":[
              {
                "head":0,
                "dep":"ROOT",
                "tag":"A",
                "orth":"a",
                "ner":"U-DATE",
                "id":0
              }
            ],
            "brackets":[

            ]
          }
        ]
      }
    ]
  }
]

[adrianeboyd]

Ah, wait, I think I managed to test this at just the wrong point and the failing overall alignment was what Matt fixed above.

Now it doesn't throw an error but I get incorrect results with the GoldParse for the same example:

# loaded through GoldCorpus:
gold.words  # ['a', 'a']
gold.ner    # ['B-DATE', 'U-DATE']
gold.labels # ['ROOT', 'ROOT']

The difference is i2j_multi:

cost, i2j, j2i, i2j_multi, j2i_multi = align([" ", "a"], ["a"])
i2j_multi   # {0: 0}

[tamuhey]

@adrianeboyd Thanks for catching! I'm trying to fix this bug in (#4537)