Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR implements a solution to issue #19. It simply provides the user with the option to turn on 'split_penalty' in the settings, which will add a 0.1 penalty to the edit distance of Alignments that split tokens apart in a replacement.
I have racked my brain and I think there is no issue with setting a penalty of 0.1 for operations that split a token, because it will only impact which Alignment is chosen in cases where the edit distance is equal. The unit tests all still pass, but I added it as a setting instead of hardcoding it in the source so that we might find a way to test it on a larger dataset, maybe with Jan's help/input. Please let me know what you think or if you see any issues with this approach.