Data format

Splitting and merging tokens

Effect on foreign_ids

When a token is split, the original value of foreign_ids is parenthesised and _1 is appended to the result and used as the new value for the first token. Similarly, _2 is appended and used as the new value for the second token. Example:

Before split                        After split
------------                        -----------
token.foreign_ids = "note=foobar"   token1.foreign_ids = "(note=foobar)_1"
token.token_number = 10             token1.token_number = 10
                                    token2.foreign_ids = "(note=foobar)_2"
                                    token2.token_number = 11

Repeated splits will add further parentheses and suffixes:

note=foobar
(note=foobar)_1
((note=foobar)_1)_1

When two tokens are merged, the original values of foreign_ids are parenthesised and concatenated with . as separator. Example:

Before merge                        After merge
------------                        -----------
token1.foreign_ids = "note=foobar"  token.foreign_ids = "(note=foobar).(note=barfoo)"
token1.token_number = 10            token.token_number = 10
token2.foreign_ids = "note=barfoo"
token2.token_number = 11

Repeated mergers will add further parentheses and concatenation symbols:

note=foobar
(note=foobar).(note=barfoo)
((note=foobar).(note=barfoo)).(note=foobaz)