Add a cross entropy beam layer for learning to search model.

[lcy-seso]

• This layer is used in learning to search models, which is to solve complex joint prediction problems based on learning to search through a problem-defined search space.

  1. Specifically, the learning to search process for this layer begins with searching a target sequence from a nested sequence. In the first search step, top beam size sequences with highest scores, indices of these top k sequences in the original nested sequence, and the ground truth (also called gold) altogether (a triple) make up of the first beam.

  2. Then, several special positions, for example, start and end positions that define meaningful segments are searched. In these searches, top k positions with highest scores are selected, and then sequence, starting from the selected starts till ends of the sentences (or any fixed position) are taken to search next.

  3. We call the possible top k results returned in one search the beam. This search process can be repeated for pre-defined turns and leads to several beam expansions.

  4. Finally, the layer cross_entropy_over_beam takes all the beam expansions which contain several candidate targets found along the multi-step search process. cross_entropy_over_beam calculates cross entropy over the expanded beams with all the candidates in the beam as the normalized factor.

  5. Note that, if gold falls off the beam at search step $t$, then the cost is calculated over the beam at step $t$.

  6. This cost layer always works together with kmax_sequence_score_layer, sub_nested_seq_layer, and sequence_slice_layer to trim the input to form a sub-search space.