modify lstm ouput for aggregated model

[rickymf4]

Using the h value of the lstm top layer as the input for aggregated model is better, and easier to converge.

• Train log:
  INFO:tensorflow:/job:master/task:0: training step 218206| Hit@1: 0.87 PERR: 0.72 GAP: 0.81 Loss: 5.09105
  INFO:tensorflow:/job:master/task:0: training step 218207| Hit@1: 0.85 PERR: 0.73 GAP: 0.81 Loss: 5.16746
  INFO:tensorflow:/job:master/task:0: training step 218208| Hit@1: 0.79 PERR: 0.68 GAP: 0.80 Loss: 5.13253
  INFO:tensorflow:/job:master/task:0: training step 218209| Hit@1: 0.90 PERR: 0.76 GAP: 0.82 Loss: 5.16415
  INFO:tensorflow:/job:master/task:0: training step 218210| Hit@1: 0.88 PERR: 0.76 GAP: 0.80 Loss: 5.36496
  INFO:tensorflow:/job:master/task:0: training step 218211| Hit@1: 0.83 PERR: 0.72 GAP: 0.82 Loss: 4.48507
  INFO:tensorflow:/job:master/task:0: training step 218212| Hit@1: 0.85 PERR: 0.71 GAP: 0.83 Loss: 4.91971

• Eval on frame level validation set,
  Avg_Hit@1: 0.838 | Avg_PERR: 0.705 | MAP: 0.367 | GAP: 0.780 | Avg_Loss: 5.795533

• The following are the training status
  The orange curves: original code using all concatenated state.
  The blue curves: modified code using top layer h state.
  

[LeegleechN]

Thanks! I've wanted to investigate this for a while.

[csyanbin]

Hi, what is the base_learning_rate and num_moe_experts in video_level_models?