Recently, there have been certain important advances in the field of driver attention prediction in dynamic scenarios for the bionic perception of intelligent vehicles. However, there has been no unified paradigm on how to predict driver fixation points flexibly and accurately based on spatial-temporal features using an attention mechanism. To overcome this limitation, this paper proposes an improved multi-scale spatial-temporal fusion network model, which adopts an encoder-fusion-decoder architecture and can fully use the scale, spatial, and temporal information in video data. First, in the encoder, two independent feature extraction backbones, one 2D and another 3D, are used to extract four temporal-spatial features of different scales from the input video clip and align them in the feature dimension. Then, in the hierarchical spatial-temporal feature fusion, features from different levels are added to the channel and fused using an attention mechanism to achieve the 3D-2D soft combination effect guided by spatial features. Finally, in the hierarchical decoder and prediction module, hierarchical decoding and prediction are performed on temporal-spatial features of different branches, and the results of multiple branches are fused to generate saliency maps. Experiments on three challenging datasets show that the proposed method is superior to the state-of-the-art methods regarding several saliency evaluation metrics and can predict driver attention more accurately. By using an effective spatial-temporal fusion strategy, the proposed driver attention prediction method can detect important targets and identify risk areas for a human-like autonomous driving system.
In order to be able to clearly demonstrate the contribution of our proposed MTSF, we made a video demo, you can find it from here.
Code will be published after the article is accepted.